On Thu, 2020-04-02 at 11:52 +0100, Leigh Griffin wrote:

[up front: apologies for any weird formatting in this email, Evolution
is crashing on me like crazy while I try to edit it, so I'm sending it
from Roundcube which I don't normally do]

> I'm also not entirely sure about Adam Saleh, but he does not have any
> > infra activity I can find since the end of January and lists himself as
> > working for Exponea on Github and his personal blog.
> >
>
> Adam is working in CPE. CPE isn't entirely about Infra, he is working
> on CI
> improvements at the moment alongside some others

Okay, well, next time I see him I'll mention he might want to update the
Exponea references :P

But "CPE isn't entirely about Infra" is sort of the point here: my
initial assertion was that there are fewer people working on
*Fedora-relevant app development* than there were before.

I'm happy to
accept that there may be more people in the CPE team than there were at
some point in its past, but that's not what I'm actually interested in
here. CI is great (whichever CI you mean, I lose track sometimes :>) but
someone working on CI is not someone working on the Fedora app
infrastructure, or on sourcing/deploying/integrating/maintaining
outsourced replacements for it, which is the actual problem space here.

> > If I'm missing anyone, please do correct me.
>
> Other developers in our pool right now are:
> - Ryan Lerch

I mentioned already that Ryan could go in either list or neither - he
was around at both times but I wasn't sure whether to count him as a
developer or not. But he can't count to one list but not the other, as
he's been here all along. :)

> - Michal Konecny
> - Mohan Boddu
> - Tomas Hrcka
> - Nils Philippsen
> - Vipul Siddharth
> - James Richardson
>
> We additionally have 2 new Ops folks joining us over the next 2 weeks.
>
> Across them, the majority are working on the Fedora aspects (Infra,
> Dev,
> Releng) of the CPE remit.

So yeah, let's discount the releng folks first, because releng has
existed all along

and - as I said - my original statement was not about
"people who are organizationally in the same team as the people who work
on Fedora app stuff" but "people who work on Fedora app stuff". So that
lets out Mohan and Tomas.

As for the others: so, I didn't count Michal at first as I could not
find any infra-related activity for him, but since you included him I
looked closer, and found he's just hiding himself really well - his
github account name, for some inaccountable reason, is "Zlopez", and his
profile doesn't have his real name in it.

So I finally got his logs:

https://github.com/Zlopez
https://pagure.io/user/zlopez

...and yeah, there's some infra activity there, so add him to the list.

Initially I was just looking at Github logs, as all the infra stuff I
could find was hosted there, and Nils' list is:

https://github.com/nphilipp

which was busy up to the end of December but looked extremely empty all
of this year, so I figured he'd switched track or role or something and
didn't count him. But now I look closer, it seems what happened is he's
been working almost exclusively on a thing called rpmautospec, which is
hosted on Pagure:

https://pagure.io/Fedora-Infra/rpmautospec

so, he's clearly working hard on something, although I'm not sure it's
directly a part of Fedora app/infra stuff - it's an automatic packaging
thingy, it looks like, I'm not sure what it's being used for. In fact,
it seems like pingou and Adam Saleh are doing quite a bit of work on
this thing too, so it's clearly sucking up quite a lot of developer
time. It's also a fairly new project, which seems interesting given that
"we don't have time to maintain all the projects we have already" is the
recurring refrain here.

Here's Vipul's lists:

https://github.com/siddharthvipul
https://pagure.io/user/siddharthvipul1

he seems to work exclusively on CentOS CI. Okay, Fedora *uses* CentOS
CI, but presumably back in the 2018 timeframe, someone (whether that's
Vipul or someone else) was working on CentOS CI who wasn't included in
my list, because I only listed people working on Fedora stuff. So this
still seems like a wash. 

James I didn't count as he's listed as an intern. But here's his Github
log (I can't find him on Pagure):

https://github.com/james02135

so I don't mean this as a knock at all, but he's obviously not
equivalent to one regular full-time dev, nor would we (I hope) expect
him to be.

So if we include Ryan on both lists and add Michal, we get to this.
Previous:

puiterwijk
Randy (bowlofeggs)
pingou
jcline
Ralph
jflory
abompard
rlerch

Current:

abompard
lrossetti/odra
pingou
scoady
cverna
mkonecny (Zlopez)

If we also count rpmautospec, we add Nils and Adam to the current list:

abompard
lrossetti/odra
pingou
scoady
cverna
mkonecny (Zlopez)
nphilipp
asaleh

and it looks like it's about even. But that's counting rpmautospec,
which seems to be an odd counterpoint to the overall CPE narrative that
"we have too many projects and we're trying to get rid of the
maintenance burden" - you already don't have resources to maintain the
things that Fedora very definitely needs and is using right now, but you
*do* have resources to dedicate some or all of the time of three
engineers to a brand new project which certainly looks cool but is
definitely a new invented-here thing that isn't absolutely essential to
Fedora's needs and isn't replacing an existing project?

To be really clear: I don't want the takeaway from this to be "Adam is
very mad and doesn't want CPE to be allowed to work on cool new projects
any more". I like cool new projects! Cool new projects are great! I
write them myself sometimes! I'm just having trouble joining up the dots
here in terms of high-level strategy.

> The number of active developers on Fedora initiatives has gone up
> drastically since I joined the team in 2019. You are possibly not
> seeing
> that as the team have moved from a model of siloed work on multiple
> apps,
> swimming against the tid working 16 hour days, to working on team
> oriented
> initiatives to add real value to the ecosystem. So the noise of working
> on
> multiple small things at once is not as loud as it was in 2018 which is
> giving that illusion.

What I'm looking at is the commit logs. That's all that ultimately
matters. But see above revisions, of course.

[snip the stuff about whether we need elections apps and blahblah
because I don't have anything to add there]

> > > Now when the CPE team goes and ask for more people because we struggle
> > with
> > > current situation, I can only guess that these non critical applications
> > > are mentioned. If I was putting my own money to sponsor a team to help
> > > building a Linux distribution I would be asking why do we have to
> > develop a
> > > calendar application or why do we need a custom git forge. I personally
> > > find really great that the different use cases and requirements for the
> > use
> > > of Pagure were gathered and I am convinced that people working on this
> > did
> > > their very best to find a use case to justify the investment needed in
> > > Pagure but it seems that we don't have such a thing.
> >
> > I think other people are following up on this, but from following the
> > discussion, it appears to me that there seems to have been a large slip
> > somewhere along the line from Ben submitting a list of ~50 Fedora
> > submissions, to Leigh suggesting that (after those were, mysteriously,
> > "summarized" somehow)
>
> Can you help me understand what the mystery is about? We took in 300+
> requirements that we distilled down into the generic list that we came
> up
> with, many of which are buckets / Epics. Every single requirement was
> analysed. I have said this multiple times but please, if this is still
> a
> mystery to you, let me know how I can help clarify it.

The specific gap I am talking about is the gap between this list
submitted by Ben Cotton on behalf of Fedora:

https://lists.fedoraproject.org/archives/list/council-discuss@lists.fedoraproject.org/thread/OEPDGVKYAJIQ6YYZU5J3XT3LHXFROFI5/

and the 'summarized' list you have pointed to here:

https://hackmd.io/@My419-DISUGgo4gcmO1D6A/HJB74lcL8

Now, right off the bat, we have a huge problem. The "summarized" list
claims this:

"after duplications and similar in theme requirements were merged
together, we were left with the following unique User Story list:"

you've also phrased the same thing slightly differently on the mailing
list:

"As a team we evaluated every single requirement (over 300 of them) and
the presentation in the combined User Story list is an amalgamation of
like for like User Stories to capture at a high level the spirit of the
requests." 

However, the top three points on the Fedora list relate to F/OSS and
self-hosting principles. These points are entirely missing from the
"summarized" list.

They have not been "merged" with "duplications" or
"similar in theme requirements". The "spirit" of them has not been
"captured" in a "User Story". They are just *not there*. They have not
been summarized. They were dropped. So the claim is false, and there was
not just a summarization process here, there was some sort of filtering
process. Things from the stakeholder lists were outright left out of the
"summarization". I'll pick this up again later.

Of the other 44 points on the Fedora list, 23 mention dist-git
specifically. The term "dist-git" is mentioned 40 times. Some of these
aren't truly "dist-git" specific, and are really just generic git or
forge requirements, but many of them *are* specific to dist-git and the
packager workflow integration; I'd count at least points 9, 17, 18, 19,
20, 22, 23, 24, 25, 26, 28, 29, and maybe 37 in this bucket. In the
"summarized" list, the string "dist-git" does not appear once. I can buy
that a few of the requirements are *possibly* "summarized" into the
points about access control, but generic access control requirements do
not at all cover all the details of dist-git integration. Effectively,
all the details relating to dist-git and packager workflow interaction -
which constitute the large majority of significant requirements on the
Fedora list that wouldn't be satisfied by all of the candidates - are
lost in the summarized list.

You say that all the (original 300+) requirements were analyzed, but
when you have been asked for specific details on any of the issues
relating to dist-git / packager workflow integration, these are the
kinds of answers you have given:

"The Fedora specific requirements (as I am on the Fedora lists here) had
very few unique needs such that both Gitlab and Pagure would have
satisfied their desire."

If you read the two lists above and my notes, I do not see how you can
possibly claim that Fedora "had very few unique needs".

This was the
quote that first got me really worried that Fedora's needs had not been
properly considered and understood. (We can also note, of course, that
while both Gitlab CE and Pagure can potentially satisfy the "open
source" and "self-hosted" requirements, Gitlab Ultimate - to which the
decision appears to be weighted, a suggestion no-one has yet denied,
beyond a very mealy-mouth and deniable "no option is off the table yet"
- does not satisfy either). You later made more or less the same claim
again:

"The needs of Fedora, CentOS, RHEL and the CPE team were weighed equally
in our decision. Fedora had very few specific needs in their
requirements where as some stakeholder groups had."

which only makes me worry more.

[to a question about Bugzilla package workflow integration] "I don't
have an answer to this as we haven't done that deep level of tooling
analysis and integration analysis yet."

How can you have considered Fedora's requirements with regard to tooling
and integration if you haven't "done a deep level of tooling analysis
and integration analysis yet"?

[when Till attempted to make similar points about the dist-git
requirements] "The User Stories are deliberately vague and that
represents around 10 unique requirements that boil down to having group
membership and management capabilities."

"Having group memembership and management capabilities" is nowhere close
to covering the dist-git integration requirements. Pagure had these more
or less from day one, but we still had to spend months on engineering
the dist-git integration. You cannot count generic group membership and
ACL capabilities as "covering" these requirements, and if you did, that
was a fundamental misunderstanding of what Fedora was requesting.

[when asked if you know how many existing integrations with Pagure there
are] "No. Our analysis will tell us that in due course"

Once again - if you don't even know that, how can you possibly have
fully evaluated Fedora's detailed requirements regarding dist-git /
packager workflow integration? You defended this by saying "I do know of
many important integrations that I come into contact with during my day
to day job in the team, that doesn't mean I know every single one of
them. We analysed the critical path needs via the requirements and any
reference to tooling integrations (e.g. fedora messaging was reference
in multiple conversations) will be brought forward with us for deeper
technical conversations", but the extent to which the detail in Fedora's
list is lost in the "summarized" list gives me (and I suspect others) no
confidence at all that this analysis was done correctly.

To put it slightly more generally: I think Fedora as a whole would have
expected that CPE was *already* deeply familiar with the importance of
dist-git / packager workflow integration.

That CPE would *already*
understand that this - along with F/OSS principles - would be the meat
of Fedora's "unique" requirements in this process.

And that the rolling
of these into "user stories" would be a useful technique for
facilitating the process, but not the Holy Grail of the process outside
of which nothing would matter at all; I would have expected the
knowledge that (for instance) pingou already has about this stuff to
have just been included directly in the decision-making process.

However, as you've described it, that doesn't seem to have been what's
happened. Instead, this deep base of existing knowledge within your team
seems to have been sidelined in the decision process, and instead the
decision has been made based mostly or entirely on this apparently
somewhat wobbly process of bundling up and "summarizing" requirements
(through three levels of the telephone game, from Fedora lists to the
Council to this "summarized" list) as "User Stories". Or at least, that
is how you are presenting the process as having worked and the decision
as having been made, and the focus on generic forge-type requirements
and the repeated contention that Fedora had "very few" "unique"
requirements reinforces that impression.

On a general note, I'm really having trouble understanding the overall
logic behind this "requirements" process at all. You've mentioned there
were over 300 requirements from the stakeholders and you summarized them
into this list. But when people have pressed you on individual items in
the summarized list, you've said stuff like this:

[on private PR comments] "It is a valid requirement from a stakeholder
that we had to take at face value."

OK, but why did *this* requirement make the "summarized" list when
others - as I've mentioned above - didn't?

I am discounting the claim
that the list somehow summarizes all the given requirements, because it
very clearly and inarguably does not. You had to "take it at face
value", but there were 300 other requirements that had to be taken "at
face value" as well, and not all of them made the list. As that's true,
you can't duck Neal's question: why did *this specific request*, which
Neal gives various reasons for not considering a great priority, make
the summarized list? You can say "we think you're wrong that it's not a
very important request, we think it is an important request for reason
X", but you can't just duck the question by saying there was no choice
made that can be questioned. Clearly there *was*.

[on the mobile app requirement] "It's a requirement that came from the
Fedora Community and one we could not satisfy as soon as Github was
ruled out. I do agree the experience is not great on it."

Again, given the context that there clearly was some sort of filtering
process here, this reads as very bizarre. A bells-and-whistles feature
requirement that you acknowledge only Github barely fulfils, badly,
makes the summarized list, but "we want it to be F/OSS" and multiple
very specific fundamental dist-git integration requirements somehow
don't make the summarized list or are garbled beyond recognition?

[on gists] "It's not up to me to gauge the merit of that as a use case
but it's a valid requirement which we considered."

But the question is why *this* item from the list of 300+ made the
"summarized" list rather than being smooshed into something very vaguely
related, or dropped entirely. Did someone *else* gauge the merit of that
as a use case? Or did you make the decision on some other grounds?

You have also implied there was some kind of "weighting" involved, when
asked if a requirement that "Github" be at the top of the UI would have
been accepted:

"Would it be weighed equally with a more core functional requirement?
The answer is of course no but we would
have respected that request."

but you have not provided any further detail on exactly how this
"weighting" worked and how Fedora's requirements (those which were not,
apparently, entirely dropped) were "weighted". How was Fedora's "we want
it to be open source" "weighed" against requests that cannot currently
be satisfied by an open source choice, if it was considered at all?

I will also make a side note: it was claimed earlier in this thread that
the "mobile app" requirement "came from Fedora", but there is a rather
interesting discrepancy there. The Fedora requirement reads:

"As a Fedora contributor, I can perform issue and pull request actions
on mobile devices via a native mobile app or a mobile-friendly webapp so
that I can contribute while away from my desk."

The "summarized" version reads:

"As a General User
I want a mobile native app
To allow me contribute while away from my desk"

somehow the "or a mobile-friendly webapp" bit got lost. (And the
specific actions).
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net