On Tue, Aug 25, 2020 at 7:25 PM Dusty Mabe <dusty@dustymabe.com> wrote:

On 8/21/20 8:06 AM, Kamil Paral wrote:
> Hello folks,
>

Hey Kamil!

Hey Dusty, sorry for a late reply.

> wanted to thank you all for supporting the FCOS Test Day which turned out to be a very successful event! As we looked back, we thought there might be room for more community participation in FCOS QA. On those lines, we were wondering if there are good ways for other Fedora teams and for the community in large to get a good picture of how quality validation and decisions are done for the FCOS stable stream, and how people who are interested can participate in this process.
>
> In particular, I tried to find some documentation on:
> * The pool of bugs which are currently reported against the testing and next stream
> * The subset of aforementioned bugs which were decided to be "blockers" and must be fixed before the stream can be promoted to stable
> * Where this blocker decision-making takes place and how people can get involved
> * What are the procedures and rules for this blocker decision-making process
>
> Unfortunately, I was not able to find any. I might be looking at a wrong location. Can you please point me to the correct place, if such documentation exists?

Our high level bugs mostly exist in the issue tracker. You are right we could use more transparency in our process.

https://github.com/coreos/fedora-coreos-tracker/issues

So if there is an important issue in bugzilla, do you create a copy of that issue in your tracker? Or what do you do?

Do you have some labels or other ways to specify which bugs are blocking the next stable update? How do you track whether the current testing stream is good to go?

>
> Since FCOS might soon be an official Fedora edition, it would be great to have insight into the FCOS community release process. But this is far from just a QA team thing. This process transparency is important for recruiting new passionate community members who can help you run and improve things. Another use case might be a party affected by a bug who might feel motivated to visit such a blocker discussion and argue their case about why they think such a bug should not hit the stable stream.

+1. That's a good point. Obviously our release process happens more often and we don't want to add a lot of
manual "process" around it because then it gets hard to manage, but I think we should improve.

I understand that. But you still need to manually inspect bugs and figure out which are critical ones, right? That's the process I'm talking about. How do you currently decide which ones are critical? During a meeting? Do you have some ways to partially automate this decision (I can imagine you could e.g. pinpoint a build which makes your testsuite fail and automatically create a ticket for it)?

>
> We can put together all the relevant documents in one place that is accessible to the public so that the community is aware of important decisions made during the release process and feel included. I imagine that there could be a "Participate!" section in the FCOS docs page [1] which would point people to areas where they can help FCOS, and one of those areas could be quality control. What do you think?

I think the most organic way people can get this information into the community today is just open a tracker issue
and come to our meeting.

Sounds good. That's simple enough, but serves the purpose well. It just needs to be visibly stated somewhere.

For example some people raised issues with the new version of podman and we're currently
holding it back in FCOS. See the following two issues:

- https://github.com/coreos/fedora-coreos-tracker/issues/560
- https://github.com/coreos/fedora-coreos-tracker/issues/575

>
> As an example, the current quality validation workflow in Fedora includes a regular "blocker review" meeting in a dedicated IRC channel, where bugs proposed as "blockers" get discussed [2][3]. Anyone can propose a bug as a blocker and we regularly receive bug nominations from the community. The blocker review meeting is open to anybody and we try to base our votes on "release criteria" [4] after assessing that particular case and its circumstances. Release criteria help us bring a bit more consistency into this blocker decision-making and it saves us some time from looking up historical evidence ("how did we vote last time on a similar bug?"). All currently proposed and accepted blockers are visible from our custom page [5]. This whole process is a collaborative effort between Release Engineering, Quality Assurance, Development, and Project Management. Please note that while I'm describing our current process as an example, I'm not implying you need to do it in the same way.
> I'm just showing how we do it in general. Also, our documentation is quite complex and lengthy, but essentially it can be just a couple of paragraphs with links, to steer people into the right direction.
>
> Looking forward to hearing your thoughts. Thanks!

The existing process for fedora is well thought out but maybe a little overwhelming for us for right now. Clement, who
recently joined our team, is going to try to help us with this in the future.

Perfect. I'll be happy to give him some advice, if he's interested.

In the mean time, I think the best way to go about making change here would be for some concrete proposals for how we
could implement some baby steps to increase transparency. Kamil, do you have any suggestions there?

Sure. So let's talk about visibility first. If I open your documentation [1] I see a link to your IRC and Discourse and then in the FAQ I found this list of links [2]. Quite interestingly, the link to the fedora-coreos-tracker github is missing. I also haven't found any mention of reporting bugs nor your meetings. I found the tracker mentioned in Getting Started [3], but I would expect it in the FAQ "communicate" section (or I'd expect a follow-up "reporting issues" section). So perhaps the first step could be to make sure the tracker is visibly mentioned among the communication means. As a bonus, you can then avoid duplicating that information in other places (like Getting Started) and just link to that section from everywhere. That "canonical location" doesn't even need to be in the FAQ, but can be in your tracker [4] and the FAQ section can link there as well. It depends how much you want to deduplicate information. It helps to make sure some channels are not forgotten in one place, and other channels in a different place.

A second step could be to create some kind of Participate/Join us section. There you could list ways how people can help improve FCOS and become part of the community. I'm sure it's not just about QA. But regarding QA, there could be things like reporting issues, going through the existing issues and trying to reproduce them on their end, supplying more information/debugging the problems, and also participating in the meetings.

Currently your Meetings section [5] is more of a guide on how to run meetings, and the Voting section below contains rules, but little information on what topics you actually vote. It might help to extend the Meetings section to state that it is open to the public, that you'll be happy to see new faces, and what is the usual agenda on such meetings. Similarly for voting, it could say what topics you usually vote on and that you'll be happy to hear opinions from the public before the vote. You could move some of the technical info ("Steps to run the meeting") into a separate document and link it, to increase readability.

Does that sound reasonable, is it helpful?

Most of this has been about community participation, but there is a bigger picture. For example, understanding your processes and being able to quickly assess the current state of the in-development version of FCOS is very helpful even for QA or release management. It's all linked.

Thanks,

Kamil

[1] https://docs.fedoraproject.org/en-US/fedora-coreos/

[2] https://docs.fedoraproject.org/en-US/fedora-coreos/faq/#_what_are_the_communication_channels_around_fedora_coreos

[3] https://docs.fedoraproject.org/en-US/fedora-coreos/getting-started/#_getting_in_touch

[4] https://github.com/coreos/fedora-coreos-tracker#communication-channels-for-fedora-coreos

[5] https://github.com/coreos/fedora-coreos-tracker#meetings