Fedora Review Service

[Jakub Kadlcik]

Hello folks,

for a couple of years now, I've been interested in the Fedora package
review process. Our queue is in hundreds of waiting packages for as
long as I can remember. I believe the situation can be improved.

Since summer, I did ~40 package reviews to get a better grasp of the
situation and realized what I want to do.

I started working on a service [2] that listens to fedora-messaging
and for every new RHBZ review ticket or a new comment with updated
packages, it submits a build in Copr. Thanks to this [1] feature, Copr
automatically runs the fedora-review tool and generates the
review.txt file. Once the build is finished, my service gets the
message and generates a helpful comment (so far only to STDOUT).

**Unless there is general disapproval, I am planning to let it post
the comments to Bugzilla.**

So far the benefit is limited but, it still can immediately tell the
contributor that their package is broken and fails to build, and also
saves a reviewer the time of running the fedora-review tool
manually. However, I also implemented support for the fedora-review
tool to generate a JSON output next to the standard review.txt. The PR
is still pending [2], please review it if you can. Once this is
released, I can parse its contents and generate much helpful comments
for the contributors.

The end goal is to let contributors go back and forth against the CI
to fix the most obvious mistakes and then let the reviewers take
only the final look.

Hopefully, it will be a better experience for everyone.

[1] http://frostyx.cz/posts/running-fedora-review-after-copr-build
[2] https://github.com/FrostyX/fedora-review-service
[3] https://pagure.io/FedoraReview/pull-request/463

Jakub

[Iñaki Ucar]

On Fri, 2 Dec 2022 at 18:21, Jakub Kadlcik  wrote:
> Hello folks,
>
> for a couple of years now, I've been interested in the Fedora package
> review process. Our queue is in hundreds of waiting packages for as
> long as I can remember. I believe the situation can be improved.
>
> Since summer, I did ~40 package reviews to get a better grasp of the
> situation and realized what I want to do.
>
> I started working on a service [2] that listens to fedora-messaging
> and for every new RHBZ review ticket or a new comment with updated
> packages, it submits a build in Copr. Thanks to this [1] feature, Copr
> automatically runs the fedora-review tool and generates the
> review.txt file. Once the build is finished, my service gets the
> message and generates a helpful comment (so far only to STDOUT).
>
> **Unless there is general disapproval, I am planning to let it post
> the comments to Bugzilla.**

Yes, please! This seems like a **very** helpful service to me. Thanks
for this! :)

Iñaki
> So far the benefit is limited but, it still can immediately tell the
> contributor that their package is broken and fails to build, and also
> saves a reviewer the time of running the fedora-review tool
> manually. However, I also implemented support for the fedora-review
> tool to generate a JSON output next to the standard review.txt. The PR
> is still pending [2], please review it if you can. Once this is
> released, I can parse its contents and generate much helpful comments
> for the contributors.
>
> The end goal is to let contributors go back and forth against the CI
> to fix the most obvious mistakes and then let the reviewers take
> only the final look.
>
> Hopefully, it will be a better experience for everyone.
>
> [1] http://frostyx.cz/posts/running-fedora-review-after-copr-build
> [2] https://github.com/FrostyX/fedora-review-service
> [3] https://pagure.io/FedoraReview/pull-request/463
>
> Jakub
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Adam Williamson]

On Fri, 2022-12-02 at 18:20 +0100, Jakub Kadlcik wrote:
> Hello folks,
>
> for a couple of years now, I've been interested in the Fedora package
> review process. Our queue is in hundreds of waiting packages for as
> long as I can remember. I believe the situation can be improved.
>
> Since summer, I did ~40 package reviews to get a better grasp of the
> situation and realized what I want to do.
>
> I started working on a service [2] that listens to fedora-messaging
> and for every new RHBZ review ticket or a new comment with updated
> packages, it submits a build in Copr. Thanks to this [1] feature, Copr
> automatically runs the fedora-review tool and generates the
> review.txt file. Once the build is finished, my service gets the
> message and generates a helpful comment (so far only to STDOUT).
>
> **Unless there is general disapproval, I am planning to let it post
> the comments to Bugzilla.**

This sounds great! Thanks for working on it.

Where does this service currently run? If it's going to be used "in
anger", perhaps it would be good to put it under fedora-infra control?

[Jakub Kadlcik]

Hello,
> Where does this service currently run? If it's going to be used "in
> anger", perhaps it would be good to put it under fedora-infra control?

Currently running on my laptop. Unpackaged and with hardcoded configuration :-)
But I agree with you. I will soon create a Fedora infra issue and
discuss their requirements and deployment options.

Jakub

On Fri, Dec 2, 2022 at 6:49 PM Adam Williamson
 wrote:
> On Fri, 2022-12-02 at 18:20 +0100, Jakub Kadlcik wrote:
> > Hello folks,
> >
> > for a couple of years now, I've been interested in the Fedora package
> > review process. Our queue is in hundreds of waiting packages for as
> > long as I can remember. I believe the situation can be improved.
> >
> > Since summer, I did ~40 package reviews to get a better grasp of the
> > situation and realized what I want to do.
> >
> > I started working on a service [2] that listens to fedora-messaging
> > and for every new RHBZ review ticket or a new comment with updated
> > packages, it submits a build in Copr. Thanks to this [1] feature, Copr
> > automatically runs the fedora-review tool and generates the
> > review.txt file. Once the build is finished, my service gets the
> > message and generates a helpful comment (so far only to STDOUT).
> >
> > **Unless there is general disapproval, I am planning to let it post
> > the comments to Bugzilla.**
>
> This sounds great! Thanks for working on it.
>
> Where does this service currently run? If it's going to be used "in
> anger", perhaps it would be good to put it under fedora-infra control?
> --
> Adam Williamson
> Fedora QA
> IRC: adamw | Twitter: adamw_ha
> https://www.happyassassin.net
>
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Vitaly Zaitsev]

On 02/12/2022 18:20, Jakub Kadlcik wrote:
> I started working on a service [2] that listens to fedora-messaging
> and for every new RHBZ review ticket or a new comment with updated
> packages, it submits a build in Copr. Thanks to this [1] feature, Copr
> automatically runs the fedora-review tool and generates the
> review.txt file. Once the build is finished, my service gets the
> message and generates a helpful comment (so far only to STDOUT).

fedora-review require a lot of manual checks from reviewer (especially 
licenses). It can't be fully automated.

But posting this as a template for the reviewer is good idea. +1.

[Jakub Kadlcik]

> fedora-review require a lot of manual checks from reviewer (especially
> licenses). It can't be fully automated.

That's true but my goal isn't to fully automate the process. A person
will always be required to take the final look and make sure the spec
is reasonably written and the package is acceptable.

I am looking at it the other way around. I did many reviews where I
run fedora-review and it printed an issues section and/or some [!]
checks, so I basically just copy-pasted them into the RHBZ as a
comment and provided some context about them for the contributor. The
problem is, that the contributor needed to wait days/weeks/months for
feedback that CI could have provided immediately.

My hope is also that by saving reviewers the time to point out the
easy mistakes, we will have time to focus only on the things that, as
you said, can't be automatized, and therefore cover more tickets as a
consequence.

Jakub

On Sat, Dec 3, 2022 at 12:25 PM Vitaly Zaitsev via devel
 wrote:
> On 02/12/2022 18:20, Jakub Kadlcik wrote:
> > I started working on a service [2] that listens to fedora-messaging
> > and for every new RHBZ review ticket or a new comment with updated
> > packages, it submits a build in Copr. Thanks to this [1] feature, Copr
> > automatically runs the fedora-review tool and generates the
> > review.txt file. Once the build is finished, my service gets the
> > message and generates a helpful comment (so far only to STDOUT).
>
> fedora-review require a lot of manual checks from reviewer (especially
> licenses). It can't be fully automated.
>
> But posting this as a template for the reviewer is good idea. +1.
>
> --
> Sincerely,
>    Vitaly Zaitsev ([e-mail])
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Jakub Kadlcik]

I created the Fedora Infra ticket
https://pagure.io/fedora-infrastructure/issue/11028

Jakub

On Sat, Dec 3, 2022 at 1:00 PM Jakub Kadlcik  wrote:
> > fedora-review require a lot of manual checks from reviewer (especially
> > licenses). It can't be fully automated.
>
> That's true but my goal isn't to fully automate the process. A person
> will always be required to take the final look and make sure the spec
> is reasonably written and the package is acceptable.
>
> I am looking at it the other way around. I did many reviews where I
> run fedora-review and it printed an issues section and/or some [!]
> checks, so I basically just copy-pasted them into the RHBZ as a
> comment and provided some context about them for the contributor. The
> problem is, that the contributor needed to wait days/weeks/months for
> feedback that CI could have provided immediately.
>
> My hope is also that by saving reviewers the time to point out the
> easy mistakes, we will have time to focus only on the things that, as
> you said, can't be automatized, and therefore cover more tickets as a
> consequence.
>
> Jakub
>
> On Sat, Dec 3, 2022 at 12:25 PM Vitaly Zaitsev via devel
>  wrote:
> > On 02/12/2022 18:20, Jakub Kadlcik wrote:
> > > I started working on a service [2] that listens to fedora-messaging
> > > and for every new RHBZ review ticket or a new comment with updated
> > > packages, it submits a build in Copr. Thanks to this [1] feature, Copr
> > > automatically runs the fedora-review tool and generates the
> > > review.txt file. Once the build is finished, my service gets the
> > > message and generates a helpful comment (so far only to STDOUT).
> >
> > fedora-review require a lot of manual checks from reviewer (especially
> > licenses). It can't be fully automated.
> >
> > But posting this as a template for the reviewer is good idea. +1.
> >
> > --
> > Sincerely,
> >    Vitaly Zaitsev ([e-mail])
> > _______________________________________________
> > devel mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
> > Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

[Miro Hrončok]

On 02. 12. 22 18:20, Jakub Kadlcik wrote:
> Hello folks,
>
> for a couple of years now, I've been interested in the Fedora package
> review process. Our queue is in hundreds of waiting packages for as
> long as I can remember. I believe the situation can be improved.
>
> Since summer, I did ~40 package reviews to get a better grasp of the
> situation and realized what I want to do.
>
> I started working on a service [2] that listens to fedora-messaging
> and for every new RHBZ review ticket or a new comment with updated
> packages, it submits a build in Copr. Thanks to this [1] feature, Copr
> automatically runs the fedora-review tool and generates the
> review.txt file. Once the build is finished, my service gets the
> message and generates a helpful comment (so far only to STDOUT).
>
> **Unless there is general disapproval, I am planning to let it post
> the comments to Bugzilla.**
>
> So far the benefit is limited but, it still can immediately tell the
> contributor that their package is broken and fails to build, and also
> saves a reviewer the time of running the fedora-review tool
> manually. However, I also implemented support for the fedora-review
> tool to generate a JSON output next to the standard review.txt. The PR
> is still pending [2], please review it if you can. Once this is
> released, I can parse its contents and generate much helpful comments
> for the contributors.
>
> The end goal is to let contributors go back and forth against the CI
> to fix the most obvious mistakes and then let the reviewers take
> only the final look.
>
> Hopefully, it will be a better experience for everyone.
>
> [1] http://frostyx.cz/posts/running-fedora-review-after-copr-build
> [2] https://github.com/FrostyX/fedora-review-service
> [3] https://pagure.io/FedoraReview/pull-request/463

Jakub, could this service be misbehaving?

See https://bugzilla.redhat.com/show_bug.cgi?id=2133080#c19

[Frank Crawford]

On Wed, 2023-01-04 at 23:41 +0100, Miro Hrončok wrote:
> On 02. 12. 22 18:20, Jakub Kadlcik wrote:
> > Hello folks,
> >
> > for a couple of years now, I've been interested in the Fedora
> > package
> > review process. Our queue is in hundreds of waiting packages for as
> > long as I can remember. I believe the situation can be improved.
> >
> > Since summer, I did ~40 package reviews to get a better grasp of
> > the
> > situation and realized what I want to do.
> >
> > I started working on a service [2] that listens to fedora-messaging
> > and for every new RHBZ review ticket or a new comment with updated
> > packages, it submits a build in Copr. Thanks to this [1] feature,
> > Copr
> > automatically runs the fedora-review tool and generates the
> > review.txt file. Once the build is finished, my service gets the
> > message and generates a helpful comment (so far only to STDOUT).
> >
> > **Unless there is general disapproval, I am planning to let it post
> > the comments to Bugzilla.**
> >
> > So far the benefit is limited but, it still can immediately tell
> > the
> > contributor that their package is broken and fails to build, and
> > also
> > saves a reviewer the time of running the fedora-review tool
> > manually. However, I also implemented support for the fedora-review
> > tool to generate a JSON output next to the standard review.txt. The
> > PR
> > is still pending [2], please review it if you can. Once this is
> > released, I can parse its contents and generate much helpful
> > comments
> > for the contributors.
> >
> > The end goal is to let contributors go back and forth against the
> > CI
> > to fix the most obvious mistakes and then let the reviewers take
> > only the final look.
> >
> > Hopefully, it will be a better experience for everyone.
> >
> > [1] http://frostyx.cz/posts/running-fedora-review-after-copr-build
> > [2] https://github.com/FrostyX/fedora-review-service
> > [3] https://pagure.io/FedoraReview/pull-request/463
>
> Jakub, could this service be misbehaving?
>
> See https://bugzilla.redhat.com/show_bug.cgi?id=2133080#c19

We also saw the same sort of thing yesterday on an update to a review
request:

https://bugzilla.redhat.com/show_bug.cgi?id=2119983
> -- 
> Miro Hrončok

Regards
Frank

[Jakub Kadlcik]

Hello guys,
I just wanted to announce that the service is already deployed and
that I am trying to figure out all the quirks and ask you to please be
patient.

But you were faster :-)
Thank you for the reports, please ping me with anything more that you
find. I am also trying to monitor what is happening.

The issue that Miro found
https://bugzilla.redhat.com/show_bug.cgi?id=2133080#c19
was caused by an URL-encoded caret symbol in the package name, which
is IMHO a Copr bug. I already sent a PR
https://github.com/fedora-copr/copr/pull/2454

The issue that Frank found
https://bugzilla.redhat.com/show_bug.cgi?id=2119983
I am not sure, looks like a temporary unavailability of the URL
because when I resubmitted the package, it built successfully. Not
sure what should be done about this, I will probably wait if this
happens again.

Jakub

On Thu, Jan 5, 2023 at 1:45 AM Frank Crawford  wrote:
> On Wed, 2023-01-04 at 23:41 +0100, Miro Hrončok wrote:
> > On 02. 12. 22 18:20, Jakub Kadlcik wrote:
> > > Hello folks,
> > >
> > > for a couple of years now, I've been interested in the Fedora
> > > package
> > > review process. Our queue is in hundreds of waiting packages for as
> > > long as I can remember. I believe the situation can be improved.
> > >
> > > Since summer, I did ~40 package reviews to get a better grasp of
> > > the
> > > situation and realized what I want to do.
> > >
> > > I started working on a service [2] that listens to fedora-messaging
> > > and for every new RHBZ review ticket or a new comment with updated
> > > packages, it submits a build in Copr. Thanks to this [1] feature,
> > > Copr
> > > automatically runs the fedora-review tool and generates the
> > > review.txt file. Once the build is finished, my service gets the
> > > message and generates a helpful comment (so far only to STDOUT).
> > >
> > > **Unless there is general disapproval, I am planning to let it post
> > > the comments to Bugzilla.**
> > >
> > > So far the benefit is limited but, it still can immediately tell
> > > the
> > > contributor that their package is broken and fails to build, and
> > > also
> > > saves a reviewer the time of running the fedora-review tool
> > > manually. However, I also implemented support for the fedora-review
> > > tool to generate a JSON output next to the standard review.txt. The
> > > PR
> > > is still pending [2], please review it if you can. Once this is
> > > released, I can parse its contents and generate much helpful
> > > comments
> > > for the contributors.
> > >
> > > The end goal is to let contributors go back and forth against the
> > > CI
> > > to fix the most obvious mistakes and then let the reviewers take
> > > only the final look.
> > >
> > > Hopefully, it will be a better experience for everyone.
> > >
> > > [1] http://frostyx.cz/posts/running-fedora-review-after-copr-build
> > > [2] https://github.com/FrostyX/fedora-review-service
> > > [3] https://pagure.io/FedoraReview/pull-request/463
> >
> > Jakub, could this service be misbehaving?
> >
> > See https://bugzilla.redhat.com/show_bug.cgi?id=2133080#c19
>
> We also saw the same sort of thing yesterday on an update to a review
> request:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=2119983
> > --
> > Miro Hrončok
>
> Regards
> Frank