Automating Package Review (Was: fedora-review -- do we have a maintainer?)

[Stephen Gallagher]

On Thu, Aug 16, 2018 at 8:30 AM Michal Novotny  wrote:
> On Thu, Aug 16, 2018 at 10:49 AM Zbigniew Jędrzejewski-Szmek <
> [e-mail]> wrote:
> > f-r currently fails to build (#1603956), it has a bunch of bugs open [1]
> > and many issues and unhandled pull requests in the upstream repo [2, 3].
> > The last upstream commit was 2 years ago.
> >
> > f-r has is annoyingly outdated and gives often outright bad advice
> > (for example about BR:gcc or BR:g++). The situation would be
> > significantly
> > improved if the outstanding PRs were merged.
> >
> > f-r is also python2-only now, which will be a problem soon since
> > support for python2 is waning [4].
> >
> > Is there any hope of upstream and downstream activity on f-r?
>
> I was thinking about getting the fedora-review checks rewritten into the
> standard Test interface
> (
> https://qa.fedoraproject.org/docs/libtaskotron/latest/standard-test-interfac...)
> so that they
> can be run in Taskotron. We can also just probably run one big
> fedora-review check from
> a taskotron test, well, this just came to my mind recently, getting the
> actual solution ready
> might take a little bit of time.
>  '

I'd *really* like to see us get to a point where package review is
fully-automated. Basically we could just have a web-service that you pass a
URL to an SRPM plus authenticate with your FAS account and it will perform
all of the validity checks and if they all pass would go ahead and request
the branches for you and import the SRPM.

Once this is fully automated, we can then *also* add the same checks to CI
(taskotron, OSCI or whatever) so that on each build it gets rerun, which
will allow us to help reduce the rate of packages falling out of compliance
(as well as being updated whenever the checks get made more comprehensive).

Historically, we've had human review mainly to protect against two things,
bundling and unacceptable licenses. In both of these cases, I'd like for us
to move towards a culture of assuming goodwill on behalf of our packagers.
Most of the packagers in Fedora have been doing it for a long time and know
what is and is not acceptable. Optimizing for the minority case is
wasteful, especially when it adds hurdles and delays to getting software
delivered.

I think what we should instead do is allow things through immediately
following automated review and just assume that those few cases that slip
through that should not will get handled after the fact as soon as they are
noticed (either by someone noticing or an improvement in the automated tool
discovering the problem).

I feel strongly that automated, continuous review would be of far greater
value to Fedora than front-loading the review process the way we have been
doing (which serves mostly to discourage people from even starting).

[Neal Gompa]

On Thu, Aug 16, 2018 at 5:04 PM Stephen Gallagher  wrote:
> On Thu, Aug 16, 2018 at 8:30 AM Michal Novotny  wrote:
> > On Thu, Aug 16, 2018 at 10:49 AM Zbigniew Jędrzejewski-Szmek  wrote:
> > > f-r currently fails to build (#1603956), it has a bunch of bugs open [1]
> > > and many issues and unhandled pull requests in the upstream repo [2, 3].
> > > The last upstream commit was 2 years ago.
> > >
> > > f-r has is annoyingly outdated and gives often outright bad advice
> > > (for example about BR:gcc or BR:g++). The situation would be significantly
> > > improved if the outstanding PRs were merged.
> > >
> > > f-r is also python2-only now, which will be a problem soon since
> > > support for python2 is waning [4].
> > >
> > > Is there any hope of upstream and downstream activity on f-r?
> >
> > I was thinking about getting the fedora-review checks rewritten into the standard Test interface
> > (https://qa.fedoraproject.org/docs/libtaskotron/latest/standard-test-interfac...) so that they
> > can be run in Taskotron. We can also just probably run one big fedora-review check from
> > a taskotron test, well, this just came to my mind recently, getting the actual solution ready
> > might take a little bit of time.
> >  '
>
> I'd *really* like to see us get to a point where package review is fully-automated. Basically we could just have a web-service that you pass a URL to an SRPM plus authenticate with your FAS account and it will perform all of the validity checks and if they all pass would go ahead and request the branches for you and import the SRPM.
>
> Once this is fully automated, we can then *also* add the same checks to CI (taskotron, OSCI or whatever) so that on each build it gets rerun, which will allow us to help reduce the rate of packages falling out of compliance (as well as being updated whenever the checks get made more comprehensive).
>
> Historically, we've had human review mainly to protect against two things, bundling and unacceptable licenses. In both of these cases, I'd like for us to move towards a culture of assuming goodwill on behalf of our packagers. Most of the packagers in Fedora have been doing it for a long time and know what is and is not acceptable. Optimizing for the minority case is wasteful, especially when it adds hurdles and delays to getting software delivered.
>
> I think what we should instead do is allow things through immediately following automated review and just assume that those few cases that slip through that should not will get handled after the fact as soon as they are noticed (either by someone noticing or an improvement in the automated tool discovering the problem).
>
> I feel strongly that automated, continuous review would be of far greater value to Fedora than front-loading the review process the way we have been doing (which serves mostly to discourage people from even starting).

I fully agree with this, which is why Tom (Cc'd to this email) and I
have been sketching out a plan to start moving towards this.

It won't be particularly easy, but we're looking at a step-by-step
approach to get there. However, if more people are interested in
contributing to make the end-goal possible, we might be able to get
there more quickly.

[Michal Novotny]

On Thu, Aug 16, 2018 at 11:09 PM Neal Gompa  wrote:
> On Thu, Aug 16, 2018 at 5:04 PM Stephen Gallagher 
> wrote:
> > On Thu, Aug 16, 2018 at 8:30 AM Michal Novotny  wrote:
> > > On Thu, Aug 16, 2018 at 10:49 AM Zbigniew Jędrzejewski-Szmek <
>
> [e-mail]> wrote:
> > > > f-r currently fails to build (#1603956), it has a bunch of bugs open
>
> [1]
> > > > and many issues and unhandled pull requests in the upstream repo [2,
>
> 3].
> > > > The last upstream commit was 2 years ago.
> > > >
> > > > f-r has is annoyingly outdated and gives often outright bad advice
> > > > (for example about BR:gcc or BR:g++). The situation would be
>
> significantly
> > > > improved if the outstanding PRs were merged.
> > > >
> > > > f-r is also python2-only now, which will be a problem soon since
> > > > support for python2 is waning [4].
> > > >
> > > > Is there any hope of upstream and downstream activity on f-r?
> > >
> > > I was thinking about getting the fedora-review checks rewritten into
>
> the standard Test interface
> > > (
>
> https://qa.fedoraproject.org/docs/libtaskotron/latest/standard-test-interfac...)
> so that they
> > > can be run in Taskotron. We can also just probably run one big
>
> fedora-review check from
> > > a taskotron test, well, this just came to my mind recently, getting the
>
> actual solution ready
> > > might take a little bit of time.
> > >  '
> >
> > I'd *really* like to see us get to a point where package review is
>
> fully-automated. Basically we could just have a web-service that you pass a
> URL to an SRPM plus authenticate with your FAS account and it will perform
> all of the validity checks and if they all pass would go ahead and request
> the branches for you and import the SRPM.
> > Once this is fully automated, we can then *also* add the same checks to
>
> CI (taskotron, OSCI or whatever) so that on each build it gets rerun, which
> will allow us to help reduce the rate of packages falling out of compliance
> (as well as being updated whenever the checks get made more comprehensive).
> > Historically, we've had human review mainly to protect against two
>
> things, bundling and unacceptable licenses. In both of these cases, I'd
> like for us to move towards a culture of assuming goodwill on behalf of our
> packagers. Most of the packagers in Fedora have been doing it for a long
> time and know what is and is not acceptable. Optimizing for the minority
> case is wasteful, especially when it adds hurdles and delays to getting
> software delivered.
> > I think what we should instead do is allow things through immediately
>
> following automated review and just assume that those few cases that slip
> through that should not will get handled after the fact as soon as they are
> noticed (either by someone noticing or an improvement in the automated tool
> discovering the problem).
> > I feel strongly that automated, continuous review would be of far
>
> greater value to Fedora than front-loading the review process the way we
> have been doing (which serves mostly to discourage people from even
> starting).
>
> I fully agree with this, which is why Tom (Cc'd to this email) and I
> have been sketching out a plan to start moving towards this.
>
> It won't be particularly easy, but we're looking at a step-by-step
> approach to get there. However, if more people are interested in
> contributing to make the end-goal possible, we might be able to get
> there more quickly.

Copr team is willing to help.

I think my colleagues will agree with me.
clime
> --
> 真実はいつも一つ！/ Always, there's only one truth!
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]/...

[Ben Rosser]

On Thu, Aug 16, 2018 at 4:09 PM, Stephen Gallagher  wrote:
> I'd *really* like to see us get to a point where package review is
> fully-automated. Basically we could just have a web-service that you pass a
> URL to an SRPM plus authenticate with your FAS account and it will perform
> all of the validity checks and if they all pass would go ahead and request
> the branches for you and import the SRPM.
>
> Once this is fully automated, we can then *also* add the same checks to CI
> (taskotron, OSCI or whatever) so that on each build it gets rerun, which
> will allow us to help reduce the rate of packages falling out of compliance
> (as well as being updated whenever the checks get made more comprehensive).
>
> Historically, we've had human review mainly to protect against two things,
> bundling and unacceptable licenses. In both of these cases, I'd like for us
> to move towards a culture of assuming goodwill on behalf of our packagers.
> Most of the packagers in Fedora have been doing it for a long time and know
> what is and is not acceptable. Optimizing for the minority case is wasteful,
> especially when it adds hurdles and delays to getting software delivered.

Also (at least in my experience), generally licensing issues get
caught by a human inspecting the output of "licensecheck", which
fedora-review currently runs automatically anyway. If the automated
review process did this and showed the results to the packager, I bet
we would catch a lot of the licensing/bundling problems.

Anyway, I really like this idea. Maybe we should still require
quasi-manual reviews for new contributors as part of the sponsorship
process, though?

Ben Rosser

[Richard W.M. Jones]

While I agree that this is a good idea, I have one note of caution:
What's to stop someone adding a malicious package which did something
like ‘Provides: glibc’ and subsequently infects everyone's machine?
I think we'd want to consider the security implications of accepting
packages after only automated review.

Rich.

[Fabio Valentini]

On Fri, Aug 17, 2018, 20:53 Richard W.M. Jones  wrote:
> While I agree that this is a good idea, I have one note of caution:
> What's to stop someone adding a malicious package which did something
> like ‘Provides: glibc’ and subsequently infects everyone's machine?
> I think we'd want to consider the security implications of accepting
> packages after only automated review.

I agree. I think a pair of human eyes will have to look at package
submissions at least until we have a sufficiently advanced FPC AI to do it
;)

However, I think using automated checks for existing packages would be a
nice thing (although fedora-review isn't suited to do that right now, and
is out of sync with current guidelines).

Fabio
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat
> http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-builder quickly builds VMs from scratch
> http://libguestfs.org/virt-builder.1.html
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]/...

Automating Package Review

[Jason L Tibbitts III]

> > > > > "RWMJ" == Richard W M Jones  writes:

RWMJ> I think we'd want to consider the security implications of
RWMJ> accepting packages after only automated review.

New packagers would need to be sponsored and so a human would still have
to be involved at some point.  (At least I certainly hope that nobody is
thinking of removing that step.)  Existing packagers could just push a
malicious change to one of the packages they already maintain; that's
not a change from the current system.

Now, it happens that I strongly disagree with the idea that we should
let anything more than a narrowly defined set of new packages into the
distribution without human review.  I just don't think the security
argument is one I'd use personally to argue my position.

- J<

[Stephen Gallagher]

On Fri, Aug 17, 2018 at 2:08 PM Richard W.M. Jones 
wrote:
> While I agree that this is a good idea, I have one note of caution:
> What's to stop someone adding a malicious package which did something
> like ‘Provides: glibc’ and subsequently infects everyone's machine?
> I think we'd want to consider the security implications of accepting
> packages after only automated review.

Literally nothing prevents a packager from doing this *today*. As soon as
package-review is complete and the dist-git repo is created, the packager
can make whatever changes they want and push it with impunity.

Let’s be wary of the Nirvana Fallacy while discussing this: a perfect
solution doesn’t need to be found before implementing one that improves on
the current state.

That being said, it wouldn’t be particularly difficult for the review
script to run `dnf repoquery --whatprovides` for everything this new
package provides and fail if it replaces something else without Obsoletes.
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat
> http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-builder quickly builds VMs from scratch
> http://libguestfs.org/virt-builder.1.html
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]/...

[Sérgio Basto]

Hi,
Any news ? 
"But I guess nothing's getting released, for some reason? fedora-review 
has been on version 0.6.1 since May 2016; all package activity since
then has been housekeeping rebuilds. "
may you add me as admin to Fedora-review package ? to release a new
version . 
Thanks
On Sat, 2018-08-18 at 06:12 -0400, Stephen Gallagher wrote:
> On Fri, Aug 17, 2018 at 2:08 PM Richard W.M. Jones <[e-mail]
> > wrote:
> > While I agree that this is a good idea, I have one note of caution:
> >
> > What's to stop someone adding a malicious package which did
> > something
> >
> > like ‘Provides: glibc’ and subsequently infects everyone's machine?
> >
> > I think we'd want to consider the security implications of
> > accepting
> >
> > packages after only automated review.
>
> Literally nothing prevents a packager from doing this *today*. As
> soon as package-review is complete and the dist-git repo is created,
> the packager can make whatever changes they want and push it with
> impunity.
> Let’s be wary of the Nirvana Fallacy while discussing this: a perfect
> solution doesn’t need to be found before implementing one that
> improves on the current state.
> That being said, it wouldn’t be particularly difficult for the review
> script to run `dnf repoquery --whatprovides` for everything this new
> package provides and fail if it replaces something else without
> Obsoletes.
> > Rich.
> >
> > devel mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidel
> > ines
> > List Archives: https://lists.fedoraproject.org/archives/list/devel@
> > lists.fedoraproject.org/message/CWZEBZ5ND23U4TKAG3L3Z37CYSV6GQAY/

[Neal Gompa]

On Tue, Dec 11, 2018 at 10:30 AM Sérgio Basto  wrote:
> Hi,
>
> Any news ?
>
> "But I guess nothing's getting released, for some reason? fedora-review has been on version 0.6.1 since May 2016; all package activity since then has been housekeeping rebuilds. "
>
> may you add me as admin to Fedora-review package ? to release a new version .

There's really one remaining thing for a new release of FedoraReview:
porting to Python 3. There's a WIP PR here:
https://pagure.io/FedoraReview/pull-request/312

If it doesn't budge this week, I'm hoping to take a crack at it in the
next week or so and try to pull it over the finish line.

[Sérgio Basto]

On Tue, 2018-12-11 at 16:36 -0500, Neal Gompa wrote:
> On Tue, Dec 11, 2018 at 10:30 AM Sérgio Basto 
> wrote:
> > Hi,
> >
> > Any news ?
> >
> > "But I guess nothing's getting released, for some reason? fedora-
> > review has been on version 0.6.1 since May 2016; all package
> > activity since then has been housekeeping rebuilds. "
> >
> > may you add me as admin to Fedora-review package ? to release a new
> > version .
>
> There's really one remaining thing for a new release of FedoraReview:
> porting to Python 3. There's a WIP PR here:
> https://pagure.io/FedoraReview/pull-request/312
>
> If it doesn't budge this week, I'm hoping to take a crack at it in
> the
> next week or so and try to pull it over the finish line.

Hi, Neal Gompa

I also would like be admin in https://pagure.io/FedoraReview , can youadd me ? please.

We have lots of pull request to review . Version 0.6.1 is not tagged ,
in resume lots of work to do .

Thanks.
> -- 
> 真実はいつも一つ！/ Always, there's only one truth!
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelin
> es
> List Archives: https://lists.fedoraproject.org/archives/list/devel@li
> sts.fedoraproject.org

[Sérgio Basto]

Hi, (sorry for duplicates I sent from wrong email before)

Nothing happened last week .

Can you add me to https://pagure.io/FedoraReview/ and to
https://src.fedoraproject.org/rpms/fedora-review please .

My fas user is sergiomb , people want revert mock configurations of
RPMFusion because is not working with current release , we have a non
functional fedora-review in repos , so IMHO this is the most urgent
task to do .

Thanks

On Tue, 2018-12-11 at 16:36 -0500, Neal Gompa wrote:
> On Tue, Dec 11, 2018 at 10:30 AM Sérgio Basto <
> [e-mail]> wrote:
> > Hi,
> >
> > Any news ?
> >
> > "But I guess nothing's getting released, for some reason? fedora-
> > review has been on version 0.6.1 since May 2016; all package
> > activity since then has been housekeeping rebuilds. "
> >
> > may you add me as admin to Fedora-review package ? to release a new
> > version .
>
> There's really one remaining thing for a new release of FedoraReview:
> porting to Python 3. There's a WIP PR here:
> https://pagure.io/FedoraReview/pull-request/312
>
> If it doesn't budge this week, I'm hoping to take a crack at it in
> the
> next week or so and try to pull it over the finish line.
>
> -- 
> 真実はいつも一つ！/ Always, there's only one truth!
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: 
> https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:

https://lists.fedoraproject.org/archives/list/[e-mail]

[Neal Gompa]

On Tue, Dec 18, 2018 at 3:10 PM Sérgio Basto  wrote:
> Hi, (sorry for duplicates I sent from wrong email before)
>
> Nothing happened last week .
>
> Can you add me to https://pagure.io/FedoraReview/ and to
> https://src.fedoraproject.org/rpms/fedora-review please .
>
> My fas user is sergiomb , people want revert mock configurations of
> RPMFusion because is not working with current release , we have a non
> functional fedora-review in repos , so IMHO this is the most urgent
> task to do .

It doesn't matter at the moment. Currently I can't merge *any* PRs in
fedora-review, due to a bug in Pagure[1].

I've already got three PRs slated for merge, and once those are out,
I'll make a release.

[1]: https://pagure.io/pagure/issue/4142

[Sérgio Basto]

On Tue, 2018-12-18 at 15:16 -0500, Neal Gompa wrote:
> On Tue, Dec 18, 2018 at 3:10 PM Sérgio Basto 
> wrote:
> > Hi, (sorry for duplicates I sent from wrong email before)
> >
> > Nothing happened last week .
> >
> > Can you add me to https://pagure.io/FedoraReview/ and to
> > https://src.fedoraproject.org/rpms/fedora-review please .
> >
> > My fas user is sergiomb , people want revert mock configurations of
> > RPMFusion because is not working with current release , we have a
> > non
> > functional fedora-review in repos , so IMHO this is the most urgent
> > task to do .
>
> It doesn't matter at the moment. Currently I can't merge *any* PRs in
> fedora-review, due to a bug in Pagure[1].
>
> I've already got three PRs slated for merge, and once those are out,
> I'll make a release.

Friend let me do the work, for that I need acls .

Thanks

[Jeff Johnson]

To answer your question solely because I don't like FUD driven phears monger int discussions

RPM based depsolvers select packages based on heuristics, including what is already installed.

Any malicious package that had Provides: glibc would most likely be ignored because glibc is already installed.

Automating Package Review

[Nicolas Mailhot]

Le 2018-08-16 22:09, Stephen Gallagher a écrit :
> I'd *really* like to see us get to a point where package review is
> fully-automated. Basically we could just have a web-service that you
> pass a URL to an SRPM plus authenticate with your FAS account and it
> will perform all of the validity checks and if they all pass would go
> ahead and request the branches for you and import the SRPM.

Please oh please make it an SRPM set. Broadband and git made it so easy 
to reuse other software projects, software is increasingly spread over 
many many components. So old-style "review srpm in isolation, because it 
uses at most a dozen other first-level projects, that are probably 
already packaged" does not work anymore.

Better guidelines and rpm tooling made it easy to create unbundled srpm 
sets, but review didn't evolve and is a painful choke point.

So one of the main reasons for Fedora manual reviews (catching bundling) 
is also one of the main reasons people push for bundling exemptions 
(manual review of unbundled packages does not scale). The current 
process is actively working against its stated objectives.

Regards,