openQA tests in Pull Requests?

[Miro Hrončok]

Hello QA,

is there a way (even a hacky one) to run openQA tests (the ones that run on 
Bodhi updates of critpath packages) on an open Pull Request?

Thanks,
-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Adam Williamson]

On Tue, 2021-12-07 at 13:21 +0100, Miro Hrončok wrote:
>  Hello QA,
>  
>  is there a way (even a hacky one) to run openQA tests (the ones that run on 
>  Bodhi updates of critpath packages) on an open Pull Request? 
Hey Miro! Right now there really isn't, I'm afraid. The least official
thing we can run the tests for right now is a scratch build, so you
need to at least turn the PR into a scratch build somehow.

There's no public/self-serve way to schedule for a scratch build at the
moment; someone with admin/moderator access (so, usually, me) needs to
do it for you.

If there is substantial interest I could look at implementing something
here...I would expect that some system or other (CI? Koschei?) must
surely have a way of turning pull requests into package builds already,
so I'd want to find that and build on it rather than reinventing it if
possible.

The other stumbling block is that we still don't run these tests on
Rawhide, so if you want to test the PR in the context of the
rawhide/main branch, it would be more difficult (even just doing a one-
off, because the tests use pre-rolled disk images as a base, and we
don't build several of the needed images for Rawhide currently).

That's something that could change, but it would be a fairly major
thing and might possibly require me to beg infra for more hardware or
finally find the time to see if we can run openQA tests In The Cloud. I
can provide more details on that whole area if you're interested...
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

[Miro Hrončok]

On 07. 12. 21 18:21, Adam Williamson wrote:
>  On Tue, 2021-12-07 at 13:21 +0100, Miro Hrončok wrote:
> > Hello QA,
> >
> > is there a way (even a hacky one) to run openQA tests (the ones that run on
> > Bodhi updates of critpath packages) on an open Pull Request?
>  
>  Hey Miro! Right now there really isn't, I'm afraid. The least official
>  thing we can run the tests for right now is a scratch build, so you
>  need to at least turn the PR into a scratch build somehow. 
That we already have. Each pull request has a "Fedora CI - scratch build" job.

>  There's no public/self-serve way to schedule for a scratch build
> at the
>  moment; someone with admin/moderator access (so, usually, me) needs to
>  do it for you. 
Got it. At the moment, I am mostly interested in 
https://src.fedoraproject.org/rpms/python3.10/pull-request/87 but it fails for 
now, will let you know when it is ready.
The reason is to avoid the failure wqe have seen in Fedora 34 recently.

>  If there is substantial interest I could look at implementing
> something
>  here...I would expect that some system or other (CI? Koschei?) must
>  surely have a way of turning pull requests into package builds already,
>  so I'd want to find that and build on it rather than reinventing it if
>  possible. 
Yes, this exists. Both in Zuul and as a "Fedora CI" job. How to retrieve that 
job programmatically, I have no idea. But it is there.

>  The other stumbling block is that we still don't run these tests
> on
>  Rawhide, so if you want to test the PR in the context of the
>  rawhide/main branch, it would be more difficult (even just doing a one-
>  off, because the tests use pre-rolled disk images as a base, and we
>  don't build several of the needed images for Rawhide currently). 
For now, I am interested in Fedora 34/35. Once the tests run in Bodhi for 
Rawhide, we would want them run in PRs as well. The point is that I don't want 
to merge+build+bodhi an update just to find out it fails the tests -- I want to 
know before I merge the PR.

>  That's something that could change, but it would be a fairly
> major
>  thing and might possibly require me to beg infra for more hardware or
>  finally find the time to see if we can run openQA tests In The Cloud. I
>  can provide more details on that whole area if you're interested... 
As much as I'd love to see those tests on Rawhide as well, all I ask for here 
is to have it 1:1 Bodhi updates and Pull Requests.


Thanks,
-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Miro Hrončok]

On 07. 12. 21 19:25, Miro Hrončok wrote:
> > There's no public/self-serve way to schedule for a scratch
> build at the
> > moment; someone with admin/moderator access (so, usually, me) needs to
> > do it for you.
>  
>  Got it. At the moment, I am mostly interested in 
>  https://src.fedoraproject.org/rpms/python3.10/pull-request/87 but it fails for 
>  now, will let you know when it is ready.
>  The reason is to avoid the failure wqe have seen in Fedora 34 recently. 
This is now ready to test. I've pinged you on the PR but I am replying here as 
well just in case.

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Adam Williamson]

On Tue, 2021-12-07 at 19:25 +0100, Miro Hrončok wrote:
>  > If there is substantial interest I could look at implementing
>  > something
>  > here...I would expect that some system or other (CI? Koschei?) must
>  > surely have a way of turning pull requests into package builds
>  > already,
>  > so I'd want to find that and build on it rather than reinventing it
>  > if
>  > possible.
>  
>  Yes, this exists. Both in Zuul and as a "Fedora CI" job. How to
>  retrieve that 
>  job programmatically, I have no idea. But it is there. 
Cool. If we ever resolve the capacity problems (see other email), I'll
have to try and figure out how nicely we can get from "pull request" to
"Koji scratch build task IDs" programmatically...
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net

[Dan Čermák]

Hi,

Adam Williamson <adamwill(a)fedoraproject.org&gt; writes:

>  On Tue, 2021-12-07 at 13:21 +0100, Miro Hrončok wrote:
> > Hello QA,
> > 
> > is there a way (even a hacky one) to run openQA tests (the ones that run on 
> > Bodhi updates of critpath packages) on an open Pull Request?
>
>  Hey Miro! Right now there really isn't, I'm afraid. The least official
>  thing we can run the tests for right now is a scratch build, so you
>  need to at least turn the PR into a scratch build somehow.
>
>  There's no public/self-serve way to schedule for a scratch build at the
>  moment; someone with admin/moderator access (so, usually, me) needs to
>  do it for you.
>
>  If there is substantial interest I could look at implementing something
>  here...I would expect that some system or other (CI? Koschei?) must
>  surely have a way of turning pull requests into package builds already,
>  so I'd want to find that and build on it rather than reinventing it if
>  possible.
>
>  The other stumbling block is that we still don't run these tests on
>  Rawhide, so if you want to test the PR in the context of the
>  rawhide/main branch, it would be more difficult (even just doing a one-
>  off, because the tests use pre-rolled disk images as a base, and we
>  don't build several of the needed images for Rawhide currently).
>
>  That's something that could change, but it would be a fairly major
>  thing and might possibly require me to beg infra for more hardware or
>  finally find the time to see if we can run openQA tests In The Cloud. I
>  can provide more details on that whole area if you're interested... 
So, opensuse and suse do the following for the released distributions
(e.g. opensuse Leap):
each update gets put into a staging area in the build service, which
rebuilds a subset of the distribution, creates the installation images
and publishes repositories from the staging area. Then openqa is
triggered for the staging area and a bot reports back.

This is not exactly a test on a pull request though, because the actual
pull request equivalent happens in a different place. It is more
equivalent to testing a bodhi update or a side tag.

What I could imagine would be the following: some CI (e.g. Zuul) builds
an image from the pull request and publishes that image somewhere
publicly accessible. Then Zuul would trigger an openqa job and set
ISO_1_URL or HDD_1_URL to the published image, openqa then pulls these
and runs all tests. It would however require that no repositories are
accessed during the tests. And we'd also need some bot to report back an
pagure, but that shouldn't be impossible either, as openqa publishes all
test results via mqtt.

And yes, that will require more hardware to run so many tests.


Hope this helps,

Dan

> -- 
> Adam Williamson
> Fedora QA
> IRC: adamw | Twitter: adamw_ha
> https://www.happyassassin.net
>
> _______________________________________________
> qa-devel mailing list -- qa-devel(a)lists.fedoraproject.org
> To unsubscribe send an email to qa-devel-leave(a)lists.fedoraproject.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[e-mail]...
> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure

[Adam Williamson]

On Tue, 2021-12-07 at 22:54 +0100, Dan Čermák wrote:
>  
>  So, opensuse and suse do the following for the released distributions
>  (e.g. opensuse Leap):
>  each update gets put into a staging area in the build service, which
>  rebuilds a subset of the distribution, creates the installation images
>  and publishes repositories from the staging area. Then openqa is
>  triggered for the staging area and a bot reports back.
>  
>  This is not exactly a test on a pull request though, because the actual
>  pull request equivalent happens in a different place. It is more
>  equivalent to testing a bodhi update or a side tag. 
Yeah. We already do the equivalent of this in Fedora, with automated
tests on updates (critical path only, currently, but that's an
arbitrary restraint for reasons of capacity).

>  What I could imagine would be the following: some CI (e.g. Zuul)
> builds
>  an image from the pull request and publishes that image somewhere
>  publicly accessible. Then Zuul would trigger an openqa job and set
>  ISO_1_URL or HDD_1_URL to the published image, openqa then pulls these
>  and runs all tests. It would however require that no repositories are
>  accessed during the tests. 
We don't need exactly this, because in Fedora we have the concept of
testing an update as opposed to a compose. And *really* this is set up
so it can be a test of an update or a Koji task. So as long as we can
get a Koji task that's a scratch build with the PR included, we can
test it.

For the sake of interest, this type of testing is done by booting from
a base disk image for the release in question and setting up a side
repository that contains the packages from the update or scratch build;
we do an immediate update from that side repo before proceeding with
the rest of the test, and also ensure it is enabled for all operations
during the rest of the test that do package transactions.

We actually do sort of the inverse of what you suggest, because one of
the tests we run on updates is "build an image with the update included
at all stages of the image build process, then test it's installable".
We did it this way around because the mechanism to have something
external build images from updates just didn't (and still doesn't,
really) exist.

>   And we'd also need some bot to report back an
>  pagure, but that shouldn't be impossible either, as openqa publishes all
>  test results via mqtt. 
I think you mean AMQP here? But yes, it does, and this should be
workable.
>  
>  And yes, that will require more hardware to run so many tests. 
Yeah, that's the restraint we're increasingly running into. For x86_64
we currently have a small amount of headroom, but not enough to turn on
critpath update tests for Rawhide, I don't think. I'm not sure about
pull request tests, it really depends how much people use the pull
request mechanism for packages...but of course, we would want to be in
a position to handle the load if people start using it a *lot*, because
we don't want to get into the position where we build a cool system
that falls over when people start really using it :)

I'm gonna look into this some more in the new year, because I *do* want
to try and get critpath update testing on Rawhide running at least.
-- 
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net