On Tue, 2021-12-07 at 22:54 +0100, Dan Čermák wrote:
So, opensuse and suse do the following for the released distributions (e.g. opensuse Leap): each update gets put into a staging area in the build service, which rebuilds a subset of the distribution, creates the installation images and publishes repositories from the staging area. Then openqa is triggered for the staging area and a bot reports back.
This is not exactly a test on a pull request though, because the actual pull request equivalent happens in a different place. It is more equivalent to testing a bodhi update or a side tag.
Yeah. We already do the equivalent of this in Fedora, with automated tests on updates (critical path only, currently, but that's an arbitrary restraint for reasons of capacity).
What I could imagine would be the following: some CI (e.g. Zuul) builds an image from the pull request and publishes that image somewhere publicly accessible. Then Zuul would trigger an openqa job and set ISO_1_URL or HDD_1_URL to the published image, openqa then pulls these and runs all tests. It would however require that no repositories are accessed during the tests.
We don't need exactly this, because in Fedora we have the concept of testing an update as opposed to a compose. And *really* this is set up so it can be a test of an update or a Koji task. So as long as we can get a Koji task that's a scratch build with the PR included, we can test it.
For the sake of interest, this type of testing is done by booting from a base disk image for the release in question and setting up a side repository that contains the packages from the update or scratch build; we do an immediate update from that side repo before proceeding with the rest of the test, and also ensure it is enabled for all operations during the rest of the test that do package transactions.
We actually do sort of the inverse of what you suggest, because one of the tests we run on updates is "build an image with the update included at all stages of the image build process, then test it's installable". We did it this way around because the mechanism to have something external build images from updates just didn't (and still doesn't, really) exist.
And we'd also need some bot to report back an pagure, but that shouldn't be impossible either, as openqa publishes all test results via mqtt.
I think you mean AMQP here? But yes, it does, and this should be workable.
And yes, that will require more hardware to run so many tests.
Yeah, that's the restraint we're increasingly running into. For x86_64 we currently have a small amount of headroom, but not enough to turn on critpath update tests for Rawhide, I don't think. I'm not sure about pull request tests, it really depends how much people use the pull request mechanism for packages...but of course, we would want to be in a position to handle the load if people start using it a *lot*, because we don't want to get into the position where we build a cool system that falls over when people start really using it :)
I'm gonna look into this some more in the new year, because I *do* want to try and get critpath update testing on Rawhide running at least.