Hi bowlofeggs, I saw this on IRC after I had left for the evening. I hope it's okay to answer on the mailing list.
These are good points, and solving them depends on work by many other parties.
bowlofeggs said: I can't stop thinking about something that i think doesn't make sense> to me about the current design around the CI pipeline: why are we connecting commit hash based tests to bodhi? by the time you've gotten to bodhi two problems leap to mind:> 0) the build being shipped isn't the same as the build tested in the CI pipeline (even if the sources are the same, the buildroot isn't guaranteed to have been)
Isn't Bodhi looking up the results for the exact NVR in Koji?
If so, then this is something that needs to be reconciled in the pipeline.
The pipeline folks are working towards registering the builds with Koji and/or doing the build in Koji. That means exactly the RPM which is built is the one which will be published by Bodhi.
Ditto for modules, MBS, and module based artifacts such as qcow2 Atomic Host images.
- commit results should be reported to the developer when > they make the commit (or when they propose the commit to be merged in
pagure)
Indeed, we should also report commit results there. That's future work to be done. In addition, modules come into play here ... more on that below.
<bowlofeggs> there's also a third problem: the CI pipeline didn't test the list of builds in the bodhi update, but it instead tested > individual builds - builds frequently pass or fail in combination with
others
Yup, definitely.
Keep in mind that even if they pass or fail is combination with those specific others, there's still no guarantee that the end user doesn't mix and match things in a way that causes the whole thing to fail dramatically on their system.
This is where modules come into the picture. Modules complete this story. A module represents all the software that is guaranteed to work together, tested together, installed together.
The pipeline folks are looking to integrate with the modularity work in Fedora so that we can test known sets of software together with known buildroots, and gate them together.
In addition the Atomic Host folks are looking to define Atomic Host from a module.
<bowlofeggs> from my perspective, i think it makes more sense to run> bodhi gating tests in response to the creation of bodhi updates and it makes sense to write PR gating tests in response to the > creation of pagure PRs but, gating bodhi tests in response to the creation of pagure PRs causes problems, and i think also doesn't solve the intended problems
These would *definitely* be the same tests, but you're right they are running in a different context.
So what we're trying to do is avoid having to boil the ocean before getting started, and actually start to get feedback to people who are porting the tests.
Once all the modularity work lands in Bodhi, pipeline, module level tests, module definitions, arbitrary branching ... we will converge on a workable and complete solution. However until then, would it work to report results and gate on packages in Bodhi in a more limited but workable fashion?
Does Bodhi have what it needs today to report results present in ResultsDB for an update. If so, then overtime those results will get better and better (see above, work on integrating with Koji, MBS, modules etc.).
The most important missing ingredient today is missing display of results and gating of the pipeline for packages that opt-into the system.
Without that we don't have forward progress on tests or a myriad of other aspects of this work.
Cheers,
Stef