Hi bowlofeggs, I saw this on IRC after I had left for the evening. I
hope it's okay to answer on the mailing list.
These are good points, and solving them depends on work by many other
parties.
> bowlofeggs said:
> I can't stop thinking about something that i think doesn't make sense> to me about the current design around the CI pipeline: why are we
> connecting commit hash based tests to bodhi? by the time you've gotten
> to bodhi two problems leap to mind:>
> 0) the build being shipped isn't the same as the build tested in the CI
> pipeline (even if the sources are the same, the buildroot isn't
> guaranteed to have been)
Isn't Bodhi looking up the results for the exact NVR in Koji?
If so, then this is something that needs to be reconciled in the pipeline.
The pipeline folks are working towards registering the builds with Koji
and/or doing the build in Koji. That means exactly the RPM which is
built is the one which will be published by Bodhi.
Ditto for modules, MBS, and module based artifacts such as qcow2 Atomic
Host images.
> 1) commit results should be reported to the developer when > they make the commit (or when they propose the commit to be merged in
pagure)
Indeed, we should also report commit results there. That's future work
to be done. In addition, modules come into play here ... more on that below.
> <bowlofeggs> there's also a third problem: the CI pipeline didn't
> test the list of builds in the bodhi update, but it instead tested > individual builds - builds frequently pass or fail in combination with
others
Yup, definitely.
Keep in mind that even if they pass or fail is combination with those
specific others, there's still no guarantee that the end user doesn't
mix and match things in a way that causes the whole thing to fail
dramatically on their system.
This is where modules come into the picture. Modules complete this
story. A module represents all the software that is guaranteed to work
together, tested together, installed together.
The pipeline folks are looking to integrate with the modularity work in
Fedora so that we can test known sets of software together with known
buildroots, and gate them together.
In addition the Atomic Host folks are looking to define Atomic Host from
a module.
> <bowlofeggs> from my perspective, i think it makes more sense to run> bodhi gating tests in response to the creation of bodhi updates
> and it makes sense to write PR gating tests in response to the > creation of pagure PRs
> but, gating bodhi tests in response to the creation of pagure PRs
>causes problems, and i think also doesn't solve the intended problems
These would *definitely* be the same tests, but you're right they are
running in a different context.
So what we're trying to do is avoid having to boil the ocean before
getting started, and actually start to get feedback to people who are
porting the tests.
Once all the modularity work lands in Bodhi, pipeline, module level
tests, module definitions, arbitrary branching ... we will converge on a
workable and complete solution. However until then, would it work to
report results and gate on packages in Bodhi in a more limited but
workable fashion?
Does Bodhi have what it needs today to report results present in
ResultsDB for an update. If so, then overtime those results will get
better and better (see above, work on integrating with Koji, MBS,
modules etc.).
The most important missing ingredient today is missing display of
results and gating of the pipeline for packages that opt-into the system.
Without that we don't have forward progress on tests or a myriad of
other aspects of this work.
Cheers,
Stef