On Fri, 2020-02-28 at 10:06 +0100, Jan Pazdziora wrote:
On Fri, Feb 28, 2020 at 09:20:16AM +0100, Pierre-Yves Chibon wrote:
> > Depending on how the pipeline actually *does* testing of updates, you
> > *may* be able to schedule on the update being created or edited
> > instead/as well (this is how the openQA scheduler does it; we don't use
> > the koji-build-group.build.complete topic at all, we use
> > bodhi.update.request.testing (published when an update is created) and
> > bodhi.update.edit (published when an update is edited) instead.
> > However, you can only do that if the pipeline does not rely on the
> > packages actually being in the updates-testing repo, but retrieves them
> > itself. At the point those messages are published, the package files
> > cannot be relied upon to be present anywhere outside of Koji.
>
> Well, when the update is created the RPMs are signed yet and we want to test the
> signed RPM to make sure that we test what is being pushed to the users.
>
> From reading this, I think things work as designed, it's just that there were
no
> push to update-testing for F32 yet.
The question is, could the CI run based on some other messages, like
those that Adam mentioned? Will the test setup be able to pull the
bits from koji or does it rely on updates-testing? Bruno was able to
schedule the CI job for my Fedora 32 update and it passed so it looks
like it was able to get the bits in spite of no update-testing pushes
for Fedora 32.
Pierre was answering that question, I think: his answer is that he
doesn't want CI to pull direct from Koji because when you do that, you
get RPMs which have not been through the signing process, and he thinks
it's important to test exactly what actually goes out to users. It's a
reasonable point.
In general it seems like a suboptimal user experience from
maintainers'
point of view when they create an errata and there is an undefined
time for which the errata will show failing tests because the gating
tests haven't been even scheduled. Any change we can make to get the
erratas all ready (with tests run and passed) will increase the chance
that maintainers will be willing to add gating tests to their
packages.
I mean, it should be possible to fudge this, though it may involve the
sort of magic some folks tend to be unhappy with. If we know this is a
thing that happens, we can maybe make the UI look less angry and red
when the state is 'tests not yet scheduled' as opposed to 'a test
actually failed', right? We can even start doing more speculative stuff
like having a timer - if all the required tests aren't scheduled *24
hours after the update is submitted*, that's more worrying that if they
aren't scheduled 5 minutes after it's submitted. Or we could teach
Greenwave or Bodhi or something about updates-testing pushes in some
way...
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net