Status of CI for Fedora 32 errata?

[Jan Pazdziora]

Hello,

I've built a package today for rawhide, it got errata

	https://bodhi.fedoraproject.org/updates/FEDORA-2020-a9b4fc5c9d

created automatically and

	org.centos.prod.ci.pipeline.allpackages-build.package.test.functional.complete

finished for it in half an hour or so.

The I've done the same build for Fedora 32, I've filed

	https://bodhi.fedoraproject.org/updates/FEDORA-2020-8f05f05489

for it but since then

	org.centos.prod.ci.pipeline.allpackages-build.package.test.functional.complete

is stuck as missing. The red rectangle is not clickable, so it's hard
to see if the test was actually scheduled, if CI knows about the new
build, what is going on.

Could anyone with insight into the Fedora CI process take a look to
see where things got stuck, and also recommend how to do the check
in the future myself?

Thank you,

-- 
Jan Pazdziora
Product Owner, Platform Security Readiness, Red Hat

[Bruno Goncalves]

On Thu, Feb 27, 2020 at 4:19 PM Jan Pazdziora <jpazdziora(a)redhat.com&gt; wrote:
>  Hello,
>
>  I've built a package today for rawhide, it got errata
>
>          https://bodhi.fedoraproject.org/updates/FEDORA-2020-a9b4fc5c9d
>
>  created automatically and
>
>          org.centos.prod.ci.pipeline.allpackages-build.package.test.functional.complete
>
>  finished for it in half an hour or so.
>
>  The I've done the same build for Fedora 32, I've filed
>
>          https://bodhi.fedoraproject.org/updates/FEDORA-2020-8f05f05489
>
>  for it but since then
>
>          org.centos.prod.ci.pipeline.allpackages-build.package.test.functional.complete
>
>  is stuck as missing. The red rectangle is not clickable, so it's hard
>  to see if the test was actually scheduled, if CI knows about the new
>  build, what is going on. 
That is interesting, it looks like the pipeline didn't trigger because
bodhi didn't send a message for it.

At least the only recent message that it sent about this package
recently was for rawhide....
https://apps.fedoraproject.org/datagrepper/raw?topic=org.fedoraproject.pr...


Bruno

>  Could anyone with insight into the Fedora CI process take a look to
>  see where things got stuck, and also recommend how to do the check
>  in the future myself?
>
>  Thank you,
>
>  --
>  Jan Pazdziora
>  Product Owner, Platform Security Readiness, Red Hat
>  _______________________________________________
>  CI mailing list -- ci(a)lists.fedoraproject.org
>  To unsubscribe send an email to ci-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]

[Jan Pazdziora]

On Thu, Feb 27, 2020 at 04:56:28PM +0100, Bruno Goncalves wrote:
>  
>  That is interesting, it looks like the pipeline didn't trigger because
>  bodhi didn't send a message for it.
>  
>  At least the only recent message that it sent about this package
>  recently was for rawhide....
>
> https://apps.fedoraproject.org/datagrepper/raw?topic=org.fedoraproject.pr...
>   
Thank you for looking into it.

Does that sound like a bug in bodhi, so something to file an issue at

	https://github.com/fedora-infra/bodhi/issues

for?

Is the only way to force that CI job to run to make bodhi send the
message?

-- 
Jan Pazdziora
Product Owner, Platform Security Readiness, Red Hat

[Bruno Goncalves]

On Thu, Feb 27, 2020 at 5:04 PM Jan Pazdziora <jpazdziora(a)redhat.com&gt; wrote:
>  On Thu, Feb 27, 2020 at 04:56:28PM +0100, Bruno Goncalves wrote:
>  >
>  > That is interesting, it looks like the pipeline didn't trigger because
>  > bodhi didn't send a message for it.
>  >
>  > At least the only recent message that it sent about this package
>  > recently was for rawhide....
>  >
> https://apps.fedoraproject.org/datagrepper/raw?topic=org.fedoraproject.pr...
>  >
>
>  Thank you for looking into it.
>
>  Does that sound like a bug in bodhi, so something to file an issue at
>
>          https://github.com/fedora-infra/bodhi/issues
>
>  for? 
I'm not sure either, I'd rather if someone with more experience with
bodhi could say when
"org.fedoraproject.prod.bodhi.update.status.testing.koji-build-group.build.complete"
is sent.
Or if we should trigger with some other topic...

I've been triggering on it since https://pagure.io/fedora-ci/general/issue/70

>  Is the only way to force that CI job to run to make bodhi send the
>  message? 
I've rescheduled it manually, let's see how it goes...
https://jenkins-continuous-infra.apps.ci.centos.org/job/fedora-rawhide-bu...

>  --
>  Jan Pazdziora
>  Product Owner, Platform Security Readiness, Red Hat
>  _______________________________________________
>  CI mailing list -- ci(a)lists.fedoraproject.org
>  To unsubscribe send an email to ci-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]

[Jan Pazdziora]

On Thu, Feb 27, 2020 at 05:49:07PM +0100, Bruno Goncalves wrote:
>  
>  I've rescheduled it manually, let's see how it goes...
>
> https://jenkins-continuous-infra.apps.ci.centos.org/job/fedora-rawhide-bu...
>   
It passed and

	https://bodhi.fedoraproject.org/updates/FEDORA-2020-8f05f05489

is now happy with six passing tests.

Thank you!

-- 
Jan Pazdziora
Product Owner, Platform Security Readiness, Red Hat

[Adam Williamson]

On Thu, 2020-02-27 at 17:49 +0100, Bruno Goncalves wrote:
>  On Thu, Feb 27, 2020 at 5:04 PM Jan Pazdziora
> <jpazdziora(a)redhat.com&gt; wrote:
>  > On Thu, Feb 27, 2020 at 04:56:28PM +0100, Bruno Goncalves wrote:
>  > > That is interesting, it looks like the pipeline didn't trigger because
>  > > bodhi didn't send a message for it.
>  > > 
>  > > At least the only recent message that it sent about this package
>  > > recently was for rawhide....
>  > >
> https://apps.fedoraproject.org/datagrepper/raw?topic=org.fedoraproject.pr...
>  > > 
>  > 
>  > Thank you for looking into it.
>  > 
>  > Does that sound like a bug in bodhi, so something to file an issue at
>  > 
>  >         https://github.com/fedora-infra/bodhi/issues
>  > 
>  > for?
>  
>  I'm not sure either, I'd rather if someone with more experience with
>  bodhi could say when
>
> "org.fedoraproject.prod.bodhi.update.status.testing.koji-build-group.build.complete"
>  is sent. 
Per the relevant doc string:

"        Signal that the update has been moved to testing.

        This happens in the following cases:
        - for stable releases: the update lands in the testing repository
        - for rawhide: all packages in an update have been built by koji
"

https://github.com/fedora-infra/bodhi/blob/develop/bodhi/server/models.py...
(that function does nothing but publish the message in question, and
nothing else publishes it).

Note that "the update lands in the testing repository" definitely is
*not* the same point in time as "the update is created". It will
usually be at least several minutes later, more usually several hours
later (pushes to updates-testing usually happen once a day per release,
more can be done manually if requested for some reason).

This was worse for F32 up until now - there haven't been *any* updates-
testing pushes for it yet. I just asked releng, and Mohan is going to
do one now. From then on there should be one a day.

So the 'typical' experience, as things stand, will (I expect) be that
updates come up as failing gating shortly after they're created; then a
few hours later, an updates-testing push should happen, the tests
should be scheduled, and then the status should change to passing
gating (as long as the tests pass).

Depending on how the pipeline actually *does* testing of updates, you
*may* be able to schedule on the update being created or edited
instead/as well (this is how the openQA scheduler does it; we don't use
the koji-build-group.build.complete topic at all, we use
bodhi.update.request.testing (published when an update is created) and
bodhi.update.edit (published when an update is edited) instead.
However, you can only do that if the pipeline does not rely on the
packages actually being in the updates-testing repo, but retrieves them
itself. At the point those messages are published, the package files
cannot be relied upon to be present anywhere outside of Koji.

HTH!
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net

[Pierre-Yves Chibon]

On Thu, Feb 27, 2020 at 01:27:46PM -0800, Adam Williamson wrote:
>  On Thu, 2020-02-27 at 17:49 +0100, Bruno Goncalves wrote:
>  > On Thu, Feb 27, 2020 at 5:04 PM Jan Pazdziora <jpazdziora(a)redhat.com&gt; wrote:
>  > > On Thu, Feb 27, 2020 at 04:56:28PM +0100, Bruno Goncalves wrote:
>  > > > That is interesting, it looks like the pipeline didn't trigger
> because
>  > > > bodhi didn't send a message for it.
>  > > > 
>  > > > At least the only recent message that it sent about this package
>  > > > recently was for rawhide....
>  > > >
> https://apps.fedoraproject.org/datagrepper/raw?topic=org.fedoraproject.pr...
>  > > > 
>  > > 
>  > > Thank you for looking into it.
>  > > 
>  > > Does that sound like a bug in bodhi, so something to file an issue at
>  > > 
>  > >         https://github.com/fedora-infra/bodhi/issues
>  > > 
>  > > for?
>  > 
>  > I'm not sure either, I'd rather if someone with more experience with
>  > bodhi could say when
>  >
> "org.fedoraproject.prod.bodhi.update.status.testing.koji-build-group.build.complete"
>  > is sent.
>  
>  Per the relevant doc string:
>  
>  "        Signal that the update has been moved to testing.
>  
>          This happens in the following cases:
>          - for stable releases: the update lands in the testing repository
>          - for rawhide: all packages in an update have been built by koji
>  "
>  
>  https://github.com/fedora-infra/bodhi/blob/develop/bodhi/server/models.py...
>  (that function does nothing but publish the message in question, and
>  nothing else publishes it).
>  
>  Note that "the update lands in the testing repository" definitely is
>  *not* the same point in time as "the update is created". It will
>  usually be at least several minutes later, more usually several hours
>  later (pushes to updates-testing usually happen once a day per release,
>  more can be done manually if requested for some reason).
>  
>  This was worse for F32 up until now - there haven't been *any* updates-
>  testing pushes for it yet. I just asked releng, and Mohan is going to
>  do one now. From then on there should be one a day. 
This is all correct :)

>  So the 'typical' experience, as things stand, will (I expect)
> be that
>  updates come up as failing gating shortly after they're created; then a
>  few hours later, an updates-testing push should happen, the tests
>  should be scheduled, and then the status should change to passing
>  gating (as long as the tests pass). 
Correct again, it "fails" gating because the expected tests aren't present
yet
indeed.

>  Depending on how the pipeline actually *does* testing of updates,
> you
>  *may* be able to schedule on the update being created or edited
>  instead/as well (this is how the openQA scheduler does it; we don't use
>  the koji-build-group.build.complete topic at all, we use
>  bodhi.update.request.testing (published when an update is created) and
>  bodhi.update.edit (published when an update is edited) instead.
>  However, you can only do that if the pipeline does not rely on the
>  packages actually being in the updates-testing repo, but retrieves them
>  itself. At the point those messages are published, the package files
>  cannot be relied upon to be present anywhere outside of Koji. 
Well, when the update is created the RPMs are signed yet and we want to test the
signed RPM to make sure that we test what is being pushed to the users.

From reading this, I think things work as designed, it's just that there were no
push to update-testing for F32 yet.


Pierre

[Jan Pazdziora]

On Fri, Feb 28, 2020 at 09:20:16AM +0100, Pierre-Yves Chibon wrote:
>  
>  > Depending on how the pipeline actually *does* testing of updates, you
>  > *may* be able to schedule on the update being created or edited
>  > instead/as well (this is how the openQA scheduler does it; we don't use
>  > the koji-build-group.build.complete topic at all, we use
>  > bodhi.update.request.testing (published when an update is created) and
>  > bodhi.update.edit (published when an update is edited) instead.
>  > However, you can only do that if the pipeline does not rely on the
>  > packages actually being in the updates-testing repo, but retrieves them
>  > itself. At the point those messages are published, the package files
>  > cannot be relied upon to be present anywhere outside of Koji.
>  
>  Well, when the update is created the RPMs are signed yet and we want to test the
>  signed RPM to make sure that we test what is being pushed to the users.
>  
>  From reading this, I think things work as designed, it's just that there were no
>  push to update-testing for F32 yet. 
The question is, could the CI run based on some other messages, like
those that Adam mentioned? Will the test setup be able to pull the
bits from koji or does it rely on updates-testing? Bruno was able to
schedule the CI job for my Fedora 32 update and it passed so it looks
like it was able to get the bits in spite of no update-testing pushes
for Fedora 32.

In general it seems like a suboptimal user experience from maintainers'
point of view when they create an errata and there is an undefined
time for which the errata will show failing tests because the gating
tests haven't been even scheduled. Any change we can make to get the
erratas all ready (with tests run and passed) will increase the chance
that maintainers will be willing to add gating tests to their
packages.

-- 
Jan Pazdziora
Product Owner, Platform Security Readiness, Red Hat

[Jan Pazdziora]

On Fri, Feb 28, 2020 at 10:06:18AM +0100, Jan Pazdziora wrote:
>  On Fri, Feb 28, 2020 at 09:20:16AM +0100, Pierre-Yves Chibon wrote:
>  > 
>  > > Depending on how the pipeline actually *does* testing of updates, you
>  > > *may* be able to schedule on the update being created or edited
>  > > instead/as well (this is how the openQA scheduler does it; we don't use
>  > > the koji-build-group.build.complete topic at all, we use
>  > > bodhi.update.request.testing (published when an update is created) and
>  > > bodhi.update.edit (published when an update is edited) instead.
>  > > However, you can only do that if the pipeline does not rely on the
>  > > packages actually being in the updates-testing repo, but retrieves them
>  > > itself. At the point those messages are published, the package files
>  > > cannot be relied upon to be present anywhere outside of Koji.
>  > 
>  > Well, when the update is created the RPMs are signed yet and we want to test the
>  > signed RPM to make sure that we test what is being pushed to the users.
>  > 
>  > From reading this, I think things work as designed, it's just that there were
> no
>  > push to update-testing for F32 yet.
>  
>  The question is, could the CI run based on some other messages, like
>  those that Adam mentioned? Will the test setup be able to pull the
>  bits from koji or does it rely on updates-testing? Bruno was able to
>  schedule the CI job for my Fedora 32 update and it passed so it looks
>  like it was able to get the bits in spite of no update-testing pushes
>  for Fedora 32. 
Checking the

	https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/je...

the rpms well pulled in from the package-specific repo

	swid-tools.noarch	0.8.9-1.fc32	@test-swid-tools

So it seems like the gating tests don't have to wait for the
updates-testing push and some other message could be used.

-- 
Jan Pazdziora
Product Owner, Platform Security Readiness, Red Hat

[Bruno Goncalves]

On Fri, Feb 28, 2020 at 4:02 PM Jan Pazdziora <jpazdziora(a)redhat.com&gt; wrote:
>  On Fri, Feb 28, 2020 at 10:06:18AM +0100, Jan Pazdziora wrote:
>  > On Fri, Feb 28, 2020 at 09:20:16AM +0100, Pierre-Yves Chibon wrote:
>  > >
>  > > > Depending on how the pipeline actually *does* testing of updates, you
>  > > > *may* be able to schedule on the update being created or edited
>  > > > instead/as well (this is how the openQA scheduler does it; we don't
> use
>  > > > the koji-build-group.build.complete topic at all, we use
>  > > > bodhi.update.request.testing (published when an update is created) and
>  > > > bodhi.update.edit (published when an update is edited) instead.
>  > > > However, you can only do that if the pipeline does not rely on the
>  > > > packages actually being in the updates-testing repo, but retrieves them
>  > > > itself. At the point those messages are published, the package files
>  > > > cannot be relied upon to be present anywhere outside of Koji.
>  > >
>  > > Well, when the update is created the RPMs are signed yet and we want to test
> the
>  > > signed RPM to make sure that we test what is being pushed to the users.
>  > >
>  > > From reading this, I think things work as designed, it's just that there
> were no
>  > > push to update-testing for F32 yet.
>  >
>  > The question is, could the CI run based on some other messages, like
>  > those that Adam mentioned? Will the test setup be able to pull the
>  > bits from koji or does it rely on updates-testing? Bruno was able to
>  > schedule the CI job for my Fedora 32 update and it passed so it looks
>  > like it was able to get the bits in spite of no update-testing pushes
>  > for Fedora 32.
>
>  Checking the
>
>         
> https://jenkins-continuous-infra.apps.ci.centos.org/blue/organizations/je...
>
>  the rpms well pulled in from the package-specific repo
>
>          swid-tools.noarch       0.8.9-1.fc32    @test-swid-tools
>
>  So it seems like the gating tests don't have to wait for the
>  updates-testing push and some other message could be used.
>  
It might be some other components require some newer version of
package as dependency that are not still on stable repositories.
We could probably discuss to trigger on different message, but it
would be nice to have all gating tests triggering on the same topic.

Jan, in case you want suggest to trigger on a different message, could
you request it as an issue on
https://pagure.io/fedora-ci/general/issues ?

Bruno
>  --
>  Jan Pazdziora
>  Product Owner, Platform Security Readiness, Red Hat
>  _______________________________________________
>  CI mailing list -- ci(a)lists.fedoraproject.org
>  To unsubscribe send an email to ci-leave(a)lists.fedoraproject.org
>  Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>  List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>  List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]

[Adam Williamson]

On Fri, 2020-02-28 at 10:06 +0100, Jan Pazdziora wrote:
>  On Fri, Feb 28, 2020 at 09:20:16AM +0100, Pierre-Yves Chibon wrote:
>  > > Depending on how the pipeline actually *does* testing of updates, you
>  > > *may* be able to schedule on the update being created or edited
>  > > instead/as well (this is how the openQA scheduler does it; we don't use
>  > > the koji-build-group.build.complete topic at all, we use
>  > > bodhi.update.request.testing (published when an update is created) and
>  > > bodhi.update.edit (published when an update is edited) instead.
>  > > However, you can only do that if the pipeline does not rely on the
>  > > packages actually being in the updates-testing repo, but retrieves them
>  > > itself. At the point those messages are published, the package files
>  > > cannot be relied upon to be present anywhere outside of Koji.
>  > 
>  > Well, when the update is created the RPMs are signed yet and we want to test the
>  > signed RPM to make sure that we test what is being pushed to the users.
>  > 
>  > From reading this, I think things work as designed, it's just that there were
> no
>  > push to update-testing for F32 yet.
>  
>  The question is, could the CI run based on some other messages, like
>  those that Adam mentioned? Will the test setup be able to pull the
>  bits from koji or does it rely on updates-testing? Bruno was able to
>  schedule the CI job for my Fedora 32 update and it passed so it looks
>  like it was able to get the bits in spite of no update-testing pushes
>  for Fedora 32. 
Pierre was answering that question, I think: his answer is that he
doesn't want CI to pull direct from Koji because when you do that, you
get RPMs which have not been through the signing process, and he thinks
it's important to test exactly what actually goes out to users. It's a
reasonable point.

>  In general it seems like a suboptimal user experience from
> maintainers'
>  point of view when they create an errata and there is an undefined
>  time for which the errata will show failing tests because the gating
>  tests haven't been even scheduled. Any change we can make to get the
>  erratas all ready (with tests run and passed) will increase the chance
>  that maintainers will be willing to add gating tests to their
>  packages. 
I mean, it should be possible to fudge this, though it may involve the
sort of magic some folks tend to be unhappy with. If we know this is a
thing that happens, we can maybe make the UI look less angry and red
when the state is 'tests not yet scheduled' as opposed to 'a test
actually failed', right? We can even start doing more speculative stuff
like having a timer - if all the required tests aren't scheduled *24
hours after the update is submitted*, that's more worrying that if they
aren't scheduled 5 minutes after it's submitted. Or we could teach
Greenwave or Bodhi or something about updates-testing pushes in some
way...
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net