Rawhide update from side tag pending for 2 days

[Richard W.M. Jones]

https://bodhi.fedoraproject.org/updates/FEDORA-2020-d120de33c5

This one is a Rawhide update from a side tag, submitted on Sunday
morning which has been in pending for 2 days.  (As it's Rawhide it's
supposed to spend 0 days in testing.)  Is there any problem with it,
or do we just have to wait longer?  It is blocking builds of other
packages.

Rich.

[Fabio Valentini]

On Tue, Apr 7, 2020 at 8:56 AM Richard W.M. Jones  wrote:
> https://bodhi.fedoraproject.org/updates/FEDORA-2020-d120de33c5
>
> This one is a Rawhide update from a side tag, submitted on Sunday
> morning which has been in pending for 2 days.  (As it's Rawhide it's
> supposed to spend 0 days in testing.)  Is there any problem with it,
> or do we just have to wait longer?  It is blocking builds of other
> packages.

Yeah, this update looks stuck ... the koji builds aren't even tagged
into the right tags yet, neither are they signed (from what I can
see).

I seem to have the same - or at least a similar - issue with three
updates that are also stuck in pending for two days already (and the
builds are not even signed yet, either):

- https://bodhi.fedoraproject.org/updates/FEDORA-2020-460f0f4d48
- https://bodhi.fedoraproject.org/updates/FEDORA-2020-6ea279079e
- https://bodhi.fedoraproject.org/updates/FEDORA-2020-755f442b4c

Those three were also made from side tags, but for stable releases, so
I'm not sure whether it's the same issue or not.

Fabio
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-df lists disk usage of guests without needing to install any
> software inside the virtual machine.  Supports Linux and Windows.
> http://people.redhat.com/~rjones/virt-df/
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]

[Nicolas Mailhot]

There is a problem right now with the part of koji that tags builds and
adds them to the various repos koji uses for new builds. So you can
build new packages, but can not rely on further builds seeing your
just-built packages.

[Stephen John Smoogen]

For each of these stuck builds could you open up tickets in
https://pagure.io/releng/new_issue so we can track them down and fix them.
Currently there are 1.5 sysadmins and 1.5 release engineer and we are not
going to be able to track what and where we are doing things from mailing
list posts.

On Tue, 7 Apr 2020 at 08:41, Nicolas Mailhot via devel <
[e-mail]> wrote:
> There is a problem right now with the part of koji that tags builds and
> adds them to the various repos koji uses for new builds. So you can
> build new packages, but can not rely on further builds seeing your
> just-built packages.
>
> --
> Nicolas Mailhot
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Mohan Boddu]

On Tue, Apr 7, 2020 at 8:34 AM Fabio Valentini  wrote:
> On Tue, Apr 7, 2020 at 8:56 AM Richard W.M. Jones  wrote:
> > https://bodhi.fedoraproject.org/updates/FEDORA-2020-d120de33c5
> >
> > This one is a Rawhide update from a side tag, submitted on Sunday
> > morning which has been in pending for 2 days.  (As it's Rawhide it's
> > supposed to spend 0 days in testing.)  Is there any problem with it,
> > or do we just have to wait longer?  It is blocking builds of other
> > packages.
>
> Yeah, this update looks stuck ... the koji builds aren't even tagged
> into the right tags yet, neither are they signed (from what I can
> see).
>
> I seem to have the same - or at least a similar - issue with three
> updates that are also stuck in pending for two days already (and the
> builds are not even signed yet, either):
> https://bodhi.fedoraproject.org/updates/FEDORA-2020-460f0f4d48
> https://bodhi.fedoraproject.org/updates/FEDORA-2020-6ea279079e
> https://bodhi.fedoraproject.org/updates/FEDORA-2020-755f442b4c

This is a different case, can you file a ticket at
https://pagure.io/releng/new_issue. I am guessing this is caused by
the recent release of enabling side tags in now rawhide releases.
> Those three were also made from side tags, but for stable releases, so
> I'm not sure whether it's the same issue or not.
>
> Fabio
> > Rich.
> >
> > --
> > Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
> > Read my programming and virtualization blog: http://rwmj.wordpress.com
> > virt-df lists disk usage of guests without needing to install any
> > software inside the virtual machine.  Supports Linux and Windows.
> > http://people.redhat.com/~rjones/virt-df/
> > _______________________________________________
> > devel mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
>
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]

[Rex Dieter]

Richard W.M. Jones wrote:
> https://bodhi.fedoraproject.org/updates/FEDORA-2020-d120de33c5
>
> This one is a Rawhide update from a side tag, submitted on Sunday
> morning which has been in pending for 2 days.  (As it's Rawhide it's
> supposed to spend 0 days in testing.)  Is there any problem with it,
> or do we just have to wait longer?  It is blocking builds of other
> packages.

I saw the same thing, and submitted a infra ticket about it,
https://pagure.io/fedora-infrastructure/issue/8819

-- Rex

[Mohan Boddu]

On Tue, Apr 7, 2020 at 8:57 AM Rex Dieter  wrote:
> Richard W.M. Jones wrote:
> > https://bodhi.fedoraproject.org/updates/FEDORA-2020-d120de33c5
> >
> > This one is a Rawhide update from a side tag, submitted on Sunday
> > morning which has been in pending for 2 days.  (As it's Rawhide it's
> > supposed to spend 0 days in testing.)  Is there any problem with it,
> > or do we just have to wait longer?  It is blocking builds of other
> > packages.
>
> I saw the same thing, and submitted a infra ticket about it,
> https://pagure.io/fedora-infrastructure/issue/8819

In https://bodhi.fedoraproject.org/updates/FEDORA-2020-641e69f08c, I
can see that qt-creator-4.12.0-0.3.rc1.fc33 is the latest version and
the latest build compared to the qt-creator-4.12.0-0.2.beta1.fc33
build in the update. You can remove the build from the update to move
it to stable.
> -- Rex
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]

[Clement Verna]

On Tue, 7 Apr 2020 at 14:57, Rex Dieter  wrote:
> Richard W.M. Jones wrote:
> > https://bodhi.fedoraproject.org/updates/FEDORA-2020-d120de33c5
> >
> > This one is a Rawhide update from a side tag, submitted on Sunday
> > morning which has been in pending for 2 days.  (As it's Rawhide it's
> > supposed to spend 0 days in testing.)  Is there any problem with it,
> > or do we just have to wait longer?  It is blocking builds of other
> > packages.
>
> I saw the same thing, and submitted a infra ticket about it,
> https://pagure.io/fedora-infrastructure/issue/8819

So I started to look at that ticket, and Bodhi had a problem creating the
pending-signing and pending-testing tags in koji. I have fixed that and now
the update was not pushed to stable because there are builds in the side
tag conflicting with builds in the f33 buildroot. You need to edit your
update to either include a rebuild of these conflicting packages or just
remove them  from the update.

There is an handy comment on the update that tells you which builds are
conflicting.
> -- Rex
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Mohan Boddu]

I guess it should be a fixed in bodhi. But for now you can remove the
builds that it's complaining about in the update.

https://github.com/fedora-infra/bodhi/issues/3991

On Tue, Apr 7, 2020 at 2:56 AM Richard W.M. Jones  wrote:
> https://bodhi.fedoraproject.org/updates/FEDORA-2020-d120de33c5
>
> This one is a Rawhide update from a side tag, submitted on Sunday
> morning which has been in pending for 2 days.  (As it's Rawhide it's
> supposed to spend 0 days in testing.)  Is there any problem with it,
> or do we just have to wait longer?  It is blocking builds of other
> packages.
>
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-df lists disk usage of guests without needing to install any
> software inside the virtual machine.  Supports Linux and Windows.
> http://people.redhat.com/~rjones/virt-df/
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]

[Clement Verna]

On Tue, 7 Apr 2020 at 23:38, Mohan Boddu  wrote:
> I guess it should be a fixed in bodhi. But for now you can remove the
> builds that it's complaining about in the update.
>
> https://github.com/fedora-infra/bodhi/issues/3991

Answered in the ticket, but I think Bodhi is behaving correctly there. When
trying to merge the side tag, if Bodhi find a more recent build in the
target tag it will push back the update to pending and comment on the
update that there are more recent builds in the target tag. Then it is up
to the packager to resolve the conflict (either rebuild the conflicting
packages or remove them from the update).
> On Tue, Apr 7, 2020 at 2:56 AM Richard W.M. Jones 
> wrote:
> > https://bodhi.fedoraproject.org/updates/FEDORA-2020-d120de33c5
> >
> > This one is a Rawhide update from a side tag, submitted on Sunday
> > morning which has been in pending for 2 days.  (As it's Rawhide it's
> > supposed to spend 0 days in testing.)  Is there any problem with it,
> > or do we just have to wait longer?  It is blocking builds of other
> > packages.
> >
> > Rich.
> >
> > --
> > Richard Jones, Virtualization Group, Red Hat
>
> http://people.redhat.com/~rjones
> > Read my programming and virtualization blog: http://rwmj.wordpress.com
> > virt-df lists disk usage of guests without needing to install any
> > software inside the virtual machine.  Supports Linux and Windows.
> > http://people.redhat.com/~rjones/virt-df/
> > _______________________________________________
> > devel mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct:
>
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
>
> https://lists.fedoraproject.org/archives/list/[e-mail]
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Igor Gnatenko]

I agree, if this would not happen then everything would just blow up.

On Wed, Apr 8, 2020 at 8:33 AM Clement Verna  wrote:
> On Tue, 7 Apr 2020 at 23:38, Mohan Boddu  wrote:
> > I guess it should be a fixed in bodhi. But for now you can remove the
> > builds that it's complaining about in the update.
> >
> > https://github.com/fedora-infra/bodhi/issues/3991
>
> Answered in the ticket, but I think Bodhi is behaving correctly there. When trying to merge the side tag, if Bodhi find a more recent build in the target tag it will push back the update to pending and comment on the update that there are more recent builds in the target tag. Then it is up to the packager to resolve the conflict (either rebuild the conflicting packages or remove them from the update).
> > On Tue, Apr 7, 2020 at 2:56 AM Richard W.M. Jones  wrote:
> > > https://bodhi.fedoraproject.org/updates/FEDORA-2020-d120de33c5
> > >
> > > This one is a Rawhide update from a side tag, submitted on Sunday
> > > morning which has been in pending for 2 days.  (As it's Rawhide it's
> > > supposed to spend 0 days in testing.)  Is there any problem with it,
> > > or do we just have to wait longer?  It is blocking builds of other
> > > packages.
> > >
> > > Rich.
> > >
> > > --
> > > Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
> > > Read my programming and virtualization blog: http://rwmj.wordpress.com
> > > virt-df lists disk usage of guests without needing to install any
> > > software inside the virtual machine.  Supports Linux and Windows.
> > > http://people.redhat.com/~rjones/virt-df/
> > > _______________________________________________
> > > devel mailing list -- [e-mail]
> > > To unsubscribe send an email to [e-mail]
> > > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
> >
> > devel mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
>
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]

[Richard W.M. Jones]

On Wed, Apr 08, 2020 at 08:53:38AM +0200, Igor Gnatenko wrote:
> I agree, if this would not happen then everything would just blow up.

Wouldn't the lower NVR builds simply be ignored?

Rich.

[Clement Verna]

On Wed, 8 Apr 2020 at 11:29, Richard W.M. Jones  wrote:
> On Wed, Apr 08, 2020 at 08:53:38AM +0200, Igor Gnatenko wrote:
> > I agree, if this would not happen then everything would just blow up.
>
> Wouldn't the lower NVR builds simply be ignored?

The merging a side tag is not considering NVRs but the date the build was
done. Bodhi just checks if a there is a more recent build in the rawhide
buildroot, if that is the case it does not try to be too smart about it and
let a human find out what is the best thing to do :-).

The logic was taken from how releng is merging side tag, the human part of
finding out what needs to be done was then I guess done by a release
engineer.
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat
> http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-df lists disk usage of guests without needing to install any
> software inside the virtual machine.  Supports Linux and Windows.
> http://people.redhat.com/~rjones/virt-df/
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Richard W.M. Jones]

On Tue, Apr 07, 2020 at 05:36:44PM -0400, Mohan Boddu wrote:
> On Tue, Apr 7, 2020 at 2:56 AM Richard W.M. Jones  wrote:
> > https://bodhi.fedoraproject.org/updates/FEDORA-2020-d120de33c5
>
> I guess it should be a fixed in bodhi. But for now you can remove the
> builds that it's complaining about in the update.
>
> https://github.com/fedora-infra/bodhi/issues/3991

So it is complaining:

"This update cannot be pushed to stable. These builds
  brltty-6.0-14.fc33, graphviz-2.42.2-10.fc33 have a more recent build
  in koji's f33 tag."

Those builds could just be removed before pushing.  But I don't see a
way to remove the builds from the UI.

(This update took 3 days of work to create so I _really_ do not want
to have to build the whole thing again.)

Rich.

[Fabio Valentini]

On Wed, Apr 8, 2020, 11:27 Richard W.M. Jones  wrote:
> On Tue, Apr 07, 2020 at 05:36:44PM -0400, Mohan Boddu wrote:
> > On Tue, Apr 7, 2020 at 2:56 AM Richard W.M. Jones
>
> wrote:
> > > https://bodhi.fedoraproject.org/updates/FEDORA-2020-d120de33c5
> >
> > I guess it should be a fixed in bodhi. But for now you can remove the
> > builds that it's complaining about in the update.
> >
> > https://github.com/fedora-infra/bodhi/issues/3991
>
> So it is complaining:
>
> "This update cannot be pushed to stable. These builds
>   brltty-6.0-14.fc33, graphviz-2.42.2-10.fc33 have a more recent build
>   in koji's f33 tag."
>
> Those builds could just be removed before pushing.  But I don't see a
> way to remove the builds from the UI.
>
> (This update took 3 days of work to create so I _really_ do not want
> to have to build the whole thing again.)

You should be able to run

koji untag f3x-build-side-nnnnn offending-nvr

for every package that's blocking the update.

Fabio
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat
> http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> libguestfs lets you edit virtual machines.  Supports shell scripting,
> bindings from many languages.  http://libguestfs.org
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Clement Verna]

On Wed, 8 Apr 2020 at 11:38, Fabio Valentini  wrote:
> On Wed, Apr 8, 2020, 11:27 Richard W.M. Jones  wrote:
> > On Tue, Apr 07, 2020 at 05:36:44PM -0400, Mohan Boddu wrote:
> > > On Tue, Apr 7, 2020 at 2:56 AM Richard W.M. Jones
> >
> > wrote:
> > > > https://bodhi.fedoraproject.org/updates/FEDORA-2020-d120de33c5
> > >
> > > I guess it should be a fixed in bodhi. But for now you can remove the
> > > builds that it's complaining about in the update.
> > >
> > > https://github.com/fedora-infra/bodhi/issues/3991
> >
> > So it is complaining:
> >
> > "This update cannot be pushed to stable. These builds
> >   brltty-6.0-14.fc33, graphviz-2.42.2-10.fc33 have a more recent build
> >   in koji's f33 tag."
> >
> > Those builds could just be removed before pushing.  But I don't see a
> > way to remove the builds from the UI.
> >
> > (This update took 3 days of work to create so I _really_ do not want
> > to have to build the whole thing again.)
>
> You should be able to run
>
> koji untag f3x-build-side-nnnnn offending-nvr
>
> for every package that's blocking the update.

That will not work, since Bodhi does not react on changes from koji the
update will still be stuck in Pending

You need to edit the update and remove these builds. You can click on the
"edit" button on the left side of the update status, then you will have a
list builds with an "x" on the right side of each build. Clicking on that
'x' will remove the build from the update (see attached screenshot).

Otherwise you can use the bodhi updates edit --removebuilds command, see
man page
https://bodhi.fedoraproject.org/docs/user/man_pages/bodhi.html#updates
> Fabio
> > Rich.
> >
> > --
> > Richard Jones, Virtualization Group, Red Hat
> > http://people.redhat.com/~rjones
> > Read my programming and virtualization blog: http://rwmj.wordpress.com
> > libguestfs lets you edit virtual machines.  Supports shell scripting,
> > bindings from many languages.  http://libguestfs.org
> > _______________________________________________
> > devel mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> > https://lists.fedoraproject.org/archives/list/[e-mail]
>
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Richard W.M. Jones]

On Wed, Apr 08, 2020 at 11:57:27AM +0200, Clement Verna wrote:
> > > > > https://bodhi.fedoraproject.org/updates/FEDORA-2020-d120de33c5
>
> You need to edit the update and remove these builds. You can click on the
> "edit" button on the left side of the update status, then you will have a
> list builds with an "x" on the right side of each build. Clicking on that
> 'x' will remove the build from the update (see attached screenshot).

I don't appear to have the "x" (nor the box with "Fedora 33").
Permissions?
> Otherwise you can use the bodhi updates edit --removebuilds command, see
> man page
> https://bodhi.fedoraproject.org/docs/user/man_pages/bodhi.html#updates

$ bodhi updates edit --removebuilds brltty-6.0-14.fc33,graphviz-2.42.2-10.fc33 FEDORA-2020-d120de33c5 
Cannot find release associated with build: alt-ergo-2.0.0-12.fc33, tags: ['f33-build-side-20855']

(By the way that command failed, but still exited with code 0, which
is something that happens quite a lot across our tooling and causes me
endless trouble with automating things because I have to
"screen-scrape" output to find errors.)

Rich.

[Clement Verna]

On Wed, 8 Apr 2020 at 13:15, Richard W.M. Jones  wrote:
> On Wed, Apr 08, 2020 at 11:57:27AM +0200, Clement Verna wrote:
> > > > > > https://bodhi.fedoraproject.org/updates/FEDORA-2020-d120de33c5
> >
> > You need to edit the update and remove these builds. You can click on the
> > "edit" button on the left side of the update status, then you will have a
> > list builds with an "x" on the right side of each build. Clicking on that
> > 'x' will remove the build from the update (see attached screenshot).
>
> I don't appear to have the "x" (nor the box with "Fedora 33").
> Permissions?

Hum ok sorry, so since the builds are in the side tag you indeed need to
remove them from the tag as Fabio pointed out (should have read that more
carefully). So

koji untag f3x-build-side-nnnnn offending-nvr

Once the builds are not in the tag, you still need to edit the update and
refresh the list of builds (There is a refresh/update button next to the
name of the side tag in the edit update from). That will fetch the list of
builds in the side tag.

Better documentation on how to do this is need :(
> > Otherwise you can use the bodhi updates edit --removebuilds command, see
> > man page
> > https://bodhi.fedoraproject.org/docs/user/man_pages/bodhi.html#updates
>
> $ bodhi updates edit --removebuilds
> brltty-6.0-14.fc33,graphviz-2.42.2-10.fc33 FEDORA-2020-d120de33c5
> Cannot find release associated with build: alt-ergo-2.0.0-12.fc33, tags:
> ['f33-build-side-20855']
>
> (By the way that command failed, but still exited with code 0, which
> is something that happens quite a lot across our tooling and causes me
> endless trouble with automating things because I have to
> "screen-scrape" output to find errors.)

Yes we have https://github.com/fedora-infra/bodhi/issues/3868,
contributions welcomed :-)
> Rich.
>
> --
> Richard Jones, Virtualization Group, Red Hat
> http://people.redhat.com/~rjones
> Read my programming and virtualization blog: http://rwmj.wordpress.com
> virt-df lists disk usage of guests without needing to install any
> software inside the virtual machine.  Supports Linux and Windows.
> http://people.redhat.com/~rjones/virt-df/

[Richard W.M. Jones]

On Wed, Apr 08, 2020 at 01:43:37PM +0200, Clement Verna wrote:
> On Wed, 8 Apr 2020 at 13:15, Richard W.M. Jones  wrote:
> > On Wed, Apr 08, 2020 at 11:57:27AM +0200, Clement Verna wrote:
> > > > > > > https://bodhi.fedoraproject.org/updates/FEDORA-2020-d120de33c5
>
> koji untag f3x-build-side-nnnnn offending-nvr
>
> Once the builds are not in the tag, you still need to edit the update and
> refresh the list of builds (There is a refresh/update button next to the
> name of the side tag in the edit update from). That will fetch the list of
> builds in the side tag.

Thanks - I believe I've been able to do that now.

The update is still in pending, but I'll leave it for a bit.

Rich.

[Richard W.M. Jones]

Oh actually it's just started being pushed to stable.  Thanks all.

Rich.

[Fabio Valentini]

On Wed, Apr 8, 2020 at 11:58 AM Clement Verna  wrote:
> On Wed, 8 Apr 2020 at 11:38, Fabio Valentini  wrote:
> > On Wed, Apr 8, 2020, 11:27 Richard W.M. Jones  wrote:
> > > On Tue, Apr 07, 2020 at 05:36:44PM -0400, Mohan Boddu wrote:
> > > > On Tue, Apr 7, 2020 at 2:56 AM Richard W.M. Jones  wrote:
> > > > > https://bodhi.fedoraproject.org/updates/FEDORA-2020-d120de33c5
> > > >
> > > > I guess it should be a fixed in bodhi. But for now you can remove the
> > > > builds that it's complaining about in the update.
> > > >
> > > > https://github.com/fedora-infra/bodhi/issues/3991
> > >
> > > So it is complaining:
> > >
> > > "This update cannot be pushed to stable. These builds
> > >   brltty-6.0-14.fc33, graphviz-2.42.2-10.fc33 have a more recent build
> > >   in koji's f33 tag."
> > >
> > > Those builds could just be removed before pushing.  But I don't see a
> > > way to remove the builds from the UI.
> > >
> > > (This update took 3 days of work to create so I _really_ do not want
> > > to have to build the whole thing again.)
> >
> > You should be able to run
> >
> > koji untag f3x-build-side-nnnnn offending-nvr
> >
> > for every package that's blocking the update.
>
> That will not work, since Bodhi does not react on changes from koji the update will still be stuck in Pending
>
> You need to edit the update and remove these builds. You can click on the "edit" button on the left side of the update status, then you will have a list builds with an "x" on the right side of each build. Clicking on that 'x' will remove the build from the update (see attached screenshot).
>
> Otherwise you can use the bodhi updates edit --removebuilds command, see man page https://bodhi.fedoraproject.org/docs/user/man_pages/bodhi.html#updates

Nope, that won't work - you can't edit builds for updates that were
created from side tags - neither from the Web UI (no buttons), nor
from the API / CLI ... - they only have the "from_tag" attribute, but
it's currently impossible (as far as I can tell) to "refresh" those
side tag updates :/ That's why I suggested untagging in koji. But if
bodhi doesn't see those changes, that won't work either

Fabio
> > Fabio
> > > Rich.
> > >
> > > --
> > > Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
> > > Read my programming and virtualization blog: http://rwmj.wordpress.com
> > > libguestfs lets you edit virtual machines.  Supports shell scripting,
> > > bindings from many languages.  http://libguestfs.org
> > > _______________________________________________
> > > devel mailing list -- [e-mail]
> > > To unsubscribe send an email to [e-mail]
> > > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
> >
> > devel mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]
>
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/[e-mail]

[Clement Verna]

On Wed, 8 Apr 2020 at 13:42, Fabio Valentini  wrote:
> On Wed, Apr 8, 2020 at 11:58 AM Clement Verna 
> wrote:
> > On Wed, 8 Apr 2020 at 11:38, Fabio Valentini
>
> wrote:
> > > On Wed, Apr 8, 2020, 11:27 Richard W.M. Jones
>
> wrote:
> > > > On Tue, Apr 07, 2020 at 05:36:44PM -0400, Mohan Boddu wrote:
> > > > > On Tue, Apr 7, 2020 at 2:56 AM Richard W.M. Jones
>
> wrote:
> > > > > > https://bodhi.fedoraproject.org/updates/FEDORA-2020-d120de33c5
> > > > >
> > > > > I guess it should be a fixed in bodhi. But for now you can remove the
> > > > > builds that it's complaining about in the update.
> > > > >
> > > > > https://github.com/fedora-infra/bodhi/issues/3991
> > > >
> > > > So it is complaining:
> > > >
> > > > "This update cannot be pushed to stable. These builds
> > > >   brltty-6.0-14.fc33, graphviz-2.42.2-10.fc33 have a more recent build
> > > >   in koji's f33 tag."
> > > >
> > > > Those builds could just be removed before pushing.  But I don't see a
> > > > way to remove the builds from the UI.
> > > >
> > > > (This update took 3 days of work to create so I _really_ do not want
> > > > to have to build the whole thing again.)
> > >
> > > You should be able to run
> > >
> > > koji untag f3x-build-side-nnnnn offending-nvr
> > >
> > > for every package that's blocking the update.
> >
> > That will not work, since Bodhi does not react on changes from koji the
>
> update will still be stuck in Pending
> > You need to edit the update and remove these builds. You can click on
>
> the "edit" button on the left side of the update status, then you will have
> a list builds with an "x" on the right side of each build. Clicking on that
> 'x' will remove the build from the update (see attached screenshot).
> > Otherwise you can use the bodhi updates edit --removebuilds command, see
>
> man page
> https://bodhi.fedoraproject.org/docs/user/man_pages/bodhi.html#updates
>
> Nope, that won't work - you can't edit builds for updates that were
> created from side tags - neither from the Web UI (no buttons), nor
> from the API / CLI ... - they only have the "from_tag" attribute, but
> it's currently impossible (as far as I can tell) to "refresh" those
> side tag updates :/ That's why I suggested untagging in koji. But if
> bodhi doesn't see those changes, that won't work either

Yeah I skipped the fact that the builds were in the side tag, so indeed
that needs to be done in koji, then edit the update and refresh the list of
builds from the UI. I don't think the CLI supports that currently.
> Fabio
> > > Fabio
> > > > Rich.
> > > >
> > > > --
> > > > Richard Jones, Virtualization Group, Red Hat
>
> http://people.redhat.com/~rjones
> > > > Read my programming and virtualization blog: http://rwmj.wordpress.com
> > > > libguestfs lets you edit virtual machines.  Supports shell scripting,
> > > > bindings from many languages.  http://libguestfs.org
> > > > _______________________________________________
> > > > devel mailing list -- [e-mail]
> > > > To unsubscribe send an email to [e-mail]
> > > > Fedora Code of Conduct:
>
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > > List Guidelines:
>
> https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > > List Archives:
>
> https://lists.fedoraproject.org/archives/list/[e-mail]
> > > devel mailing list -- [e-mail]
> > > To unsubscribe send an email to [e-mail]
> > > Fedora Code of Conduct:
>
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives:
>
> https://lists.fedoraproject.org/archives/list/[e-mail]
> > devel mailing list -- [e-mail]
> > To unsubscribe send an email to [e-mail]
> > Fedora Code of Conduct:
>
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
>
> https://lists.fedoraproject.org/archives/list/[e-mail]
> _______________________________________________
> devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[e-mail]

[Pierre-Yves Chibon]

On Wed, Apr 08, 2020 at 01:41:23PM +0200, Fabio Valentini wrote:
> On Wed, Apr 8, 2020 at 11:58 AM Clement Verna  wrote:
> > On Wed, 8 Apr 2020 at 11:38, Fabio Valentini  wrote:
> > > On Wed, Apr 8, 2020, 11:27 Richard W.M. Jones  wrote:
> > > > On Tue, Apr 07, 2020 at 05:36:44PM -0400, Mohan Boddu wrote:
> > > > > On Tue, Apr 7, 2020 at 2:56 AM Richard W.M. Jones  wrote:
> > > > > > https://bodhi.fedoraproject.org/updates/FEDORA-2020-d120de33c5
> > > > >
> > > > > I guess it should be a fixed in bodhi. But for now you can remove the
> > > > > builds that it's complaining about in the update.
> > > > >
> > > > > https://github.com/fedora-infra/bodhi/issues/3991
> > > >
> > > > So it is complaining:
> > > >
> > > > "This update cannot be pushed to stable. These builds
> > > >   brltty-6.0-14.fc33, graphviz-2.42.2-10.fc33 have a more recent build
> > > >   in koji's f33 tag."
> > > >
> > > > Those builds could just be removed before pushing.  But I don't see a
> > > > way to remove the builds from the UI.
> > > >
> > > > (This update took 3 days of work to create so I _really_ do not want
> > > > to have to build the whole thing again.)
> > >
> > > You should be able to run
> > >
> > > koji untag f3x-build-side-nnnnn offending-nvr
> > >
> > > for every package that's blocking the update.
> >
> > That will not work, since Bodhi does not react on changes from koji the update will still be stuck in Pending
> >
> > You need to edit the update and remove these builds. You can click on the "edit" button on the left side of the update status, then you will have a list builds with an "x" on the right side of each build. Clicking on that 'x' will remove the build from the update (see attached screenshot).
> >
> > Otherwise you can use the bodhi updates edit --removebuilds command, see man page https://bodhi.fedoraproject.org/docs/user/man_pages/bodhi.html#updates
>
> Nope, that won't work - you can't edit builds for updates that were
> created from side tags - neither from the Web UI (no buttons), nor
> from the API / CLI ... - they only have the "from_tag" attribute, but
> it's currently impossible (as far as I can tell) to "refresh" those
> side tag updates :/ That's why I suggested untagging in koji. But if
> bodhi doesn't see those changes, that won't work either

There is or was a button to refresh the side-tag updates. The idea being that
you indeed untag from the side-tag and refresh the update so that the list of
builds can be updated/refreshed.
If we're not showing that refresh button that sounds like a bug :s

Pierre