Am Freitag, den 08.02.2019, 08:41 -0800 schrieb Kevin Fenzi:
Just wanted to update everyone on our current status.
As you know from the other thread:
- The mass rebuild happened and finished.
- The mass rebuild side tag was merged into the f30-pending tag
(to make sure everything was signed).
- In the middle of the night our autosign box stopped processing.
- The next morning this was noticed and a request for a replacement
motherboard was sent.
- Since that was going to take a day, we setup another machine to do
autosigning.
- That machine started processing the backlog, but also stopped
processing a few times (waiting on koji).
- Finally we set back up the normal listening process and I retagged
all the f30-pending builds to it would "see" they needed processing.
Currently it's processing along pretty fast, but it does make sure koji has the signed rpms written out, which can take a few seconds on larger packages.
I'm hopeful that it will catch back up today and we can go back to normal.
There will likely be a short outage next week to move back to the now replaced hardware, but that should be only a few minutes.
kevin
Thank you for the extra work, Kevin and all other being involved with it!
Just asking as this came to my mind from a different thread:
Looks like that it was only the mass rebuild packages that got stuck
in
the signing queue and other builds were processed normally (they
moved
from f30-pending to f30), so they are going to be out of order when
the
mass rebuild signing/tagging into f30 finishes.
Is there any chance releng could figure out the list of packages affected once the signing/tagging is finished and fix all of them? I
can
already tell that it's affecting at least 40-50 GNOME builds done in that time frame.
Yes, there's a releng script to fix this very case... can run after everything is finished signing/tagging.
Has that issue been adressed already, too?
Björn