Hi,
On 08-02-19 17:41, Kevin Fenzi wrote:
Just wanted to update everyone on our current status.
As you know from the other thread:
- The mass rebuild happened and finished.
- The mass rebuild side tag was merged into the f30-pending tag
(to make sure everything was signed).
- In the middle of the night our autosign box stopped processing.
- The next morning this was noticed and a request for a replacement
motherboard was sent.
- Since that was going to take a day, we setup another machine to do
autosigning.
- That machine started processing the backlog, but also stopped
processing a few times (waiting on koji).
- Finally we set back up the normal listening process and I retagged all
the f30-pending builds to it would "see" they needed processing.
Currently it's processing along pretty fast, but it does make sure koji has the signed rpms written out, which can take a few seconds on larger packages.
I'm hopeful that it will catch back up today and we can go back to normal.
There will likely be a short outage next week to move back to the now replaced hardware, but that should be only a few minutes.
Thank you and the entire infra team for quickly fixing this!
And more in general a big thank you to the entire infra team for all the heroic work you are doing on an almost daily basis.
Regards,
Hans