Just wanted to update everyone on our current status.
As you know from the other thread:
* The mass rebuild happened and finished. * The mass rebuild side tag was merged into the f30-pending tag (to make sure everything was signed). * In the middle of the night our autosign box stopped processing. * The next morning this was noticed and a request for a replacement motherboard was sent. * Since that was going to take a day, we setup another machine to do autosigning. * That machine started processing the backlog, but also stopped processing a few times (waiting on koji). * Finally we set back up the normal listening process and I retagged all the f30-pending builds to it would "see" they needed processing.
Currently it's processing along pretty fast, but it does make sure koji has the signed rpms written out, which can take a few seconds on larger packages.
I'm hopeful that it will catch back up today and we can go back to normal.
There will likely be a short outage next week to move back to the now replaced hardware, but that should be only a few minutes.
kevin
On pe, 08 helmi 2019, Kevin Fenzi wrote:
Just wanted to update everyone on our current status.
As you know from the other thread:
- The mass rebuild happened and finished.
- The mass rebuild side tag was merged into the f30-pending tag
(to make sure everything was signed).
- In the middle of the night our autosign box stopped processing.
- The next morning this was noticed and a request for a replacement
motherboard was sent.
- Since that was going to take a day, we setup another machine to do
autosigning.
- That machine started processing the backlog, but also stopped
processing a few times (waiting on koji).
- Finally we set back up the normal listening process and I retagged all
the f30-pending builds to it would "see" they needed processing.
Currently it's processing along pretty fast, but it does make sure koji has the signed rpms written out, which can take a few seconds on larger packages.
I'm hopeful that it will catch back up today and we can go back to normal.
Looks like my build got caught up with this: $ fedpkg build Building freeipa-4.7.2-3.fc30 for rawhide Created task: 32666402 Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=32666402 Watching tasks (this may be safely interrupted)... 32666402 build (rawhide, /rpms/freeipa.git:4dd28889a240255e780926cb9d4b2aa2ef3d3432): free 32666402 build (rawhide, /rpms/freeipa.git:4dd28889a240255e780926cb9d4b2aa2ef3d3432): free -> open (buildvm-15.phx2.fedoraproject.org) 32666403 buildSRPMFromSCM (/rpms/freeipa.git:4dd28889a240255e780926cb9d4b2aa2ef3d3432): open (buildvm-s390x-06.s390.fedoraproject.org) 32666480 buildArch (freeipa-4.7.2-3.fc30.src.rpm, s390x): open (buildvm-s390x-13.s390.fedoraproject.org) 32666478 buildArch (freeipa-4.7.2-3.fc30.src.rpm, aarch64): open (buildvm-aarch64-19.arm.fedoraproject.org) 32666477 buildArch (freeipa-4.7.2-3.fc30.src.rpm, ppc64le): open (buildvm-ppc64le-09.ppc.fedoraproject.org) 32666479 buildArch (freeipa-4.7.2-3.fc30.src.rpm, i686): open (buildvm-31.phx2.fedoraproject.org) 32666481 buildArch (freeipa-4.7.2-3.fc30.src.rpm, armv7hl): open (buildvm-armv7-24.arm.fedoraproject.org) 32666476 buildArch (freeipa-4.7.2-3.fc30.src.rpm, x86_64): open (buildhw-07.phx2.fedoraproject.org) 32666403 buildSRPMFromSCM (/rpms/freeipa.git:4dd28889a240255e780926cb9d4b2aa2ef3d3432): open (buildvm-s390x-06.s390.fedoraproject.org) -> closed 0 free 7 open 1 done 0 failed 32666479 buildArch (freeipa-4.7.2-3.fc30.src.rpm, i686): open (buildvm-31.phx2.fedoraproject.org) -> closed 0 free 6 open 2 done 0 failed 32666480 buildArch (freeipa-4.7.2-3.fc30.src.rpm, s390x): open (buildvm-s390x-13.s390.fedoraproject.org) -> closed 0 free 5 open 3 done 0 failed 32666476 buildArch (freeipa-4.7.2-3.fc30.src.rpm, x86_64): open (buildhw-07.phx2.fedoraproject.org) -> closed 0 free 4 open 4 done 0 failed 32666477 buildArch (freeipa-4.7.2-3.fc30.src.rpm, ppc64le): open (buildvm-ppc64le-09.ppc.fedoraproject.org) -> closed 0 free 3 open 5 done 0 failed 32666478 buildArch (freeipa-4.7.2-3.fc30.src.rpm, aarch64): open (buildvm-aarch64-19.arm.fedoraproject.org) -> closed 0 free 2 open 6 done 0 failed 32666402 build (rawhide, /rpms/freeipa.git:4dd28889a240255e780926cb9d4b2aa2ef3d3432): open (buildvm-15.phx2.fedoraproject.org) -> closed 0 free 1 open 7 done 0 failed 32666898 tagBuild (noarch): closed Could not execute build: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
And now I have a build without any tags.
On 2/8/19 11:11 AM, Alexander Bokovoy wrote:
Looks like my build got caught up with this: $ fedpkg build Building freeipa-4.7.2-3.fc30 for rawhide Created task: 32666402 Task info: https://koji.fedoraproject.org/koji/taskinfo?taskID=32666402
...snip...
And now I have a build without any tags.
No, that all looks normal, the build was tagged into f30-pending as expected. So once the autosigner gets to it, it will be signed and moved to f30. So yeah, that is delayed due to the signing queue, but otherwise it looks ok to me.
https://koji.fedoraproject.org/koji/buildinfo?buildID=1206959
kevin
On 08. 02. 19 17:41, Kevin Fenzi wrote:
Just wanted to update everyone on our current status.
...
I'm hopeful that it will catch back up today and we can go back to normal.
Is this somehow relevant?
https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20190209.n...
2019-02-09 08:44:00 [ERROR ] Compose run failed: RPM(s) not found for sigs: ['CFC659B9']. Check log for details. Unsigned packages: dar-2.6.0-2.fc30 dar-debuginfo-2.6.0-2.fc30 dar-debugsource-2.6.0-2.fc30 electrum-3.3.2-3.fc30 libdar-2.6.0-2.fc30 libdar-debuginfo-2.6.0-2.fc30 libdar-devel-2.6.0-2.fc30 mysqltuner-1.7.19-2.git.59e5f40.fc30 openscap-1.3.0_alpha2-2.fc30 openscap-containers-1.3.0_alpha2-2.fc30 openscap-debuginfo-1.3.0_alpha2-2.fc30 openscap-debugsource-1.3.0_alpha2-2.fc30 openscap-devel-1.3.0_alpha2-2.fc30 openscap-engine-sce-1.3.0_alpha2-2.fc30 openscap-engine-sce-debuginfo-1.3.0_alpha2-2.fc30 openscap-engine-sce-devel-1.3.0_alpha2-2.fc30 openscap-perl-1.3.0_alpha2-2.fc30 openscap-perl-debuginfo-1.3.0_alpha2-2.fc30 openscap-python3-1.3.0_alpha2-2.fc30 openscap-scanner-1.3.0_alpha2-2.fc30 openscap-scanner-debuginfo-1.3.0_alpha2-2.fc30 openscap-utils-1.3.0_alpha2-2.fc30 openscap-utils-debuginfo-1.3.0_alpha2-2.fc30 python-oslo-messaging-8.0.0-1.fc29 python-oslo-messaging-doc-8.0.0-1.fc29 python2-oslo-messaging-8.0.0-1.fc29 python2-oslo-messaging-tests-8.0.0-1.fc29 python3-oslo-messaging-8.0.0-1.fc29 python3-oslo-messaging-tests-8.0.0-1.fc29
On 2/9/19 2:00 AM, Miro Hrončok wrote:
On 08. 02. 19 17:41, Kevin Fenzi wrote:
Just wanted to update everyone on our current status.
...
I'm hopeful that it will catch back up today and we can go back to normal.
Is this somehow relevant?
This was caused by my fixing upgrade path on f30. ie, there was an older 'dar' tagged in, so I tagged in the 'newer' (by NVR) one. It was signed, but apparently the written out rpms were garbage collected already.
I'm fixing these up and will fire a new rawhide after.
kevin --
https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20190209.n...
2019-02-09 08:44:00 [ERROR ] Compose run failed: RPM(s) not found for sigs: ['CFC659B9']. Check log for details. Unsigned packages: dar-2.6.0-2.fc30 dar-debuginfo-2.6.0-2.fc30 dar-debugsource-2.6.0-2.fc30 electrum-3.3.2-3.fc30 libdar-2.6.0-2.fc30 libdar-debuginfo-2.6.0-2.fc30 libdar-devel-2.6.0-2.fc30 mysqltuner-1.7.19-2.git.59e5f40.fc30 openscap-1.3.0_alpha2-2.fc30 openscap-containers-1.3.0_alpha2-2.fc30 openscap-debuginfo-1.3.0_alpha2-2.fc30 openscap-debugsource-1.3.0_alpha2-2.fc30 openscap-devel-1.3.0_alpha2-2.fc30 openscap-engine-sce-1.3.0_alpha2-2.fc30 openscap-engine-sce-debuginfo-1.3.0_alpha2-2.fc30 openscap-engine-sce-devel-1.3.0_alpha2-2.fc30 openscap-perl-1.3.0_alpha2-2.fc30 openscap-perl-debuginfo-1.3.0_alpha2-2.fc30 openscap-python3-1.3.0_alpha2-2.fc30 openscap-scanner-1.3.0_alpha2-2.fc30 openscap-scanner-debuginfo-1.3.0_alpha2-2.fc30 openscap-utils-1.3.0_alpha2-2.fc30 openscap-utils-debuginfo-1.3.0_alpha2-2.fc30 python-oslo-messaging-8.0.0-1.fc29 python-oslo-messaging-doc-8.0.0-1.fc29 python2-oslo-messaging-8.0.0-1.fc29 python2-oslo-messaging-tests-8.0.0-1.fc29 python3-oslo-messaging-8.0.0-1.fc29 python3-oslo-messaging-tests-8.0.0-1.fc29
On Sat, 2019-02-09 at 10:33 -0800, Kevin Fenzi wrote:
On 2/9/19 2:00 AM, Miro Hrončok wrote:
On 08. 02. 19 17:41, Kevin Fenzi wrote:
Just wanted to update everyone on our current status.
...
I'm hopeful that it will catch back up today and we can go back to normal.
Is this somehow relevant?
This was caused by my fixing upgrade path on f30. ie, there was an older 'dar' tagged in, so I tagged in the 'newer' (by NVR) one. It was signed, but apparently the written out rpms were garbage collected already.
I'm fixing these up and will fire a new rawhide after.
Hi, I don't if it helps but I don't know you notice that Fedora- Rawhide- 20190207 [1] still running and pungi.log [2] last write was 2019-02-08 16:45:52 ( 2 days ago) ...
Best regards,
[1] https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20190207.n...
[2] https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20190207.n...
kevin
https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20190209.n...
2019-02-09 08:44:00 [ERROR ] Compose run failed: RPM(s) not found for sigs: ['CFC659B9']. Check log for details. Unsigned packages: dar-2.6.0-2.fc30 dar-debuginfo-2.6.0-2.fc30 dar-debugsource-2.6.0-2.fc30 electrum-3.3.2-3.fc30 libdar-2.6.0-2.fc30 libdar-debuginfo-2.6.0-2.fc30 libdar-devel-2.6.0-2.fc30 mysqltuner-1.7.19-2.git.59e5f40.fc30 openscap-1.3.0_alpha2-2.fc30 openscap-containers-1.3.0_alpha2-2.fc30 openscap-debuginfo-1.3.0_alpha2-2.fc30 openscap-debugsource-1.3.0_alpha2-2.fc30 openscap-devel-1.3.0_alpha2-2.fc30 openscap-engine-sce-1.3.0_alpha2-2.fc30 openscap-engine-sce-debuginfo-1.3.0_alpha2-2.fc30 openscap-engine-sce-devel-1.3.0_alpha2-2.fc30 openscap-perl-1.3.0_alpha2-2.fc30 openscap-perl-debuginfo-1.3.0_alpha2-2.fc30 openscap-python3-1.3.0_alpha2-2.fc30 openscap-scanner-1.3.0_alpha2-2.fc30 openscap-scanner-debuginfo-1.3.0_alpha2-2.fc30 openscap-utils-1.3.0_alpha2-2.fc30 openscap-utils-debuginfo-1.3.0_alpha2-2.fc30 python-oslo-messaging-8.0.0-1.fc29 python-oslo-messaging-doc-8.0.0-1.fc29 python2-oslo-messaging-8.0.0-1.fc29 python2-oslo-messaging-tests-8.0.0-1.fc29 python3-oslo-messaging-8.0.0-1.fc29 python3-oslo-messaging-tests-8.0.0-1.fc29
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On 2/10/19 8:01 AM, Sérgio Basto wrote:
On Sat, 2019-02-09 at 10:33 -0800, Kevin Fenzi wrote:
On 2/9/19 2:00 AM, Miro Hrončok wrote:
On 08. 02. 19 17:41, Kevin Fenzi wrote:
Just wanted to update everyone on our current status.
...
I'm hopeful that it will catch back up today and we can go back to normal.
Is this somehow relevant?
This was caused by my fixing upgrade path on f30. ie, there was an older 'dar' tagged in, so I tagged in the 'newer' (by NVR) one. It was signed, but apparently the written out rpms were garbage collected already.
I'm fixing these up and will fire a new rawhide after.
Hi, I don't if it helps but I don't know you notice that Fedora- Rawhide- 20190207 [1] still running and pungi.log [2] last write was 2019-02-08 16:45:52 ( 2 days ago) ...
It crashed/died and the status didn't update.
It's not still composing, it will just be stuck in started forever. ;(
Yesterdays failed in an anaconda traceback on the kde live, so I am untagging the new anaconda(s) and restarting it.
Signing should be all caught up as of yesterday.
kevin --
Best regards,
[1] https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20190207.n...
[2] https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20190207.n...
kevin
https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20190209.n...
2019-02-09 08:44:00 [ERROR ] Compose run failed: RPM(s) not found for sigs: ['CFC659B9']. Check log for details. Unsigned packages: dar-2.6.0-2.fc30 dar-debuginfo-2.6.0-2.fc30 dar-debugsource-2.6.0-2.fc30 electrum-3.3.2-3.fc30 libdar-2.6.0-2.fc30 libdar-debuginfo-2.6.0-2.fc30 libdar-devel-2.6.0-2.fc30 mysqltuner-1.7.19-2.git.59e5f40.fc30 openscap-1.3.0_alpha2-2.fc30 openscap-containers-1.3.0_alpha2-2.fc30 openscap-debuginfo-1.3.0_alpha2-2.fc30 openscap-debugsource-1.3.0_alpha2-2.fc30 openscap-devel-1.3.0_alpha2-2.fc30 openscap-engine-sce-1.3.0_alpha2-2.fc30 openscap-engine-sce-debuginfo-1.3.0_alpha2-2.fc30 openscap-engine-sce-devel-1.3.0_alpha2-2.fc30 openscap-perl-1.3.0_alpha2-2.fc30 openscap-perl-debuginfo-1.3.0_alpha2-2.fc30 openscap-python3-1.3.0_alpha2-2.fc30 openscap-scanner-1.3.0_alpha2-2.fc30 openscap-scanner-debuginfo-1.3.0_alpha2-2.fc30 openscap-utils-1.3.0_alpha2-2.fc30 openscap-utils-debuginfo-1.3.0_alpha2-2.fc30 python-oslo-messaging-8.0.0-1.fc29 python-oslo-messaging-doc-8.0.0-1.fc29 python2-oslo-messaging-8.0.0-1.fc29 python2-oslo-messaging-tests-8.0.0-1.fc29 python3-oslo-messaging-8.0.0-1.fc29 python3-oslo-messaging-tests-8.0.0-1.fc29
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Sun, 2019-02-10 at 09:02 -0800, Kevin Fenzi wrote:
On 2/10/19 8:01 AM, Sérgio Basto wrote:
On Sat, 2019-02-09 at 10:33 -0800, Kevin Fenzi wrote:
On 2/9/19 2:00 AM, Miro Hrončok wrote:
On 08. 02. 19 17:41, Kevin Fenzi wrote:
Just wanted to update everyone on our current status.
...
I'm hopeful that it will catch back up today and we can go back to normal.
Is this somehow relevant?
This was caused by my fixing upgrade path on f30. ie, there was an older 'dar' tagged in, so I tagged in the 'newer' (by NVR) one. It was signed, but apparently the written out rpms were garbage collected already.
I'm fixing these up and will fire a new rawhide after.
Hi, I don't if it helps but I don't know you notice that Fedora- Rawhide- 20190207 [1] still running and pungi.log [2] last write was 2019-02-08 16:45:52 ( 2 days ago) ...
It crashed/died and the status didn't update.
It's not still composing, it will just be stuck in started forever. ;(
Yesterdays failed in an anaconda traceback on the kde live, so I am untagging the new anaconda(s) and restarting it.
Is there a bug/link to the logs ?
Signing should be all caught up as of yesterday.
kevin
Best regards,
[1] https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20190207.n...
[2] https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20190207.n...
kevin
https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20190209.n...
2019-02-09 08:44:00 [ERROR ] Compose run failed: RPM(s) not found for sigs: ['CFC659B9']. Check log for details. Unsigned packages: dar-2.6.0-2.fc30 dar-debuginfo-2.6.0-2.fc30 dar-debugsource-2.6.0-2.fc30 electrum-3.3.2-3.fc30 libdar-2.6.0-2.fc30 libdar-debuginfo-2.6.0-2.fc30 libdar-devel-2.6.0-2.fc30 mysqltuner-1.7.19-2.git.59e5f40.fc30 openscap-1.3.0_alpha2-2.fc30 openscap-containers-1.3.0_alpha2-2.fc30 openscap-debuginfo-1.3.0_alpha2-2.fc30 openscap-debugsource-1.3.0_alpha2-2.fc30 openscap-devel-1.3.0_alpha2-2.fc30 openscap-engine-sce-1.3.0_alpha2-2.fc30 openscap-engine-sce-debuginfo-1.3.0_alpha2-2.fc30 openscap-engine-sce-devel-1.3.0_alpha2-2.fc30 openscap-perl-1.3.0_alpha2-2.fc30 openscap-perl-debuginfo-1.3.0_alpha2-2.fc30 openscap-python3-1.3.0_alpha2-2.fc30 openscap-scanner-1.3.0_alpha2-2.fc30 openscap-scanner-debuginfo-1.3.0_alpha2-2.fc30 openscap-utils-1.3.0_alpha2-2.fc30 openscap-utils-debuginfo-1.3.0_alpha2-2.fc30 python-oslo-messaging-8.0.0-1.fc29 python-oslo-messaging-doc-8.0.0-1.fc29 python2-oslo-messaging-8.0.0-1.fc29 python2-oslo-messaging-tests-8.0.0-1.fc29 python3-oslo-messaging-8.0.0-1.fc29 python3-oslo-messaging-tests-8.0.0-1.fc29
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On 2/11/19 3:10 AM, Martin Kolman wrote:
On Sun, 2019-02-10 at 09:02 -0800, Kevin Fenzi wrote:
It crashed/died and the status didn't update.
It's not still composing, it will just be stuck in started forever. ;(
Yesterdays failed in an anaconda traceback on the kde live, so I am untagging the new anaconda(s) and restarting it.
Is there a bug/link to the logs ?
Yeah, sorry:
filed:
https://bugzilla.redhat.com/show_bug.cgi?id=1674605
kevin
Am Freitag, den 08.02.2019, 08:41 -0800 schrieb Kevin Fenzi:
Just wanted to update everyone on our current status.
As you know from the other thread:
- The mass rebuild happened and finished.
- The mass rebuild side tag was merged into the f30-pending tag
(to make sure everything was signed).
- In the middle of the night our autosign box stopped processing.
- The next morning this was noticed and a request for a replacement
motherboard was sent.
- Since that was going to take a day, we setup another machine to do
autosigning.
- That machine started processing the backlog, but also stopped
processing a few times (waiting on koji).
- Finally we set back up the normal listening process and I retagged
all the f30-pending builds to it would "see" they needed processing.
Currently it's processing along pretty fast, but it does make sure koji has the signed rpms written out, which can take a few seconds on larger packages.
I'm hopeful that it will catch back up today and we can go back to normal.
There will likely be a short outage next week to move back to the now replaced hardware, but that should be only a few minutes.
kevin
Thank you for the extra work, Kevin and all other being involved with it!
Just asking as this came to my mind from a different thread:
Looks like that it was only the mass rebuild packages that got stuck
in
the signing queue and other builds were processed normally (they
moved
from f30-pending to f30), so they are going to be out of order when
the
mass rebuild signing/tagging into f30 finishes.
Is there any chance releng could figure out the list of packages affected once the signing/tagging is finished and fix all of them? I
can
already tell that it's affecting at least 40-50 GNOME builds done in that time frame.
Yes, there's a releng script to fix this very case... can run after everything is finished signing/tagging.
Has that issue been adressed already, too?
Björn
On 2/9/19 4:48 AM, Björn 'besser82' Esser wrote:
Am Freitag, den 08.02.2019, 08:41 -0800 schrieb Kevin Fenzi:
Just wanted to update everyone on our current status.
As you know from the other thread:
- The mass rebuild happened and finished.
- The mass rebuild side tag was merged into the f30-pending tag
(to make sure everything was signed).
- In the middle of the night our autosign box stopped processing.
- The next morning this was noticed and a request for a replacement
motherboard was sent.
- Since that was going to take a day, we setup another machine to do
autosigning.
- That machine started processing the backlog, but also stopped
processing a few times (waiting on koji).
- Finally we set back up the normal listening process and I retagged
all the f30-pending builds to it would "see" they needed processing.
Currently it's processing along pretty fast, but it does make sure koji has the signed rpms written out, which can take a few seconds on larger packages.
I'm hopeful that it will catch back up today and we can go back to normal.
There will likely be a short outage next week to move back to the now replaced hardware, but that should be only a few minutes.
kevin
Thank you for the extra work, Kevin and all other being involved with it!
Just asking as this came to my mind from a different thread:
Looks like that it was only the mass rebuild packages that got stuck
in
the signing queue and other builds were processed normally (they
moved
from f30-pending to f30), so they are going to be out of order when
the
mass rebuild signing/tagging into f30 finishes.
Is there any chance releng could figure out the list of packages affected once the signing/tagging is finished and fix all of them? I
can
already tell that it's affecting at least 40-50 GNOME builds done in that time frame.
Yes, there's a releng script to fix this very case... can run after everything is finished signing/tagging.
Has that issue been adressed already, too?
Yes, I already fixed all those.
kevin
Hi,
On 08-02-19 17:41, Kevin Fenzi wrote:
Just wanted to update everyone on our current status.
As you know from the other thread:
- The mass rebuild happened and finished.
- The mass rebuild side tag was merged into the f30-pending tag
(to make sure everything was signed).
- In the middle of the night our autosign box stopped processing.
- The next morning this was noticed and a request for a replacement
motherboard was sent.
- Since that was going to take a day, we setup another machine to do
autosigning.
- That machine started processing the backlog, but also stopped
processing a few times (waiting on koji).
- Finally we set back up the normal listening process and I retagged all
the f30-pending builds to it would "see" they needed processing.
Currently it's processing along pretty fast, but it does make sure koji has the signed rpms written out, which can take a few seconds on larger packages.
I'm hopeful that it will catch back up today and we can go back to normal.
There will likely be a short outage next week to move back to the now replaced hardware, but that should be only a few minutes.
Thank you and the entire infra team for quickly fixing this!
And more in general a big thank you to the entire infra team for all the heroic work you are doing on an almost daily basis.
Regards,
Hans
On Sun, 10 Feb 2019 at 21:17, Kevin Fenzi kevin@scrye.com wrote:
Just wanted to update everyone on our current status.
As you know from the other thread:
- The mass rebuild happened and finished.
Is there anywhere some summary stats about mass update? How many packages has been sent to rebuild vs those which rebuild failed?
kloczek
On 2/11/19 8:13 AM, Tomasz Kłoczko wrote:
On Sun, 10 Feb 2019 at 21:17, Kevin Fenzi kevin@scrye.com wrote:
Just wanted to update everyone on our current status.
As you know from the other thread:
- The mass rebuild happened and finished.
Is there anywhere some summary stats about mass update? How many packages has been sent to rebuild vs those which rebuild failed?
There is:
https://kojipkgs.fedoraproject.org/mass-rebuild/f30-failures.html
and
https://kojipkgs.fedoraproject.org/mass-rebuild/f30-need-rebuild.html
The FTBFS bugs are being filed now I think...
kevin
On Mon, Feb 11, 2019 at 10:35:55AM -0800, Kevin Fenzi wrote:
- The mass rebuild happened and finished.
Is there anywhere some summary stats about mass update? How many packages has been sent to rebuild vs those which rebuild failed?
There is:
https://kojipkgs.fedoraproject.org/mass-rebuild/f30-failures.html
and
https://kojipkgs.fedoraproject.org/mass-rebuild/f30-need-rebuild.html
The FTBFS bugs are being filed now I think...
My packages were caught in the crossfire:
https://bugzilla.redhat.com/show_bug.cgi?id=1675617 https://bugzilla.redhat.com/show_bug.cgi?id=1675618 https://bugzilla.redhat.com/show_bug.cgi?id=1675634
All of them failed on armv7hl, with some package (for example glibc-2.29-1.fc30.1.armv7hl.rpm) missing in the mock step.
How should maintainers proceed in such situation?
On Tue, Feb 12, 2019 at 8:49 AM Jan Pazdziora jpazdziora@redhat.com wrote:
On Mon, Feb 11, 2019 at 10:35:55AM -0800, Kevin Fenzi wrote:
- The mass rebuild happened and finished.
Is there anywhere some summary stats about mass update? How many packages has been sent to rebuild vs those which rebuild failed?
There is:
https://kojipkgs.fedoraproject.org/mass-rebuild/f30-failures.html
and
https://kojipkgs.fedoraproject.org/mass-rebuild/f30-need-rebuild.html
The FTBFS bugs are being filed now I think...
My packages were caught in the crossfire:
https://bugzilla.redhat.com/show_bug.cgi?id=1675617 https://bugzilla.redhat.com/show_bug.cgi?id=1675618 https://bugzilla.redhat.com/show_bug.cgi?id=1675634
All of them failed on armv7hl, with some package (for example glibc-2.29-1.fc30.1.armv7hl.rpm) missing in the mock step.
How should maintainers proceed in such situation?
If the problem is now fixed you should just be able to do "fedpkg build" to rebuild it and it'll proceed as usual.
On Tue, Feb 12, 2019 at 09:19:03AM +0000, Peter Robinson wrote:
On Tue, Feb 12, 2019 at 8:49 AM Jan Pazdziora jpazdziora@redhat.com wrote:
All of them failed on armv7hl, with some package (for example glibc-2.29-1.fc30.1.armv7hl.rpm) missing in the mock step.
How should maintainers proceed in such situation?
If the problem is now fixed you should just be able to do "fedpkg build" to rebuild it and it'll proceed as usual.
Thanks, that passed now.