Greetings.
Astute followers of rawhide/branched composes will note that we haven't
had to many of them recently that finished successfully. ;(
As of this email the last rawhide one was 2016-08-12 (7 days ago) and
the last branched one was 2016-08-16 ( 3 days ago).
This has various anoying and ill effects:
* The next branched that actually finishes will be 100% signed, but the
one thats on mirrors now from the 16th is not.
* The failing composes still always send the broken deps emails, which
means when we run several composes a day to try and fix things people
will get a bunch of duplicate emails. There's a proposed fix for
this, hopefully landing after alpha so only successfull composes will
send emails.
* Testing out proposed fixes takes a really long time. ;(
The causes for this are several. ;(
* pungi now has the ability to tell when release blocking artifacts are
not successfully composed and fail the entire compose. This means
that in the past we would have had a partial mostly compose, now we
would not get anything.
* The bug where sometimes, seemingly randomly live composes fail has
become more common, and when it hits KDE or Workstation the compose
fails (see above).
https://bugzilla.redhat.com/show_bug.cgi?id=1315541
* There is some issue with the arm composers or rawhide that causes
them to get stuck in I/O wait and never finish composing images. We
have been trying to isolate if this is a kernel issue, userspace on
the builders or rawhide in the chroot. (no bug yet as we aren't sure
where to file it yet).
* On branched the cloud images are/were failing to compose with little
in the way of logs. Since they are release blocking, see first point.
https://bugzilla.redhat.com/show_bug.cgi?id=1365661
Anyhow, thats where we are... we are working on all these issues and
hopefully we can get things rolling again.
kevin