Yesterday's compose stalled. So, no compose yesterday. The day before
that:
2017-08-28 03:19:38 [...] Fedora-27-20170828.n.0 started [...]
2017-08-28 20:26:31 [...] Fedora-27-20170828.n.0 FINISHED_INCOMPLETE [...]
That's *over 17 hours*. We're not too far off from actually literally
taking more than 24 hours to make a day's compose.
If this were something like one, two, or even four hours, when there is
a problem, we could fix that and restart without losing a whole day
(which often cascades into months).
I understand that a large part of the slowness is simply IO, from
moving a bunch of packages around, and there are tens of thousands of
packages. But, the day-to-day changeset isn't that big. Looking back at
F26 compose reports from June, I see changes ranging from a few
megabytes up to 3GB or so. Even processing at at at one megabyte per
second, that should be less than an hour even on those busy days. (Plus
whatever time for creating images, which I understand has its own
things with RPM scripts being slow and etc.)
Anyway, there has to be some low-hanging fruit here. Is there something
we can optimize by caching day-to-day, or caching as new packages are
built?
Is there a place we can thrown money at the problem and do this
all on a machine with RAID 10 SSDs or something?
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader