On Tue, Jul 25, 2017 at 12:12:30PM -0400, Dennis Gregorovic wrote:
On 07/25/2017 10:59 AM, Paul W. Frields wrote:
> I'd meant to raise this question last week but it turned out several
> folks were out of pocket who'd probably want to discuss. One of the
> aspects of continuous integration[1] that impacts my team is the
> storage requirement. How much storage is required for keeping test
> results, composed trees and ostrees, and other artifacts? What is
> their retention policy?
>
> A policy of "keep everything ever made, forever" clearly isn't
> scalable. We don't do that in the non-CI realm either, e.g. with
> scratch builds. I do think that we must retain everything we
> officially ship, that's well understood. But atop that, anything we
> keep costs storage, and over time this storage costs money. So we
> need to draw some reasonable line that balances thrift and service.
>
> A. Retention
> ============
>
> The second question is probably a good one to start with, so we can
> answer the first. So we need to answer the retention question for
> some combination of:
>
> 1. candidate builds that fail a CI pipeline
> 2. candidate builds that pass a CI pipeline
> 3. CI composed testables
> * a tree, ISO, AMI, other image, etc. that's a unit
> * ostree change which is more like a delta (AIUI)
> 4. CI generated logs
> 5. ...other stuff I may be forgetting
The other big bucket is packages in the buildroot used to build the
builds. You may want to keep these as well if there is a desire to
be able to rebuild packages at a later point.
Thanks for that. My bet is we'd want to set the retention dial for
those no higher than "when that release goes EOL."
> My general thoughts are that these things are kept forever:
>
> * (2), but only if that build is promoted as an update or as part of a
> shipped tree/ostree/image
> * (3), but only if the output is shipped to users
> * (4), but only if corresponding to an item in (2) or (3)
>
> Outside that, artifacts and logs are kept only for a reasonable amount
> of troubleshooting time. Say 30 days, but I'm not too worried about
> the actual time period. It could be adjusted based on factors we have
> yet to encounter.
How does this proposal compare the existing practice in Fedora?
My initial guess is, pretty well in concept -- but that the *practice*
is that we've not been very aggressive about trimming ancient shipped
stuff. How many people are liable to seek Fedora <= 18 releases at
this point, for example?
--
Paul W. Frields
http://paul.frields.org/
gpg fingerprint: 3DA6 A0AC 6D58 FEC4 0233 5906 ACDB C937 BD11 3717
http://redhat.com/ - - - -
http://pfrields.fedorapeople.org/
The open source story continues to grow:
http://opensource.com