Followup on koji staging setup

[Kevin Fenzi]

On Fri, 5 Jun 2015 16:12:04 -0400
Ralph Bean <rbean at redhat.com> wrote:

> 
> This is one part a product of discussion from the FAD:
> https://fedoraproject.org/wiki/FAD_Release_Tools_and_Infrastructure_2015
> 
> and another part response to the questions about requirements from
> April:
> https://lists.fedoraproject.org/pipermail/infrastructure/2015-April/016121.html
> 
> > We now would like to add some requirements I think. What are
> > they? :)
> >
> > Do we need to do rawhide/branched composes? Daily?
> >
> > Do we need to be able to do run-pungi (RC/TC) test composes?
> >
> > Something else?
> >
> > Lets find out our requirements before we adjust things.
> 
> So, here are some requirements coming out of the FAD:
> 
> 1) to do rawhide/branched and RC/TC test composes.  Hopefully, soon
> rawhide composes will look more like RC/TC composes so we won't need
> to make the distinction.
> 2) specifically we want to do this so we can develop/iterate the
> composition tools
> 3) therefore, we need to be able to do them relatively rapidly
> 4) furthermore, we need it to be as-like production as possible so we
> know what we're testing.
> 
> At the FAD, we came up with four different strategies to get there,
> with different pros/cons:
> 
> # Leverage koji's secondary volume capabilities. #
> 
> Apparently, koji can support mounting multiple volumes.  We could:
> 
>   - Mount the prod /mnt/koji read-only in staging and tell staging
> koji that this will be it's "secondary volume".
> 
>   - Dump the prod DB and import it into staging.
> 
>   - Run a sql script that re-points all the rpm entries from what was
> the prod primary volume to now the staging secondary volume (which is
> just the prod primary volume, but mounted read-only).
> 
> We could run full composes in staging here since all the rpm data
> would be available, just like in prod. The one problem would be that
> we could not do hardlinks on the read-only mount, so we would need to
> make that portion of the compose process optional.. or work around it
> some other way.  For what it's worth, I personally like this option
> the most.

Yeah, I guess I do as well. (See further comments below). 

> Staging koji would drift out of sync with production over time, but
> we could re-sync it every week or month or so.  We could automate
> that with ansible and cron.

This is indeed the downside, but if we did get a working ansible script
anyone could run when we want to sync I think that would help. 
Also, this would mean we wouln't have to try and manually keep up
things like tags and such in stg as well as production. (ie, at branch
and other points). 
 
> # Leverage writable snapshots from the netapp #
> 
> The netapp we use for storage could potentially provide a "writable
> snapshot", which could possibly make for an easy-win solution.  If we
> could snapshot the prod mount and then remount it in staging as a
> writable snapshot, (and then also take a dump of the prod database
> and import it in staging), then staging could just write on top of
> that snapshot, without affecting prod. We would just need to take a
> new snapshot every so often to avoid growing beyond our bounds in
> staging.
> 
> If this is even possible (we have to ask), we wouldn't be able to
> easily automate syncing from prod, since it seems like it would
> require filing a ticket to get a new snapshot each time.  Correct me
> if I'm wrong, here.
> 
> A nice aspect of this solution is that it involves zero koji-specific
> compose-specific knowledge and tooling to pull it off.

We had/have a setup like this for our wiki content (thats not part of
the db). However, it did at least at the last time I looked involved
filing a RHIT storage ticket to get the snapshot refreshed, also it
needed some setup and storage. 

> # Write a custom "koji snapshotter" tool #
> 
> This approach involves writing a custom tool that knows about koji.
> It would inspect prod koji and copy over a subset of the content and
> db relations required to do a compose in staging.
> 
> This involves no mount fanciness, but it does involve a high degree
> of special development against koji's API.  It could work great, but
> we'd have to re-do it when koji2.0 comes out (eventually, eventually).

Seems overcomplex. 

> # Just fully copy all 30T of prod koji #
> 
> imcleod say that for less than the cost of the FAD, we could get a
> storage pod full of disks and have all the space in staging that we
> want: https://www.backblaze.com/blog/backblaze-storage-pod-4/
> 
> With this, our test composes could potentially be faster than in
> prod, which is nice for testing.  However, I can't speak to how easy
> or not it is or not to get, install, and maintain hardware in our
> environment.

NAK. It sounds nice and all, but it means that we will have to replace
disks all the time and debug it's setup and have not much of anyone to
fall back on if it dies. Smooge is going to be looking into if we can
move over to the Red Hat Storage setup, but thats not likely short
term. If we can, then this might be a good pilot for it, but it
wouldn't be until next FY at the earliest. 

> Any input on which of these options is preferable would be
> appreciated.

I like option 1 the best. 

kevin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.fedoraproject.org/pipermail/infrastructure/attachments/20150605/344f34fd/attachment.sig>