On Fri, 5 Jun 2015 16:12:04 -0400
Ralph Bean <rbean(a)redhat.com> wrote:
This is one part a product of discussion from the FAD:
and another part response to the questions about requirements from
> We now would like to add some requirements I think. What are
> they? :)
> Do we need to do rawhide/branched composes? Daily?
> Do we need to be able to do run-pungi (RC/TC) test composes?
> Something else?
> Lets find out our requirements before we adjust things.
So, here are some requirements coming out of the FAD:
1) to do rawhide/branched and RC/TC test composes. Hopefully, soon
rawhide composes will look more like RC/TC composes so we won't need
to make the distinction.
2) specifically we want to do this so we can develop/iterate the
3) therefore, we need to be able to do them relatively rapidly
4) furthermore, we need it to be as-like production as possible so we
know what we're testing.
At the FAD, we came up with four different strategies to get there,
with different pros/cons:
# Leverage koji's secondary volume capabilities. #
Apparently, koji can support mounting multiple volumes. We could:
- Mount the prod /mnt/koji read-only in staging and tell staging
koji that this will be it's "secondary volume".
- Dump the prod DB and import it into staging.
- Run a sql script that re-points all the rpm entries from what was
the prod primary volume to now the staging secondary volume (which is
just the prod primary volume, but mounted read-only).
We could run full composes in staging here since all the rpm data
would be available, just like in prod. The one problem would be that
we could not do hardlinks on the read-only mount, so we would need to
make that portion of the compose process optional.. or work around it
some other way. For what it's worth, I personally like this option
Yeah, I guess I do as well. (See further comments below).
Staging koji would drift out of sync with production over time, but
we could re-sync it every week or month or so. We could automate
that with ansible and cron.
This is indeed the downside, but if we did get a working ansible script
anyone could run when we want to sync I think that would help.
Also, this would mean we wouln't have to try and manually keep up
things like tags and such in stg as well as production. (ie, at branch
and other points).
# Leverage writable snapshots from the netapp #
The netapp we use for storage could potentially provide a "writable
snapshot", which could possibly make for an easy-win solution. If we
could snapshot the prod mount and then remount it in staging as a
writable snapshot, (and then also take a dump of the prod database
and import it in staging), then staging could just write on top of
that snapshot, without affecting prod. We would just need to take a
new snapshot every so often to avoid growing beyond our bounds in
If this is even possible (we have to ask), we wouldn't be able to
easily automate syncing from prod, since it seems like it would
require filing a ticket to get a new snapshot each time. Correct me
if I'm wrong, here.
A nice aspect of this solution is that it involves zero koji-specific
compose-specific knowledge and tooling to pull it off.
We had/have a setup like this for our wiki content (thats not part of
the db). However, it did at least at the last time I looked involved
filing a RHIT storage ticket to get the snapshot refreshed, also it
needed some setup and storage.
# Write a custom "koji snapshotter" tool #
This approach involves writing a custom tool that knows about koji.
It would inspect prod koji and copy over a subset of the content and
db relations required to do a compose in staging.
This involves no mount fanciness, but it does involve a high degree
of special development against koji's API. It could work great, but
we'd have to re-do it when koji2.0 comes out (eventually, eventually).
# Just fully copy all 30T of prod koji #
imcleod say that for less than the cost of the FAD, we could get a
storage pod full of disks and have all the space in staging that we
With this, our test composes could potentially be faster than in
prod, which is nice for testing. However, I can't speak to how easy
or not it is or not to get, install, and maintain hardware in our
NAK. It sounds nice and all, but it means that we will have to replace
disks all the time and debug it's setup and have not much of anyone to
fall back on if it dies. Smooge is going to be looking into if we can
move over to the Red Hat Storage setup, but thats not likely short
term. If we can, then this might be a good pilot for it, but it
wouldn't be until next FY at the earliest.
Any input on which of these options is preferable would be
I like option 1 the best.