Re: Request for test data based off of obfuscated live data
----- "Toshio Kuratomi" <a.badger(a)gmail.com> wrote:
Mike McGrath wrote:
> We're actually in a pretty unique situation in that most of
our data
is
> public anyway, replicating pkgdb and bodhi data for example should
be
> fairly easy. Replicating the fas stuff should be easy too.
>
> We're going to need to replicate not only the data but access to the
data
> and this, to me at least, sounds like another development
environment that
> is more mature then the pt setup but still not as strict as the
staging
> environment.
>
>
Yep, that seems to be where the need fits in.
> What do others think on this? I like the low overhead of the pt
servers
> since people are kind of on their own in getting stuff done and it
doesn't
> cause extra work to the sysadmin-web guys. But there are drawbacks
to it.
>
I'm not sure what's best. There's a lot of problems with doing this
in
a shared development environment. Even if we're controlling the
access
to the data we'd still be more open with it here than in production
or
staging. For instance, people who are not primary fas authors or
system
admins would have access to make modifications to fas. So I think
we'd
still end up wanting to modify the data before it hits this
environment.
We'd also have to devote resources to it.... another db server, host
to
run koji-web,hub,builder, etc. We'd have to update them. We'd have
to
work out conflicts between different developers, for instance if we
work
on CSRF fixes in this environment and it makes developing other apps
like myfedora just flat out fail for a while.
If we can munge the data enough to be comfortable releasing it to the
public, it seems like that would cost us less man hours. However, it
isn't entirely free. We'd still have to make new dumps of data,
modify
it for changes in the data model, etc. Then the developer would
become
responsible for downloading the sanitised data and running it on
their
network. Which is good because it isn't us but bad because it's not
trivial to set all this up.
I would be willing to write scripts and a kickstart file to make this trivial to get a
qemu image or test machine up and running in a couple of hours (mostly waiting for
download and installs to happen). What I was thinking was an environment that setup a
stable fedora infrastructure environment complete with puppet scripts to configure the
services to work with one another and a set of scripts for pulling fresh data, modifying
common pieces of the various dbs (like changing dates to stay current or setting up one of
the users as your test user), and pulling down code from the various source trees for
hacking on particular pieces of the infrastructure while integrating them into the
environment.
A note on the code drop, by requiring the author to modify a spec file if needed in order
to deploy their changes into their environment (revision numbers would be automated)
patches would include spec file changes instead of the maintainer having to sync by hand.
This would also make sure the build files are kept up to date as the author would have to
make their changes work in an RPM environment just to test them as opposed to just
installing from their source tree which often leads to annoying bugs (like missing files
in a distributed tarball). Also by making it easy to generate a patch and submit it to
trac we will get more consistent formatted patches (such as using the VCS's patch
format) and most likely more people getting involved as the overhead shrinks a bit (how
many people want to go to individual trac instances to file a patch?).
--
John (J5) Palmieri
Software Engineer
Red Hat, Inc.