On Mon, 2011-01-24 at 08:27 -0500, Kamil Paral wrote:
----- Original Message -----
> This replaces the method previously used in
> lib/autotest/site_packages.py. To
> avoid non-standard file permissions for tests in
> /usr/share/autotest/client/site_tests, this patch creates
> site_autotest.py to
> perform additional steps when preparing an autotest client for jobs.
> In
> site_autotest.py, we extend the existing BaseAutotest._install()
> method to
> perform several tasks prior to the standard autotest client
> preparation. Those
> tasks include:
>
> * Optionally install additional yum repositories (based on optional
> global_config.ini option)
> * Install, or upgrade, autoqa
> * Copy all files in /etc/autoqa to the client
>
> This patch also updates autoqa.spec and Makefile to include the new
> file.
Outstanding, you win the 'Patch Of The Week' prize! :-) This is really
great, it's much better hack than mine was.
Haha, no ... this is just another hack :)
I have tested a little and found out following:
1. rsync must be installed on both server and the client in order
to work, otherwise it falls back to scp. Maybe we could add rsync
as autotest dependency?
Good call, I can do that.
2. In order for scp to work, openssh-clients must be installed
on both server and the client. (And of course also openssh-server
on the client).
Sounds like another autotest-server dependency?
3. There are some differences how scp and rsync handle destination
directories when they exist/don't exist. The only reliable way when
the transfer worked well for both scp and rsync is to delete and
re-create the destination directory prior to the transfer. This is
probably a bug in autotest, it should handle it internally.
Agreed, it seems autotest expects to handle this for us as well. More
comments on this below.
This is my current implementation:
class SiteAutotest(BaseAutotest):
def _install(self, host=None, autodir=None, use_autoserv=True,
use_packaging=True):
# NOTE: Due to inconsistent rsync/scp behavior inside autotest, all
# the transferred directories must end with a slash and must be removed
# and re-created at the destination machine prior to the transfer
# copy autoqa config files
logging.debug("Copying autoqa configuration files")
dir_ = '/etc/autoqa/'
host.run('rm -rf %s; mkdir -p %s' % (dir_, dir_))
host.send_file(dir_, dir_)
# copy autoqa library
logging.debug("Copying autoqa library")
dir_ = '/usr/lib/python2.7/site-packages/autoqa/'
I don't think this will work since it hard-codes the python version
string. Oh nm ... you noted this below as well.
host.run('rm -rf %s; mkdir -p %s' % (dir_, dir_))
host.send_file(dir_, dir_)
The delete_dest=True argument for autotest host.send_file() attempts to
handle this for us. In fact, they use the same/similar commands that
you list above.
delete_dest: if this is true, the command will also clear
out any old files at dest that are not in the
source
I didn't do exhaustive testing to ensure that delete_dest was honored,
but can queue that up shortly. I'd like to have the built-in autotest
delete_dest= support working, but we can certainly use the above if the
autotest support is broken or not patchable in the short-term.
logging.info("Installation of autoqa completed")
# Finally, continue installation by calling our parent
super(SiteAutotest, self)._install(host, autodir, use_autoserv, use_packaging)
You surely noticed that I don't install autoqa package anymore.
That's right, I believe we don't want to install that package.
Automatic installation has this drawbacks:
I. For the production server: When we release a new autoqa release,
all the clients will be immediately updated to this new version,
before we have even updated the production server. That can cause
quite some problems.
Good point. I don't see this as an entirely bad thing, we want to run
the latest stable content ... but you are right, they should match on
the server+client. I could adjust this support so that the autoqa
version installed matches the version installed on the server. Hrmm, no
that's not good either. I could adjust it so that it *only* installs
the package ... no upgrades. That would solve the first-time setup use
case ... and still leave upgrading clients to the administrator.
II. For the staging server: We couldn't operate a staging server
like
this. We need to have unreleased code deployed to the clients, not
the latest release.
I believe we can address this ... see below.
Therefore I believe we need to find a way to ensure that autoqa
versions
on the server and on the client always match. Easiest solution is simply
to copy the config files and the autoqa library, as I have done in the
code above. But there are a few issues to solve:
a) The destination directory of autoqa library depends on the
python version used. We can't hardcode it. Maybe we can transfer the
library source code and do 'python setup.py install'?
b) The autoqa package dependencies are not installed that way.
We need to find a way to ensure all the dependencies are installed.
Any ideas?
How about just using packaging instead of recreating all the "packaging"
tasks manually. Seems like it would be beneficial to only maintain one
"packaging" scenario. My assumption with the original patch was that
all code would be packaged and published in a repo somewhere, even for a
staging instance. For staging code, some cronjob would git checkout,
attempt a build and update the 'testing' repo [1] with the build
results. This mimics the behavior for official Fedora updates and lets
existing mechanisms deal with packaging.
The goal with the original patchset was from an idea you gave me.
Basically, making it *really* effortless to add a new system and start
scheduling tests to it. I was able to hide all the package install and
configuration into the site_autotest.py. While it seemed to work, I'm
not convinced we want to put all AutoQA setup into this method. I liked
it, but as coded, anyone with the autoqa and autotest-server packages
installed will get autoqa installed+configured on *all* of their test
clients. I may need to make this support optional by way of a
global_config.ini setting.
So my recommendation would be ...
1. Adjust autotest-server %requires to add rsync and
openssh-{clients,server}
2. Adjust patchset to make autoqa install+config optional based on
global_config.ini setting
3. Adjust patchset to only install autoqa ... not upgrade
4. Investigate whether delete_dest=True is working ... file bug/fix
as needed
5. Further discuss how best to deploy staging packages ... my
thought is to just use rpms and repos.
How's that sound?
Thanks,
James
[1]
http://repos.fedorapeople.org/repos/fedora-qa/autoqa/fedora-autoqa-testin...