----- Original Message -----
> Okay, I'll try this soon ... I'd like to fix autotest if possible.
Sure, we should try to fix the upstream, this is just a workaround.
Reported here:
http://autotest.kernel.org/ticket/45
Will look into this as soon as I can... right now I'm packing for
FUDCon, but will soon get to it.
> > I can imagine this could work just fine for production and
> > staging environment. But there is one use case missing, and that
> > is the development environment. Basically, when I'm developing
> > AutoQA on my server+client(s), I don't want to replicate every
> > change I make onto all those machines. That's really tedious.
> > Also I'm often quite confused where to make this change (on the
> > server/on the client/on all machines). It depends on which part
> > of AutoQA I'm currently working on (hooks/harness/library/tests).
>
> Ah, good use case. Can we think of any reason that the autoqa code on
> the server shouldn't match the code on the clients? Eventually, we
> will
> have mixed client environment (i386/x86_64/$other using f15, f14, f13
> epel5 etc...).
>
> This implies that our code must work for all supported releases.
> Meaning, we can't have autoqa-0.4.4-1.el5 but autoqa-0.4.4-2 for f14.
> I
> don't have a problem with this, I typically build new packages for all
> supported releases. But just want to confirm
That is a good question. Currently I believe our code is exactly the
same for all architectures/operating systems. But in the future, who
knows.
Gosh, I would like to use the rpm versioned approach for the production
server, as you mentioned. But I would also like to use more agile
(and painless) method for staging and development environment.
I have had a brief look at autotest. It has some support for sending
and installing packages tests. Maybe we could use that. I'll have to
study it more [1].
[1]
http://autotest.kernel.org/wiki/PackagingExample
I have started to discuss this issue with Josef, he has some interesting
ideas. I'll send further comments later on.
Below is a patch that makes maintenance-free clients. Installs everything
automatically, transfers config files and the library. You can also
specify further packages to install. The patch is also available at
copylib branch. I admit it's little hackish, but it's just a proof-of-concept.
For the current release I would recommend use this site_autotest.py
approach just for copying config files. That means everything else
would stay the same as it was before (manual installation of autoqa
on all clients needed). For the next release we can discuss the best
approach and implement it.
==================
>From 1888da9e982c31774379c80bb12cf287d10c90fe Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Kamil=20P=C3=A1ral?= <kparal(a)redhat.com>
Date: Tue, 25 Jan 2011 11:37:50 +0100
Subject: [PATCH] copy config files and autoqa library to the client
This will copy config files and autoqa library to the autotest client
prior to the test execution. It will also install any required packages
(like autoqa dependencies etc). That means the autotest clients are now
completely maintanance-free.
---
Makefile | 9 ++++++-
autoqa.spec | 7 ++++-
client-deps-custom.conf | 2 +
client-deps.conf | 4 +++
lib/autotest/site_autotest.py | 53 +++++++++++++++++++++++++++++++++++++++++
lib/autotest/site_packages.py | 34 --------------------------
6 files changed, 72 insertions(+), 37 deletions(-)
create mode 100644 client-deps-custom.conf
create mode 100644 client-deps.conf
create mode 100644 lib/autotest/site_autotest.py
delete mode 100644 lib/autotest/site_packages.py
diff --git a/Makefile b/Makefile
index f8d65ad..26bbd23 100644
--- a/Makefile
+++ b/Makefile
@@ -25,6 +25,8 @@ install: build
[ -f $(PREFIX)/etc/autoqa/autoqa.conf ] || install -m 0644 autoqa.conf
$(PREFIX)/etc/autoqa/
[ -f $(PREFIX)/etc/autoqa/fas.conf ] || install -m 0640 -g autotest fas.conf
$(PREFIX)/etc/autoqa
install -m 0644 repoinfo.conf $(PREFIX)/etc/autoqa/
+ install -m 0644 client-deps.conf $(PREFIX)/etc/autoqa/
+ [ -f $(PREFIX)/etc/autoqa/client-deps-custom.conf ] || install -m 0644
client-deps-custom.conf $(PREFIX)/etc/autoqa
install -d $(PREFIX)$(HOOK_DIR)
for h in hooks/*; do cp -a $$h $(PREFIX)$(HOOK_DIR); done
## git-post-receive setup
@@ -37,8 +39,13 @@ install: build
for t in tests/*; do cp -a $$t $(PREFIX)$(TEST_DIR); done
install -d $(PREFIX)$(AUTOTEST_DIR)/client/{bin,common_lib}
install -m 0644 lib/autotest/site_utils.py $(PREFIX)$(AUTOTEST_DIR)/client/bin/
- install -m 0644 lib/autotest/site_packages.py
$(PREFIX)$(AUTOTEST_DIR)/client/common_lib/
+ install -d $(PREFIX)$(AUTOTEST_DIR)/server
+ install -m 0644 lib/autotest/site_autotest.py $(PREFIX)$(AUTOTEST_DIR)/server/
( cd lib/python; $(PYTHON) setup.py install --skip-build --root $(PREFIX)/ )
+ ## populate client-deps.conf
+ # we don't need autotest and mod_wsgi on the client, filter them out
+ grep '^Requires: ' autoqa.spec | cut -f2 -d ' ' | grep -Ev
'(autotest|mod_wsgi)' \
+ >> $(PREFIX)/etc/autoqa/client-deps.conf
build: lib/python/build
diff --git a/autoqa.spec b/autoqa.spec
index a2189bd..8b8d13f 100644
--- a/autoqa.spec
+++ b/autoqa.spec
@@ -15,6 +15,7 @@ Requires: autotest
Requires: koji
Requires: python-fedora
Requires: mod_wsgi
+BuildRequires: autotest
BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-buildroot
BuildArch: noarch
@@ -36,7 +37,7 @@ make build PYTHON=%{__python}
%install
rm -rf $RPM_BUILD_ROOT
make install PREFIX=$RPM_BUILD_ROOT TEST_DIR=%{testdir} HOOK_DIR=%{hookdir}
PYTHON=%{__python}
-install -m 644 autoqa.conf repoinfo.conf $RPM_BUILD_ROOT%{_sysconfdir}/autoqa/
+install -m 644 autoqa.conf repoinfo.conf client-deps.conf client-deps-custom.conf
$RPM_BUILD_ROOT%{_sysconfdir}/autoqa/
install -m 640 -g autotest fas.conf $RPM_BUILD_ROOT%{_sysconfdir}/autoqa/
@@ -50,7 +51,9 @@ rm -rf $RPM_BUILD_ROOT
%dir %{_sysconfdir}/autoqa
%config(noreplace) %{_sysconfdir}/autoqa/autoqa.conf
%config(noreplace) %{_sysconfdir}/autoqa/fas.conf
+%config(noreplace) %{_sysconfdir}/autoqa/client-deps-custom.conf
%config %{_sysconfdir}/autoqa/repoinfo.conf
+%config %{_sysconfdir}/autoqa/client-deps.conf
%config(noreplace) %{testdir}/rats_sanity/irb.cfg
%config(noreplace) %{_sysconfdir}/httpd/conf.d/autoqa-git-hook.conf
%dir %attr(0775,root,autotest) %{_localstatedir}/cache/autoqa
@@ -59,7 +62,7 @@ rm -rf $RPM_BUILD_ROOT
%{testdir}
%{hookdir}
%{_datadir}/autotest/client/bin/site_utils.py*
-%{_datadir}/autotest/client/common_lib/site_packages.py*
+%{_datadir}/autotest/server/site_autotest.py*
%{python_sitelib}/autoqa*
diff --git a/client-deps-custom.conf b/client-deps-custom.conf
new file mode 100644
index 0000000..8b5e60f
--- /dev/null
+++ b/client-deps-custom.conf
@@ -0,0 +1,2 @@
+# Custom list of packages to install on autotest clients
+# Provide one package name per line
diff --git a/client-deps.conf b/client-deps.conf
new file mode 100644
index 0000000..c42ec36
--- /dev/null
+++ b/client-deps.conf
@@ -0,0 +1,4 @@
+# Auto-generated list of AutoQA dependencies, DO NOT EDIT
+# For specifying custom packages to install on the client, edit client-deps-custom.conf
+openssh-clients
+rsync
diff --git a/lib/autotest/site_autotest.py b/lib/autotest/site_autotest.py
new file mode 100644
index 0000000..8e52e90
--- /dev/null
+++ b/lib/autotest/site_autotest.py
@@ -0,0 +1,53 @@
+# site_autotest.py - site-specific autotest extensions for use in autoqa
+#
+# this must be copied to autotest/server/, because it is imported by
+# autotest/server/server.py
+#
+# Author: James Laska <jlaska(a)redhat.com>
+
+import os
+import logging
+import common
+from autotest_lib.server.autotest import BaseAutotest
+
+class SiteAutotest(BaseAutotest):
+ def _install(self, host=None, autodir=None, use_autoserv=True,
+ use_packaging=True):
+
+ # NOTE: Due to inconsistent rsync/scp behavior inside autotest, all
+ # the transferred directories must end with a slash and must be removed
+ # and re-created at the destination machine prior to the transfer
+
+ # install required packages
+ logging.debug("Installing autoqa dependencies and other required
packages")
+ packages = []
+ for conf in ['/etc/autoqa/client-deps.conf',
'/etc/autoqa/client-deps-custom.conf']:
+ packages += [l.strip() for l in file(conf).readlines() if not
l.strip().startswith('#')]
+ if packages:
+ host.run('yum install -y %s' % ' '.join(packages))
+
+ # copy autoqa config files
+ logging.debug("Copying autoqa configuration files")
+ srcdir = '/etc/autoqa/'
+ destdir = srcdir
+ host.run('rm -rf %(dir)s; mkdir -p %(dir)s' % {'dir': destdir})
+ host.send_file(srcdir, destdir)
+
+ # copy autoqa library
+ logging.debug("Copying autoqa library")
+ import autoqa
+ srcdir = os.path.dirname(autoqa.__file__)
+ assert srcdir, "Can't find autoqa library on the server"
+ srcdir += '/' # trailing slash is important
+ site_pkgs = host.run('python -c \'import sys,re; print filter(lambda s:
re.match("/usr/lib/.*/site-packages$",s),sys.path)[0]\'').stdout
+ site_pkgs = site_pkgs.strip() # remove trailing newline
+ assert site_pkgs, "Can't find python site-packages directory on the
client"
+ destdir = site_pkgs + '/autoqa/'
+ host.run('rm -rf %(dir)s; mkdir -p %(dir)s' % {'dir': destdir})
+ host.send_file(srcdir, destdir)
+
+ logging.info("Installation of autoqa completed")
+
+ # Finally, continue installation by calling our parent
+ super(SiteAutotest, self)._install(host, autodir, use_autoserv, use_packaging)
+
diff --git a/lib/autotest/site_packages.py b/lib/autotest/site_packages.py
deleted file mode 100644
index 280e064..0000000
--- a/lib/autotest/site_packages.py
+++ /dev/null
@@ -1,34 +0,0 @@
-# site_packages.py - site-specific autotest packages methods for use in autoqa
-#
-# this must be copied to autotest/client/common_lib, because it is imported by
-# autotest/client/common_lib/packages.py
-#
-# Author: Kamil Paral <kparal(a)redhat.com>
-
-import os, glob, shutil
-from autotest_lib.client.common_lib import base_packages
-
-class SitePackageManager(base_packages.BasePackageManager):
- def tar_package(self, pkg_name, src_dir, dest_dir, exclude_string=None):
- '''
- Same as common_lib.base_packages.tar_package, but it also copies all
- AutoQA config files into src_dir for our site_tests.
- '''
- autoqa_test = (os.path.basename(os.path.dirname(src_dir)) ==
'site_tests')
- configs = glob.glob('/etc/autoqa/*')
-
- # if this is our test, copy all config files to src_dir to be tarred
- # and transferred to client
- if autoqa_test:
- for config in configs:
- shutil.copy(config, src_dir)
-
- # run the parent method
- result = super(SitePackageManager, self).tar_package(pkg_name, src_dir,
dest_dir, exclude_string)
-
- # remove all the config files
- if autoqa_test:
- for config in configs:
- os.remove(os.path.join(src_dir, os.path.basename(config)))
-
- return result