[PATCH] copy config files to clients in a better way

[Kamil Paral]

From: James Laska <jlaska(a)redhat.com&gt;

This replaces the method previously used in lib/autotest/site_packages.py.
To avoid non-standard file permissions for tests in
/usr/share/autotest/client/site_tests, this patch creates site_autotest.py
to perform additional steps when preparing an autotest client for jobs.
In site_autotest.py, we extend the existing BaseAutotest._install() method
to copy /etc/autoqa directory to the client. In future we can also use
this approach to copy additional files.

This patch also updates autoqa.spec and Makefile to include the new
file.
---
 Makefile                      |    3 ++-
 autoqa.spec                   |    2 +-
 lib/autotest/site_autotest.py |   28 ++++++++++++++++++++++++++++
 lib/autotest/site_packages.py |   34 ----------------------------------
 4 files changed, 31 insertions(+), 36 deletions(-)
 create mode 100644 lib/autotest/site_autotest.py
 delete mode 100644 lib/autotest/site_packages.py

diff --git a/Makefile b/Makefile
index f8d65ad..4a2134a 100644
--- a/Makefile
+++ b/Makefile
@@ -37,7 +37,8 @@ install: build
 	for t in tests/*; do cp -a $$t $(PREFIX)$(TEST_DIR); done
 	install -d $(PREFIX)$(AUTOTEST_DIR)/client/{bin,common_lib}
 	install -m 0644 lib/autotest/site_utils.py $(PREFIX)$(AUTOTEST_DIR)/client/bin/
-	install -m 0644 lib/autotest/site_packages.py
$(PREFIX)$(AUTOTEST_DIR)/client/common_lib/
+	install -d $(PREFIX)$(AUTOTEST_DIR)/server
+	install -m 0644 lib/autotest/site_autotest.py $(PREFIX)$(AUTOTEST_DIR)/server/
 	( cd lib/python; $(PYTHON) setup.py install --skip-build --root $(PREFIX)/ )
 
 build: lib/python/build
diff --git a/autoqa.spec b/autoqa.spec
index a2189bd..4d93d52 100644
--- a/autoqa.spec
+++ b/autoqa.spec
@@ -59,7 +59,7 @@ rm -rf $RPM_BUILD_ROOT
 %{testdir}
 %{hookdir}
 %{_datadir}/autotest/client/bin/site_utils.py*
-%{_datadir}/autotest/client/common_lib/site_packages.py*
+%{_datadir}/autotest/server/site_autotest.py*
 %{python_sitelib}/autoqa*
 
 
diff --git a/lib/autotest/site_autotest.py b/lib/autotest/site_autotest.py
new file mode 100644
index 0000000..adfbc2d
--- /dev/null
+++ b/lib/autotest/site_autotest.py
@@ -0,0 +1,28 @@
+# site_autotest.py - site-specific autotest extensions for use in autoqa
+#
+# this must be copied to autotest/server/, because it is imported by
+# autotest/server/server.py
+#
+# Author: James Laska <jlaska(a)redhat.com&gt;
+
+import logging
+from autotest_lib.server.autotest import BaseAutotest
+
+class SiteAutotest(BaseAutotest):
+    def _install(self, host=None, autodir=None, use_autoserv=True,
+                 use_packaging=True):
+
+        # NOTE: Due to inconsistent rsync/scp behavior inside autotest, all
+        # the transferred directories must end with a slash and must be removed
+        # and re-created at the destination machine prior to the transfer
+
+        # copy autoqa config files
+        logging.debug("Copying autoqa configuration files")
+        srcdir = '/etc/autoqa/'
+        destdir = srcdir
+        host.run('rm -rf %(dir)s; mkdir -p %(dir)s' % {'dir': destdir})
+        host.send_file(srcdir, destdir)
+
+        # Finally, continue installation by calling our parent
+        super(SiteAutotest, self)._install(host, autodir, use_autoserv, use_packaging)
+
diff --git a/lib/autotest/site_packages.py b/lib/autotest/site_packages.py
deleted file mode 100644
index 280e064..0000000
--- a/lib/autotest/site_packages.py
+++ /dev/null
@@ -1,34 +0,0 @@
-# site_packages.py - site-specific autotest packages methods for use in autoqa
-#
-# this must be copied to autotest/client/common_lib, because it is imported by
-# autotest/client/common_lib/packages.py
-#
-# Author: Kamil Paral <kparal(a)redhat.com&gt;
-
-import os, glob, shutil
-from autotest_lib.client.common_lib import base_packages
-
-class SitePackageManager(base_packages.BasePackageManager):
-    def tar_package(self, pkg_name, src_dir, dest_dir, exclude_string=None):
-        '''
-        Same as common_lib.base_packages.tar_package, but it also copies all
-        AutoQA config files into src_dir for our site_tests.
-        '''
-        autoqa_test = (os.path.basename(os.path.dirname(src_dir)) ==
'site_tests')
-        configs = glob.glob('/etc/autoqa/*')
-
-        # if this is our test, copy all config files to src_dir to be tarred
-        # and transferred to client
-        if autoqa_test:
-            for config in configs:
-                shutil.copy(config, src_dir)
-
-        # run the parent method
-        result = super(SitePackageManager, self).tar_package(pkg_name, src_dir, dest_dir,
exclude_string)
-
-        # remove all the config files
-        if autoqa_test:
-            for config in configs:
-                os.remove(os.path.join(src_dir, os.path.basename(config)))
-
-        return result
-- 
1.7.3.5

[James Laska]

On Tue, 2011-01-25 at 16:15 +0100, Kamil Páral wrote:
>  From: James Laska <jlaska(a)redhat.com&gt;
>  
>  This replaces the method previously used in lib/autotest/site_packages.py.
>  To avoid non-standard file permissions for tests in
>  /usr/share/autotest/client/site_tests, this patch creates site_autotest.py
>  to perform additional steps when preparing an autotest client for jobs.
>  In site_autotest.py, we extend the existing BaseAutotest._install() method
>  to copy /etc/autoqa directory to the client. In future we can also use
>  this approach to copy additional files.
>  
>  This patch also updates autoqa.spec and Makefile to include the new
>  file. 
Looks good (and I read the patch properly this time!).

Does it make sense to add a reference to your upstream autotest ticket
in the comments of our site_autotest.py?

Thanks,
James

>  ---
>   Makefile                      |    3 ++-
>   autoqa.spec                   |    2 +-
>   lib/autotest/site_autotest.py |   28 ++++++++++++++++++++++++++++
>   lib/autotest/site_packages.py |   34 ----------------------------------
>   4 files changed, 31 insertions(+), 36 deletions(-)
>   create mode 100644 lib/autotest/site_autotest.py
>   delete mode 100644 lib/autotest/site_packages.py
>  
>  diff --git a/Makefile b/Makefile
>  index f8d65ad..4a2134a 100644
>  --- a/Makefile
>  +++ b/Makefile
>  @@ -37,7 +37,8 @@ install: build
>   	for t in tests/*; do cp -a $$t $(PREFIX)$(TEST_DIR); done
>   	install -d $(PREFIX)$(AUTOTEST_DIR)/client/{bin,common_lib}
>   	install -m 0644 lib/autotest/site_utils.py $(PREFIX)$(AUTOTEST_DIR)/client/bin/
>  -	install -m 0644 lib/autotest/site_packages.py
> $(PREFIX)$(AUTOTEST_DIR)/client/common_lib/
>  +	install -d $(PREFIX)$(AUTOTEST_DIR)/server
>  +	install -m 0644 lib/autotest/site_autotest.py $(PREFIX)$(AUTOTEST_DIR)/server/
>   	( cd lib/python; $(PYTHON) setup.py install --skip-build --root $(PREFIX)/ )
>   
>   build: lib/python/build
>  diff --git a/autoqa.spec b/autoqa.spec
>  index a2189bd..4d93d52 100644
>  --- a/autoqa.spec
>  +++ b/autoqa.spec
>  @@ -59,7 +59,7 @@ rm -rf $RPM_BUILD_ROOT
>   %{testdir}
>   %{hookdir}
>   %{_datadir}/autotest/client/bin/site_utils.py*
>  -%{_datadir}/autotest/client/common_lib/site_packages.py*
>  +%{_datadir}/autotest/server/site_autotest.py*
>   %{python_sitelib}/autoqa*
>   
>  
>  diff --git a/lib/autotest/site_autotest.py b/lib/autotest/site_autotest.py
>  new file mode 100644
>  index 0000000..adfbc2d
>  --- /dev/null
>  +++ b/lib/autotest/site_autotest.py
>  @@ -0,0 +1,28 @@
>  +# site_autotest.py - site-specific autotest extensions for use in autoqa
>  +#
>  +# this must be copied to autotest/server/, because it is imported by
>  +# autotest/server/server.py
>  +#
>  +# Author: James Laska <jlaska(a)redhat.com&gt;
>  +
>  +import logging
>  +from autotest_lib.server.autotest import BaseAutotest
>  +
>  +class SiteAutotest(BaseAutotest):
>  +    def _install(self, host=None, autodir=None, use_autoserv=True,
>  +                 use_packaging=True):
>  +
>  +        # NOTE: Due to inconsistent rsync/scp behavior inside autotest, all
>  +        # the transferred directories must end with a slash and must be removed
>  +        # and re-created at the destination machine prior to the transfer
>  +
>  +        # copy autoqa config files
>  +        logging.debug("Copying autoqa configuration files")
>  +        srcdir = '/etc/autoqa/'
>  +        destdir = srcdir
>  +        host.run('rm -rf %(dir)s; mkdir -p %(dir)s' % {'dir': destdir})
>  +        host.send_file(srcdir, destdir)
>  +
>  +        # Finally, continue installation by calling our parent
>  +        super(SiteAutotest, self)._install(host, autodir, use_autoserv, use_packaging)
>  +
>  diff --git a/lib/autotest/site_packages.py b/lib/autotest/site_packages.py
>  deleted file mode 100644
>  index 280e064..0000000
>  --- a/lib/autotest/site_packages.py
>  +++ /dev/null
>  @@ -1,34 +0,0 @@
>  -# site_packages.py - site-specific autotest packages methods for use in autoqa
>  -#
>  -# this must be copied to autotest/client/common_lib, because it is imported by
>  -# autotest/client/common_lib/packages.py
>  -#
>  -# Author: Kamil Paral <kparal(a)redhat.com&gt;
>  -
>  -import os, glob, shutil
>  -from autotest_lib.client.common_lib import base_packages
>  -
>  -class SitePackageManager(base_packages.BasePackageManager):
>  -    def tar_package(self, pkg_name, src_dir, dest_dir, exclude_string=None):
>  -        '''
>  -        Same as common_lib.base_packages.tar_package, but it also copies all
>  -        AutoQA config files into src_dir for our site_tests.
>  -        '''
>  -        autoqa_test = (os.path.basename(os.path.dirname(src_dir)) ==
> 'site_tests')
>  -        configs = glob.glob('/etc/autoqa/*')
>  -
>  -        # if this is our test, copy all config files to src_dir to be tarred
>  -        # and transferred to client
>  -        if autoqa_test:
>  -            for config in configs:
>  -                shutil.copy(config, src_dir)
>  -
>  -        # run the parent method
>  -        result = super(SitePackageManager, self).tar_package(pkg_name, src_dir,
> dest_dir, exclude_string)
>  -
>  -        # remove all the config files
>  -        if autoqa_test:
>  -            for config in configs:
>  -                os.remove(os.path.join(src_dir, os.path.basename(config)))
>  -
>  -        return result

[Kamil Paral]

----- Original Message -----
>  Looks good (and I read the patch properly this time!).
>  
>  Does it make sense to add a reference to your upstream autotest ticket
>  in the comments of our site_autotest.py? 
Certainly, I remembered the same thing right after hitting Send
(as usual:)).

I'll add that to the patch. If no further concerns, I'll commit it
to master.