From: "Kamil Paral" <kparal(a)redhat.com>
To: "AutoQA development" <autoqa-devel(a)lists.fedorahosted.org>
Sent: Tuesday, September 13, 2011 10:40:45 PM
Subject: Re: iso watcher, post-iso-build event, mediakit_sanity test patch for review
> Hi, all,
>
> Included are package for iso watcher, which use lftp to mirror
>
dl.fp.org/pub/alt/stage and kick off mediakit_sanity test,
> package for post-iso-build event, and package for mediakit_sanity
> test
> which test the iso's size, md5, sha256sum, repoclosure
> and file conflicts.
> During the first run of /iso/watcher.py, it will mirror all the isos
> at
dl.fp.org/pub/alt/stage and kick off mekiakit_sanity test,
> after that, it will only mirror the new isos, the tree compose and
> deltaiso are not included in the mirror.
Hello, I haven't examined the code closely, but I have some overall
architecture thoughts:
1. We need to speak to RelEng team about
http://dl.fedoraproject.org/pub/alt/stage/
We can definitely monitor that URL in the worst case, but it would be
so much better if RelEng team could ensure the following for us:
a) Consistent file naming
I have a feeling that currently all the directories there are manually
created (or at least manually named). If there were some systematic
approach that we can rely on, it would help us greatly (e.g. in
determining the latest release).
b) No irrelevant files there
Currently there are not just TC/RC composes, but also "deltaisos/" and
"f16-translation/" directories that we need to blacklist. I am afraid
"stage/" serves as a storage directory for anything needed and those
"clutter" directories will change over time. It would be great to know
that no other unwanted files appear there anytime.
c) Rsync possibility
Using rsync would simplify the mirroring process at least for our Red
Hat offices. I bet we could also use it in AutoQA if desired.
2. Mirroring everything seems overkill. Personally I don't have
Internet bandwidth nor hard disk space to run this test. Can we modify
the watcher so that it detects just the last compose and downloads
that one?
3. Only now I realized that watcher downloading files doesn't help us.
Because the files will be accessible on the AutoQA server, but we need
them on AutoQA clients. That means that your code works as long as you
run it with "--local", but won't work with external clients. It also
means we have to work on ticket #350 "Add support for using file
proxy" (I'll try to do that soon). Ideally we will set up the proxy on
AutoQA server and all clients will be using it. The first test will
download the ISO and the other tests (if there are some) will just use
the cached version (assuming there is a fast network between the
server and the client, otherwise it is of no use).
And an extra code-related question at the end:
4. Do we need labels = ['virt_capable'] in control.autoqa? This test
doesn't seem to be requiring any form of virtualization capabilities.
To sum it up, I believe we will need to do some changes. But
definitely thanks for your work!
Kamil
Kamil,
Thanks for reviewing the code, I will modify that. About dl.fp.o/alt/stage,
I will submit a ticket to relEng. I am glad to see if you can set up the proxy.
Thanks.