AutoQA Staging Related Proposals

[Tim Flink]

I put together some draft proposals on how I'd like to see us use the
new staging environment.

As a whole, the two proposals will require a decent amount of time and
effort but I've tried to break them up into smaller chunks so that we
can start seeing the benefits sooner rather than later.

This all makes sense in my head and I think that I've put enough detail
into the proposals for them to make sense to everyone else but I'm sure
that I forgot some details. Please let me know what parts of this
aren't clear and I will add detail and/or clarify.

Other comments and/or suggestions are also very welcome.

Thanks,

Tim

AutoQA in a box:
 - https://fedoraproject.org/wiki/User:Tflink/Sandbox:AutoQA_in_a_box
 - The idea here is to automate the creation and updating of the AutoQA
   environment. This would make it easier to setup dev environments and
   simplify the management of the production infrastructure.

AutoQA Staging:
 -
https://fedoraproject.org/wiki/User:Tflink/Sandbox:AutoQA_staging_environ...
 - This is the longer of the two proposals. I think that it will be
   good for us in the long run and ties together some of the things
   that we've talked about for testing AutoQA.

[Kamil Paral]

>  I put together some draft proposals on how I'd like to see us use
> the
>  new staging environment.
>  
>  As a whole, the two proposals will require a decent amount of time and
>  effort but I've tried to break them up into smaller chunks so that we
>  can start seeing the benefits sooner rather than later.
>  
>  This all makes sense in my head and I think that I've put enough
>  detail
>  into the proposals for them to make sense to everyone else but I'm
>  sure
>  that I forgot some details. Please let me know what parts of this
>  aren't clear and I will add detail and/or clarify.
>  
>  Other comments and/or suggestions are also very welcome.
>  
>  Thanks,
>  
>  Tim 
The proposals sound wonderful. Unfortunately I don't have any experience with this
stuff, so I can't really imagine the amount of work involved, the action plan, or even
how to begin. Naive questions below.

>  
>  AutoQA in a box:
>  - https://fedoraproject.org/wiki/User:Tflink/Sandbox:AutoQA_in_a_box
>  - The idea here is to automate the creation and updating of the AutoQA
>  environment. This would make it easier to setup dev environments and
>  simplify the management of the production infrastructure. 
You speak about VMs. Does that mean you suppose everything will be in VM (even autoqa
server, etc)? Not everyone may want to set up the server in a VM, is bare-metal counted
in? Furthermore some clients need to be installed on bare-metal.

Will this approach be useful even for our development purposes? For example I don't
want to re-deploy some client VM image every time I make a change in AutoQA code.

If we use VM, how does that work with infrastructure team managing our machines? Will they
manage only bare-metal machines and we will manage the VMs (upgrading Fedora releases,
etc)?

When speaking of this, I'd like to have client so lightweight that they don't
require ANYTHING have maintained on them. Only the initial SSH key setup would be needed,
and nothing else, ever. Autotest already copies itself from server to client, tests are
copied from server to client, and when we implement #254 also AutoQA library will be
copied from server to client. Then we don't really have to care about clients - no
need to update in production environment, no need to do NFS mounts in dev environment.


>  
>  AutoQA Staging:
>  -
>  https://fedoraproject.org/wiki/User:Tflink/Sandbox:AutoQA_staging_environ...
>  - This is the longer of the two proposals. I think that it will be
>  good for us in the long run and ties together some of the things
>  that we've talked about for testing AutoQA. 
As for Stage 1: Isn't good enough just setting up cron to regularly do 'git pull
&& make clean install' as our "CI process"? If we are going to
examine tests results and autoqa logs manually, what are the benefits of hudson/buildbot?

[Tim Flink]

On Thu, 11 Aug 2011 07:28:09 -0400 (EDT)
Kamil Paral <kparal(a)redhat.com&gt; wrote:> 
>  The proposals sound wonderful. Unfortunately I don't have any
>  experience with this stuff, so I can't really imagine the amount of
>  work involved, the action plan, or even how to begin. Naive questions
>  below. 
My goal was mostly to start communicating my ideas and I deliberately
left out some of the details. If we're interested in doing these things,
I can start putting together some more detailed plans and work
estimates.

>  > AutoQA in a box:
>  > - https://fedoraproject.org/wiki/User:Tflink/Sandbox:AutoQA_in_a_box
>  > - The idea here is to automate the creation and updating of the
>  > AutoQA environment. This would make it easier to setup dev
>  > environments and simplify the management of the production
>  > infrastructure.
>  
>  You speak about VMs. Does that mean you suppose everything will be in
>  VM (even autoqa server, etc)? Not everyone may want to set up the
>  server in a VM, is bare-metal counted in? Furthermore some clients
>  need to be installed on bare-metal. 
I think that it would be just VMs to start with but as you said, we do
have some need for bare metal clients. I can think of some ways to
allow for bare metal but they would take some work and I see them
coming a bit later.

>  Will this approach be useful even for our development purposes? For
>  example I don't want to re-deploy some client VM image every time I
>  make a change in AutoQA code. 
I think that for staging purposes, it would be useful to have new
clients spun up every time but part of my motivation here is to make
development easier. Going to every client and updating them as part of
development is annoying and I want to do away with that.

>  If we use VM, how does that work with infrastructure team managing
>  our machines? Will they manage only bare-metal machines and we will
>  manage the VMs (upgrading Fedora releases, etc)? 
I still have some questions on that one, too and it's on my list of
things to figure out. AFAIK, there will be some impedence mismatch
between what I have in mind and infra's current policies but at the end
of the day, we need something to work.

I think that if we have concrete reasons behind asking for something
that ends up being non-standard, we would have a decent chance of
getting it. However, my plan on this would be to make this work with
the existing infra systems as much as possible - I don't really want to
re-invent the wheel or take on more maintenance responsibilities just
for the sake of being different.

>  When speaking of this, I'd like to have client so lightweight
> that
>  they don't require ANYTHING have maintained on them. Only the initial
>  SSH key setup would be needed, and nothing else, ever. Autotest
>  already copies itself from server to client, tests are copied from
>  server to client, and when we implement #254 also AutoQA library will
>  be copied from server to client. Then we don't really have to care
>  about clients - no need to update in production environment, no need
>  to do NFS mounts in dev environment. 
Yeah, that would make things simpler. I'd still like to be able to
deploy new machines with the CLI and update the autotest server,
though.
 
>  > 
>  > AutoQA Staging:
>  > -
>  > https://fedoraproject.org/wiki/User:Tflink/Sandbox:AutoQA_staging_environ...
>  > - This is the longer of the two proposals. I think that it will be
>  > good for us in the long run and ties together some of the things
>  > that we've talked about for testing AutoQA.
>  
>  As for Stage 1: Isn't good enough just setting up cron to regularly
>  do 'git pull && make clean install' as our "CI process"? If we
> are
>  going to examine tests results and autoqa logs manually, what are the
>  benefits of hudson/buildbot? 
It gets the process started and allows us to start running things like
pylint and any unit tests that we have started gathering. While I do
agree that we aren't using the full potential of a CI system during
stage 1, it is a start and it would be nice to run some automated
checks at the same time.

James mentioned the possiblity of using autotest instead of a more
traditional CI system (hudson/jenkins, buildbot et. al.).