Next Steps for Taskotron

[Nick Coghlan]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 07/24/2014 06:16 AM, Tim Flink wrote:
> Now that we're pretty much feature-complete for replacing AutoQA
> with Taskotron, it's time to start talking about what to work on
> next.
> 
> The next big feature that I want to work on is support for
> disposable clients in Taskotron. We've been talking about doing
> this for years in AutoQA but for various reasons, it never
> happened. Disposable clients will pave the way for things like: -
> user submitted tasks - destructive tasks (things which could leave
> the client in a broken state) - tasks that require extra packages
> to be installed

I think there's an opportunity here for you to tap into the existing
investment going into Beaker. While we don't tick all the boxes on
your list *right now*, we do tick several of them, and the rest are
already on our road map for other reasons. We're also highly motivated
to better support Fedora, since it means fewer problems for us down
the road when testing RHEL :)

> While disposable clients may sound simple, there are many parts to
> this. - deciding whether to use something like openstack vs.
> libvirt * The limiting issue here is likely to be that clients need
> to be isolated on the network which may preclude use of the
> existing Fedora infra cloud. I'm not sure that setting up our own
> openstack instance is wise or efficient.

Beaker doesn't do network isolation between test systems yet, but it's
one of the things that becomes feasible with the OpenStack integration
that is part of 0.17. While that integration isn't usable at Red Hat's
scale yet, it should be a viable starting point for Fedora (and the
current scaling issues are temporary - we just need to be smarter
about how we talk to OpenStack).

Another way of tackling this problem would be to use Beaker's guest
recipe feature to install a trusted host which then created isolated
guests (our guest recipe support is managed via a task that runs on
the host, so it's fully customisable without actually changing Beaker
itself).

> - Figuring out how to store logs in a persistent way since clients 
> will be destroyed on a regular basis

Beaker is designed so logs are uploaded to the lab controller rather
than left on the client machines, and the beaker-archive daemon then
allows them to be moved somewhere else if desired. As far as I am
aware, the autotest Beaker support has been merged (we can check with
LMR), so any autotest based tasks should "just work" with the
restraint test harness (he says, optimistically).

> - If we have different client retention policies for various tasks 
> (eg keep generic clients for rpmlint, depcheck etc. around for a 
> while and destroy all other clients after a single use), enhance 
> trigger to support those policies.

Beaker already has a log & job retention mechanism to support Red
Hat's auditing requirements. The default policies are 30, 60 and 120
days, with the deletion process handled by a separate command that
just runs in cron.

> - Support systems for image maintenance/distribution * base images
> should be reasonably up to date * make sure that images are
> distributed to all required places

We don't do native image based provisioning yet, but it's one of our
most frequent requests, and OpenStack Glance addresses the image
library problem.

It's also already possible to do image based provisioning for guest
recipes today. While we don't yet publish a task for it ourselves
(although we have an open task to write one), there's also nothing
preventing anyone interested from writing one themselves.
https://git.beaker-project.org/cgit/beaker-core-tasks/tree/virt/install is
the current task for creating guest VMs, so it would be a matter of
adapting that to ignore the distro metadata and use a specified image
rather than running through the normal Beaker distro install process.

> - Explore whether buildbot's ephemeral slave support will be
> enough for our uses or if that will need enhancements.

This I don't know about. I assume there'd still be at least some work
in teaching it how to submit a job to the Beaker scheduler.

> - Test the snot out of the disposable clients feature * This
> feature will add a decent amount of complexity and we need to make
> sure most of the kinks are worked out

Beaker's been in production use for years, specifically aimed at the
task of integration testing for RHEL. This means you'd be able to
focus your own testing on the integration with Taskotron, rather than
having to debug the core dynamic provisioning capability.

Other things you'd get from going with Beaker for dynamic provisioning:

- - external watchdog timers to abort jobs that take too long
- - anaconda integration for install log capture and detecting failed
installs
- - SSH based access to systems (depending on network config)
- - ability to do fan out integration testing across multiple
architectures with integrated reporting infrastructure
- - existing CI and project management infrastructure rather than having
to do that yourself

> Also, this is for the more immediate future. As the F21 test cycle
> is starting up, we have fewer human resources available to work on 
> Taskotron and the stuff I've listed above is likely to consume a
> decent chunk of time.

Yes, yes it will. Believe me, we know just how time consuming getting
dynamic provisioning to work reliably can be, let alone all the other
things on your list :)

Cheers,
Nick.

- -- 
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane

HSS Provisioning Architect
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJT0H0NAAoJEHEkJo9fMO/LjQcH+gLn+ArBNQ3IIIY+3Zn14io6
KiqfImc9sZpg5Ez0A3CIVQ/utSzDDNkmaPV7Ucs3y57UED2HWgcx4aFwq8n8X7Sa
t78m+cZ6OP1HXPwj7SvmOTxXKIz3cIYaCGK3cxSLWfqSrRlWcVa0KWcSwrcorpbg
8lq1aTo9XiHg/HyISTmcQO78D/EaI5A73vRqcoGFqtdhmiNUS/xkgop3VQ7Ht0aa
eHOOsbmp/TLoCt6aMdyYuO5W6H02M0DXu/c2qkPDYQjiezx2jTIhzr/66wDr8WA2
2LNeRzMBeippKwrEynqJiZT6a2gjiKGoaYnIsi1Y7dXlhEotAnPUoawYqPXSboM=
=aYbV
-----END PGP SIGNATURE-----