Near-Term Taskotron Direction and Priorities

[Tim Flink]

One thing that came up in the QA devel meeting this week was a desire
to have more explicitly stated direction and priorities. This won't be
complete but it should cover the next couple releases.

I suspect that many of you have heard me say this before but my general
philosophy on developing software is:
 1. Make it work
 2. Make it work well
 3. Worry about pretty, fast, making pancakes etc.

At this point, we've got #1 covered. Taskotron as a whole is working
and has been running in staging for several weeks now. It's not
perfect, but it is running. Once we get to #2, we can start looking
into expanding our feature set and adding niceties.

The Fedora 21 branch date is currently scheduled for no earlier than
2014-07-08 [1] and that time is going to fly by before we realize it.
We have about three releases (including the current one) to get
Taskotron ready to replace AutoQA with about a week to make sure that
the production environment is stable and working before Fedora 21
branches.

I've broken shorter term stuff into two sets: one set of higher
priority thing which should be our primary focus for now and one set of
things with lower priority that can be put off for now. This is
purposely written as a higher-level document, so there are
generally no directly associated ticket numbers.

Highest Priorities
==================

The way I see it, aside from testing we have three major areas left
before we can honestly say that AutoQA can be replaced:
 - code
 - documentation
 - infrastructure

Note the use of "must" (cannot release without) versus "should" (want
to have but will consider releasing without).

Code
----

Runner:
 * The runner must work from the local command line (development mode)
   and in full service deployments
 * Command line options must be sane
 * Configuration must be optimized for people installing and running
   libtaskotron locally. Changing playbooks for production deployment
   is a small price to pay for being be able to "just install
   libtaskotron and you're good to go"
 * Common errors/tracebacks should be easy to parse/understand

Trigger:
 * Triggered jobs must be recorded for later scheduling during downtime
 * There should be a reasonable method for finding any missed jobs

Checks:
 * depcheck must function and report properly
 * upgradepath must function and report properly
 * rpmlint must function and report properly

Resultsdb:
 * Results must be stored properly
 * Results must contain a link to the job from which they were generated
 * Results must contain a link to the logs representing their execution
 * Documentation should be sufficient so that users can deploy a dev
   instance with little effort.

Documentation
-------------
 * All directives must be clearly documented with good examples and
   clearly defined parameters
 * Reasonably competent users must be able to understand how to start
   writing tasks
 * Example tasks must be well commented and clearly structured
 * Internal API docs should be well documented
 
Infrastructure
--------------
 * The production instance must be able to handle projected load
 * The staging system must reasonably approximate production
 * Public facing systems (prod, stg, dev) must be controlled by ansible
   and deployed with packages from a public repo
 * Production systems must have a reasonable backup strategy
 * Production systems should be monitored for performance and status
 

Lower Priorities For Now
========================

 * Tooling/process changes for CI or phabricator
 * Resultsdb enhacements that are not required for production deployment
 * fedmsg emission for job state change
 * new checks
 * support tool enhancements (fake_fedorainfra etc.)
 * better report/log generation (like autoqa does)
 * infra documentation
 * new features for checks which are not currently executed in autoqa


I think this is relatively complete but please let me know if something
is missing, incorrect or needs clarification.

Thanks,

Tim

[1] https://fedorahosted.org/fesco/ticket/1178#comment:31

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: not available
URL: <http://lists.fedoraproject.org/pipermail/qa-devel/attachments/20140523/6838b2a5/attachment.sig>