future f12 test days

[Adam Williamson]

On Fri, 2009-04-03 at 03:35 -0500, Callum Lerwick wrote:

> The tests should be automated and run daily, hourly, or even before
> every SCM commit. Ideally, *nothing* would be allowed to be
> committed/pushed into the repo that broke the tests. That would also
> provide motivation to keep the test scripts working...
> 
> All reported bugs should be turned into automated test cases if possible
> and added to the test suite. This ensures once something is fixed it
> never breaks, ever again. Regressions are not an option.

If you actually look at the test cases we've been doing, this is not
practical. How do you automate "you should see a smooth graphical boot
process that fades into GDM"? Or looking at the smoothness of video
playback?

I like the idea of automated testing, but a lot of stuff - especially
the really critical stuff to do with hardware - is not amenable to
automation. Or at least it would be extremely difficult.

> The lack of automated testing in the project is saddening. However a lot
> of the problem is hardware. We really need a diverse hardware testing
> lab. As it is, testing seems to get done on the latest shinyest hardware
> of the month that the (paid) developers just bought, leaving those of us
> who don't buy new hardware every month/year in a dustpile of
> regressions.

Will Woods and Jesse Keating are working very hard on automated QA
topics, but as I said, I just don't think automation can ever be the
answer for some areas.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net