2009-08-24 - 16:00 UTC - Fedora QA meeting Recap

[James Laska]

Full irc transcript available at
https://fedoraproject.org/wiki/QA/Meetings/20090824

= Attendees =

* Adam Williamson (adamw)
* David Pravec (dpravec)
* Edward Kirk (tk009)
* James Laska (jlaska)
* Kamil Paral (kparal)
* Will Woods (wwoods)
* Jesse Keating (f13)
* Kevin Fenzi (nirik)
* (nanonyme)
* Luke Macken (lmacken)
* Sebastian Dziallas (sdziallas)
* Lucas Rodrigues (lmr)

Regrets:
* Liam Li (lili)
* He Rui (rhe)

= Agenda =

*
[https://www.redhat.com/archives/fedora-test-list/2009-August/msg00608.html Proposed meeting agenda]
*
[http://meetbot.fedoraproject.org/fedora-meeting/2009-08-24/fedora-meeting.2009-08-24-15.59.html meetbot summary]

== Previous meeting follow-up ==

* [jlaska] - sync up w/ viking_ice to discuss coordination for upcoming
dracut test day

No updates, wasn't able to sync up with Viking_Ice.  However, I have
been building on his initial test day wiki draft and in communication
with Harald Hoyer on wiki test areas he'd like to see covered.

== Mini post F-12-Alpha recap ==

Jlaska asked the group for feedback on what worked well, and what areas
needed improvement for the recent F-12-Alpha release.

=== What worked well ===

* f13 suggested ... ''there was a ton more communication between releng
and QA''.  Jlaska agreed and requested continued use of tracking
collaborative efforts through the rel-eng trac tickets.  Jlaska also
noted that Liam was having some problems receiving email updates, but
would follow-up with fedora-infrastructure if issues remained.
* Jlaska suggested ... ''another thing I felt worked well was the alpha
blocker bug meetings''.  F13 agreed and also noted that the more
detailed schedule helped.  Adamw wondered if the Friday blocker day was
the best day to schedule.  However, f13 noted it gave time for folks in
EMEA and APAC timezones to get a head start.

=== What needs improvement ===

* Adamw reminded the team ... ''i think the main area for improvement
that presented itself is not in our processes but just the
anaconda-sync-with-releases thing''.  The consensus was this was an area
for improvement, but folks weren't clear how to proceed and what venue
to discuss in.
* Jlaska suggested that the current install test plan could stand to be
made more agile ... so it is a more attainable goal.
* Jlaska noted that ... ''current planned testing is only focused on
installation'' and asked if there were improvements to be made to
increase test coverage for other areas of the distro?  

Adamw noted, that thinking only in an alpha context, the other major
areas ... of concern ... are networking and the yum/rpm stack, and xorg.
Those are areas where most people are vocal about should problems
surface.  xorg is tricky like anaconda as it's quite hardware dependent,
but it might be nice to have a basic setup for testing at least if
intel, nouveau and radeon aren't entirely broken.

F13 reminded the group that there is a focus on installation test as
that is the biggest barrier to getting more test feedback from
Alpha/Beta testers.  Adamw agreed, but noted that ''if we have the
hardware (i can cover nvidia and possibly radeon at a push) we could
have a little xorg table in there.''  

Wwoods noted that ensuring basic functionality of intel/nouveau/radeon
is supposed to be part of the [[QA:Rawhide_Acceptance_Test_Plan]]  (see
https://fedoraproject.org/wiki/QA:X_basic_display_test_case).  Some
discussion followed as to where would be best to record results for
basic display testing, the consensus was in the short-term having a test
matrix to get some feedback on intel/nouveau/radeon feedback would be
good.

== AutoQA update from wwoods ==

=== Recent changes ===
Wwoods updated the group on recent AutoQA activities, including:
* updated conflicts test - it was doing Obsoletes and Conflicts wrong,
and taking 5 hours to run
* Improved email headers for autoqa-results ... includes test, arch and
a summary of errors (e.g. ''repoclosure: 34 packages with unresolved
deps in rawhide-x86_64'')
* The rats_* tests were broken over the weekend, but were fixed
* wwoods> anyway, I rewrote the option-parsing code in the autoqa
harness to be more sane (i.e. not hardcoding hook-specific code in the
main harness)

Jlaska planned to add a SDV system that supports KVM into Wwoods
autotest server.

F13 updated the group that a new autotest was released upstream (0.11)
that includes support for the latest Django in Fedora.  Wwoods asked
jlaska for an extra virt guest to test the updated autotest package.
F13 will provide an updated autotest server.  Woods, dpravec and jlaska
would provide test using the newer build.

Adamw asked,  ''aren't we now at the point where we could actually
create the much-vaunted israwhidebroken.com?''  Wwoods suggested he was
close, but would need a way to push/pull data from autotest to send to
another system.  After, having an (authenticated) way for qa team people
to fill in the boxes for unautomated bits (e.g. Xorg).

=== What's next ... ===
Wwoods gave an update on his plan for the upcoming week:
* look into that (pulling/pushing data from autotest for e.g.
israwhidebroken.com)
* writing docs so petr / dpravec / et. al. can get their hands dirty
designing new tests/hooks, like Fedora TPS and such

== Test Day updates ==

=== fedora-test-announce request? ===

David recommended fedora-test-announce mailing list for all QA related
events.
David suggested .. ''i heard people telling me that they would love to
participate some test days, but they do not read many mail lists.''
Some people only read fedora-announce, some miss the the mails on the
fedora-test-list traffic.

Adamw sends reminders to fedora planet (blog), fedora forums.  All
agreed that fedora-announce was not the right forum for test days.

The general consensus was that a new mailing list wouldn't hurt.  F13
suggested having the new list bcc all mails to the existing
fedora-test-list for folks who don't sign up to the list.

Dpravec would file a ticket in
https://fedorahosted.org/fedora-infrastructure/report and follow-up.

=== 2009-08-18 - Printing ===

Jlaska hasn't heard from mclasen on how the printing test day went and
asked the team if anyone had feedback on the success/fail of the
Printing test day?  Adamw but i was around for some of the day, seemed
to be going well.  Mostly participation was from the fnf guys and the
printing guys - but a few other testers did drop by.

=== 2009-08-20 - ABRT ===

ABRT test day was a success.  Thanks to Dpravec and Kparal.  See
[https://www.redhat.com/archives/fedora-test-list/2009-August/msg00609.html test summary From David Pravec].

Dpravec noted adamw's script for gathering feedback was helpful for
generating the ABRT test report.

=== 2009-08-27 - Dracut ===

Jlaska spoke w/ Harald quite a bit last week and created a few test
cases on the wiki (see [[Test_Day:2009-08-27]].  Additional tests needed
for remaining ''root='' boot scenarios.

The current state:
* the 3 core test cases are already defined
* These tests need to be performed in different root= environments
(root=UUID, root=LABEL, root=luks-UUID, root=/dev/md0, etc...) 

Planning to wrap up wiki today.  Any suggestions for improvement, feel
free to ping jlaska.

== Open discussion ==

=== nightly live builds (adamw) ===

Adamw reminded the group about the great work from nirik on providing
daily rawhide live images.  Adamw also updated the existing
[[Releases/Rawhide#Nightly_live_builds]] wiki to direct users to live
images.  Team consensus ... '''nirik rocks!'''

Adamw recommended folks advertise this change as much as possible
(blogs, mailing lists, irc discussion, tweets, etc...)

F13 expressed concerns on the value of advertising daily rawhide live
image
downloads.  The main concern was that by providing daily live images, we
might
be encouraging frequent/daily downloads of rawhide live images ... even
when
not required.  The end result, the increased bandwith load would limit
the ability of testers to provide test results against live images and
RC
milestones also hosted on the same server.  A long debate followed
debating
the value of providing live images and how much real testing they
receive.

Jlaska took an action item to raise this topic again with the
fedora-infrastructure team.  If there are problems or we need to limit
access
to the bits, or throttle downloads ... the infrastructure team would be
best
to advise.

=== bugzilla and test days (dpravec) ===

Dpravec suggested getting rid of the wiki tables for test results and
using
bugzilla directly to track test results against test cases.  Perhaps
using
bugzilla keywords to track ''Test Day'' and ''Test Case'' information.

Adamw noted that not all test days were a list of bugs, some events
track
results against specific test cases (e.g.
[[Test_Day:2009-03-26_Nouveau#Results]]).

Some discussion followed about the best mechanism.  Jokes followed about
a
pony (see http://i-want-a-pony.com).

No conclusions were reached.  There was a general sense that a test case
mgmt
system would be helpful.  Jlaska suggested that any small improvements
to the
current wiki tracking mechanisms should definitely be considered.  A
formal
test case mgmt system is ideal but definitely requires a lot of
time+resources.

=== No updates for rawhide x86_64 (kparal)? ===

Kparal asked why no package updates have been sent for Rawhide/x86_64 in
the
last few days.  Dpravec noted that there were package updates available.
Kparal condluced perhaps a mirror problem existed.

=== <your topic here> ===

= Upcoming QA events =

* 2009-08-27 - [[Test_Day:2009-08-27|Dracut test day]]
* 2009-08-31 - [[QA/Meetings/20090831]] at 16:00 UTC 

= Action items =

* [dpravec] - investigate creating fedora-test-announce mailing list for
Test Days, other test events, schedule updates
* [jlaska] - talk with Liam about how we can make the current install
test plan more attainable/achievable in the given time frames
* [jlaska] - cleanup remaining test cases for dracut and send
announcement
* [jlaska] - discuss bandwith implications of hosting nightly rawhide
live images with fedora-infrastructure
* [Oxf13] - will do a build of the new autotest version for testing.
* [jlaska + wwoods] to test latest autotest build
* '''UNASSIGNED'''
** include a xorg-x11-drv matrix in F-12-Beta test matrix for nouveau,
intel and radeon
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part
Url : http://lists.fedoraproject.org/pipermail/test/attachments/20090824/e460f814/attachment.bin