Minutes/Summary from today's proventester meetup

Kevin Fenzi kevin at scrye.com
Wed Sep 21 19:59:42 UTC 2011 

===========================================
#fedora-meeting: proventesters (2011-09-21)
===========================================

Meeting started by nirik at 19:00:29 UTC. The full logs are available at
http://meetbot.fedoraproject.org/fedora-meeting/2011-09-21/proventesters.2011-09-21-19.00.log.html

Meeting summary
---------------
* init process/agenda  (nirik, 19:00:29)

* What's required of proventesters?  (nirik, 19:03:49)
  * LINK: http://fedoraproject.org/wiki/Proven_tester   (nirik,
    19:03:50)

* Resources  (nirik, 19:06:37)
  * LINK:
    https://admin.fedoraproject.org/updates/rss/rss2.0?comments=True
    (nirik, 19:09:24)

* F14  (nirik, 19:13:01)
  * LINK:
    https://admin.fedoraproject.org/updates/rss/rss2.0?package=ipython&release=F16
    (nirik, 19:24:38)
  * LINK:
    http://lists.fedoraproject.org/pipermail/test/2011-September/thread.html
    (Southern_Gentlem, 19:29:57)
  * AGREED: will meet next week at 18UTC.  (nirik, 19:58:16)

Meeting ended at 19:58:37 UTC.




Action Items
------------





Action Items, by person
-----------------------
* **UNASSIGNED**
  * (none)




People Present (lines said)
---------------------------
* nirik (106)
* Southern_Gentlem (27)
* mcloaked (26)
* Viking-Ice (22)
* tflink (21)
* Cerlyn (14)
* OldFart (14)
* brunowolff (5)
* athmane (5)
* zodbot (3)
* jsmith (3)
--
19:00:29 <nirik> #startmeeting proventesters (2011-09-21)
19:00:29 <zodbot> Meeting started Wed Sep 21 19:00:29 2011 UTC.  The
chair is nirik. Information about MeetBot at
http://wiki.debian.org/MeetBot. 19:00:29 <zodbot> Useful Commands:
#action #agreed #halp #info #idea #link #topic. 19:00:29 <nirik>
#meetingname proventesters 19:00:29 <nirik> #topic init process/agenda
19:00:29 <zodbot> The meeting name has been set to 'proventesters'
19:00:37 <nirik> any proventesters around today? 19:00:45 <OldFart> yes.
19:00:56 <tflink> yep
19:01:05 <nirik> cool.
19:01:16 * nirik will wait a few minutes and see if we get any more
folks. 19:01:43 <mcloaked> I am here -
19:01:53 <nirik> welcome
19:01:56 <Cerlyn> I'm here
19:02:34 * Southern_Gentlem 
19:02:44 <nirik> excellent. A few folks. ;) Anyone here _not_ currently
a proventester? 19:02:55 * jsmith is not
19:03:10 <nirik> jsmith: cool. It's easy to become one...
19:03:20 <jsmith> nirik: I imagine it is :-)
19:03:30 <jsmith> Don't mind me though... I'm mostly just lurking
19:03:43 <nirik> ok, let me do some intro materal and then we can move
on to real business... 19:03:49 <nirik> #topic What's required of
proventesters? 19:03:50 <nirik>
http://fedoraproject.org/wiki/Proven_tester 19:04:01 <nirik> So, the
above wiki page is the main landing page about proventesters. 19:04:48
<nirik> so, you can find info there about joining and what
proventesters need to do. 19:05:09 <nirik> basically to join you have
to agree that you have read and understand what you are supposed to
do. ;) 19:05:37 <nirik> Any questions on process or the like? should be
able to answer them by reading that page... (and it's links) 19:06:33
<nirik> ok, I'd like to talk process/tools for a minute. 19:06:37
<nirik> #topic Resources 19:06:50 <nirik> If folks aren't currently
using them or know about them: 19:06:51 <mcloaked> Quick question: do
you know how many proventesters there are now? 19:07:03 <nirik>
mcloaked: yeah, I looked this morning. There's 80 people in the group.
19:07:11 <Cerlyn> A better question might be how many are active
19:07:20 <nirik> yeah, thats much harder to determine. ;( 19:07:37
<tflink> we could grab recent bodhi comments and compare to the list of
members 19:07:45 <nirik> * fedora-easy-karma makes it easy to submit
karma on updates if you have a bunch installed. Very useful. 19:07:54
<mcloaked> True - though does bodhi keep a track count of
comments/karma additions? 19:08:01 <nirik> * bodhi has rss feeds if you
want to know about just one packges updates. 19:08:13 <nirik> mcloaked:
not sure. 19:08:25 <tflink> mcloaked: not sure what it keeps track of
by user, I'm usually grabbing data for updates 19:08:34 <nirik> * bodhi
command line can be great for pulling down updates that aren't in
testing yet. 19:08:37 * brunowolff is here, but was doing work stuff
when the meeting started. 19:08:59 <nirik> There's also an overall
comments rss feed 19:09:21 <Southern_Gentlem> ?brb 19:09:24 <nirik>
https://admin.fedoraproject.org/updates/rss/rss2.0?comments=True
19:09:34 <mcloaked> Yes often getting rpms that have yet to reach the
repo is useful. 19:10:04 <tflink> either way, it is doable - even if
that does involve some ugly code that pulls a crapton of data from
bodhi :) 19:11:05 <nirik> bodhi also has a metrics page:
https://admin.fedoraproject.org/updates/metrics/?release=F16 19:11:53
<nirik> So, I thought I would focus on f14 today... but I'm happy to
discuss anything else folks feel is related and important... 19:12:57 *
nirik listens to crickets. 19:13:01 <nirik> #topic F14 19:13:31 <nirik>
so, there's currently 12 pending security updates in f14 19:13:50
<nirik> and 15 updates in critpath 19:14:01 <nirik> (there is some
overlap there) 19:14:29 <nirik> Does anyone here have f14 instances
still? ;) 19:14:29 * nirik has a vm 19:14:42 <tflink> my laptop is
still running F14 19:14:46 <OldFart> I have F14 19:14:48 * Cerlyn has
VMs and XOs 19:14:56 <Southern_Gentlem> ?can there be a mailing list
other than testing that gets all the noise  from qa 19:15:10
<brunowolff> I don't really have any f15 instances. Just an old laptop,
that I only use for special stuff and will upgrade probably the next
time I touch it. 19:15:31 <brunowolff> Everything I normally use is f16
or f17 now. 19:15:39 <nirik> Southern_Gentlem: what email/info do you
want to get thats lost in the noise currently? 19:16:01 <OldFart> The
last time I looked at F14 security updates, I didn't feel qualified to
test them. 19:16:08 <brunowolff> And I mostly have older hardware, so
haven't been too interested in setting up VMs. 19:16:52 <nirik> some of
them would seem to be pretty easy... like the kernel... 19:17:19
<nirik> but others are more esoteric. 19:17:19 <nirik> Would having
more test plans help? 19:17:42 <tflink> for some things, yes 19:17:54
<OldFart> Yeah, the kernel, but the fixes are not known to me.  I can
+1 from casual use.  Is that OK? 19:17:56 <tflink> I've run across more
than one update that I haven't the foggiest idea how to test 19:18:11
<Southern_Gentlem> nirik,  stuff for f14 and f15 that is waiting for
testing 19:18:13 <Cerlyn> A large part of my testing is not so much
based on test plans, but trying to replicate the issue that was fixed
19:18:39 <tflink> how to hit the code that was changed to verify that
it doesn't explode, rather 19:19:08 <nirik> Southern_Gentlem: so the
updates-testing report? could you filter on that? or perhaps we could
add a topic for it to make it easier to filter on? 19:19:11 <Cerlyn>
yes; it doesn't make sense in my view to test the kernel for example if
I don't have a computer with the updated module 19:19:17 <mcloaked> For
kernel I still think it is useful to get comments if there are no
regressions, but it would be helpful to have notes to remind people to
install not just the appropriate kernel rpm but also the related perf,
kernel-headers etc (I know perf has gone from the kernel builds for
f16)? 19:19:32 <nirik> OldFart: yes. You don't need to verify
everything, you need to verify that it shows no regressions for you and
isn't breaking. 19:19:43 <Southern_Gentlem> nirik,  to have the updates
testing report sent to a proven testers list 19:20:23 <nirik> that
seems like kinda a waste for an entire list, IMHO 19:21:00
<Southern_Gentlem> nirik,  i think the reason stuff is not getting
tested is no notification system for the proven testers 19:21:32
<Southern_Gentlem> we all get busy doing our things and that falls
through the cracks 19:21:35 <OldFart> Used to be mail notifications,
but too many complained about spam.... 19:21:36 <nirik> for critpath
packages, remember you simply need to test:
http://fedoraproject.org/wiki/Critical_path_action#Actions 19:22:16
<nirik> OldFart: right. 19:22:41 <mcloaked> If a proventester is
subscribed to the test list then a list of critpath and security
updates gets sent out fairly regularly? 19:22:47 <nirik> I am sure if
we started mailing proventesters on every updates push people would
complain again. 19:22:48 <Southern_Gentlem> nirik,  or a compromise is
that the package has sat there with no testing and the maintainer sends
it to a mailing list 19:22:55 <nirik> mcloaked: yes, every push
19:23:19 <OldFart> Feasible to have an opt in mailing list just for
testers? 19:23:26 <nirik> Subject: Fedora N updates-testing report
19:23:34 <nirik> OldFart: I would call the 'test' list that. 19:23:53
<OldFart> OK... 19:24:03 <Cerlyn> Goes back to my email earlier
yesterday/today on wanting to be able to subscribe to package updates
like one can for bug reports and commits.  (I was trying to phrase it
in ways that won't overload infrastructure) 19:24:19 <nirik> Cerlyn:
would the rss feeds meet that need? 19:24:38 <nirik>
https://admin.fedoraproject.org/updates/rss/rss2.0?package=ipython&release=F16
19:24:49 <Cerlyn> nirik: The full updates list per release would, if an
RSS reader exists which can filter 19:24:58 <Cerlyn> I don't want to
subscribe to every sugar* package out there 19:25:10 <mcloaked> By the
way if you add yourself to the bugzilla report for a package you are
interested in then you will get mail for new comments - and that will
usually include mail reporting when a fix package has been built.
19:25:12 <Cerlyn> that's a burden on infrastructure, and I was an
infrastructure person 19:25:44 <Cerlyn> mcloaked: But does the package
owner have to approve those memberships?  I was trying to figure out
the system 19:25:44 <nirik> right, so you want a collection of
packages, but in one feed or the like... 19:25:51 <Southern_Gentlem>
test list to me is more for QA and (yes proventesters can follow that
if they wish be it looks like alot of spam for people who want to test
updates for current releases 19:26:03 <brunowolff> This is from back a
bit, but perf isn't gone, the sub-package name changed to kernel-tools.
19:26:47 <mcloaked> Cerlyn: I think the package owner does not need to
approve if you add yourself to the bug cc: list 19:27:02 <nirik>
Cerlyn: no approval needed, just a bugzilla account. 19:27:24 <nirik>
Cerlyn: you can also add yourself in pkgdb to packages you are
interested in to be cc'ed on all bug reports against that package.
19:27:33 <nirik> (get ready for lots of email on some packages though)
19:27:43 <Cerlyn> It's a side topic but I noticed a lot of packages
didn't have a QA contact in pkgdb 19:27:55 * Viking-Ice sneaks in...
19:27:58 <nirik> Cerlyn: I don't think that field is used for
anything. ;( 19:28:19 <Cerlyn> bugzilla uses it IIRC 19:28:27
<mcloaked> brunowolff: thanks - about perf->kernel-tools 19:28:56
<nirik> Southern_Gentlem: I guess if there's enough interest for a new
list for the updates emails, we could try it... perhaps we should ask
on the test list if there is enough interest? 19:29:55
<Southern_Gentlem> example 19:29:57 <Southern_Gentlem>
http://lists.fedoraproject.org/pipermail/test/2011-September/thread.html
19:30:29 <Southern_Gentlem> you have so scroll past the first page to
see anything about f15 or f14 updates needing tested 19:30:46 <nirik>
Cerlyn: it's set to extras-qa at fedoraproject.org for all fedora bugs...
which goes to... /dev/null. ;) 19:31:13 <Cerlyn> One potentially also
could generate a report on packages which have not received feedback
after some time, and email that, as a warning that the packager can now
push at will 19:31:17 <nirik> Southern_Gentlem: yeah, but couldn't you
filter through it for the ones you want? or you mean many people aren't
subscribed and are looking at the web archive? 19:32:30
<Southern_Gentlem> nirik,  so why dont separate the list so just the
released releases updates that need testing and not devel 19:32:40
<Viking-Ice> if you are part of the QA community you should be
subscribed and all QA task requested should be posted to the test list
not -devel 19:33:15 <Southern_Gentlem> nirik,  i dropped the test list
because of 60 messages being delivered to get the ones that deal with
me 19:33:21 <nirik> Southern_Gentlem: well, I fear splitting that out
would just result in another list that not many people would subscribe
to, since most folks interested are already subscribed to test.
19:33:32 <nirik> but I could be wrong. 19:33:53 <tflink> either way,
sounds like time to ask for input from test@ ? 19:33:59 <mcloaked>
nirik: my feeling is that you are right... 19:34:01 <Southern_Gentlem>
nirik,  stuff isnt getting tested tells me that most feel the test list
is to noisey 19:34:05 <Viking-Ice> not really we dont want split which
which results in divided sub community 19:34:17 <Southern_Gentlem>
Viking-Ice,  you already have one 19:34:21 <nirik> yeah, lets ask for
input on the test list... and see if there's enough folks who want
another list... 19:34:24 <Viking-Ice> bugzapper 19:34:44 <Viking-Ice>
if that's the route you are heading have at it 19:35:10 <nirik> so,
moving back to f14. ;) What can we do to better handle these? would
folks like to go thru one by one in the meeting? or should we look at
hard to test ones? easy to test ones? 19:35:16 <Viking-Ice> we struggle
hard to not divide the zapper stuff was due to autoqa only discussion
on QA 19:35:17 <mcloaked> There could be other reasons - like people
are busy, or for install testing when there are major issues giving bad
or unbootable installs it can slow things down. 19:35:44 <nirik> I
think it's mostly that people don't know they can test/provide feedback
19:36:00 <nirik> visibility is just low. 19:36:10 <tflink> you mean
among proventesters or among fedora users? 19:36:22 <mcloaked> Is there
any chance that f14 is being sidelined by people trying to test later
(eg f16) packages and getting short of time? 19:36:28 <nirik> tflink:
fedora users. 19:36:50 <tflink> I think part of it is worry that
they'll tank their system 19:37:03 <nirik> I know as a package
maintainer when I fix a bug, ofter the reporter will say "great, works
here" in the bug, but won't add karma unless I specifically ask them
to, and often not even then. 19:37:14 * athmane is late 19:37:22
<mcloaked> tflink: do testers mainly use production systems in their
testing or spare machines? 19:37:31 <tflink> have no idea 19:37:38
<nirik> mcloaked: yes, f14 suffers from many things: 19:37:48 <tflink>
I know that my F14 system is a production machine that I use every day
19:37:52 <nirik> * fedora is a fast moving distro, so most of the
energy is on f16+ 19:38:10 <nirik> * the people running f14 just want
it to keep working, so don't want to test dangerous things. 19:38:18
<OldFart> I have F14, F15, F16 and rawhide on one machine...no VMs and
try to test when I see the updates testing report.  But, 19:38:20
<nirik> * f14 people are less likely to be contributors I suspect...
19:38:35 <mcloaked> In my own case for potentially "tanking" tests I
use a machine that won't matter if it breaks - but for production
machines I will test packages selectively. 19:38:54 <OldFart> most
often I am not familiar enough with the package or, if familiar, with
the bug or security fix. 19:39:15 <athmane> imho f14 (fedora old stable
in general) needs a less strict update policy 19:39:27 <nirik> OldFart:
for that perhaps the test list or #fedora-qa could help... ie, someone
who knows the package could help others test it 19:39:49 <mcloaked>
OldFart: you can still install a package in critpath and check if there
are no regressions - that is valid process. 19:39:49 <nirik> athmane:
EBOYONDTHESCOPEOFTHISMEETING. ;) 19:40:15 <OldFart> Test cases would
help....but they add a burden to the developer/maintainer. 19:41:10
<nirik> sure, but they are great to have... 19:41:22 <nirik> but most
of them time currently, we don't have them 19:42:05 <nirik> as a side
note, some of the f14 critpath updates should drop off the report
soon... some of the obscure X drivers are dropping out of critpath
19:42:32 <nirik> qxl and openchrome 19:43:21 <mcloaked> On testcases -
sometimes people could meet on here and discuss how to test a
particular package perhaps? Might help where there is no testcase
available? 19:43:41 <nirik> I would think these would be not too bad to
verify critpath functionality on: kernel, gnupg2, curl, udev, 19:43:51
<nirik> mcloaked: absolutely. 19:44:09 <OldFart> +1 19:44:36 <nirik>
that was the last item I had for today: should we bother meeting again?
are meetups helpfull or useful in any way? 19:45:02 <athmane> there's a
ticket for criticalpath test cases creation =>
https://fedorahosted.org/fedora-qa/ticket/154 19:45:13 <OldFart>
Helpful....more should show up over time 19:45:17 <Viking-Ice> it
should be mandantory for maintainers to provide how to debug their
component and provide test cases 19:45:20 <mcloaked> Also I guess there
is #fedora-qa for discussions? 19:46:11 <nirik> mcloaked: yep. 19:46:38
<nirik> athmane: good link... 19:46:47 <mcloaked> I guess that the log
of meetings such as this can be seen when they are sent out to the test
list - so they are useful even if only small numbers 19:46:54 <nirik>
Viking-Ice: not really enforceable... but in an ideal world... 19:47:05
<athmane> we have ~15 test case of 432 critical path pkg 19:47:12
<tflink> forcing test cases != good test cases 19:47:37
<Southern_Gentlem> and no testing ==no testing 19:47:38 <tflink> does
not necessarily lead to, anyways 19:47:39 <nirik> athmane: long way to
go for sure. 19:48:10 <Viking-Ice> nirik, why not 19:48:14 <mcloaked>
Also worth stressing that if testing leads to problems seen then it is
important to file bz.... not just a comment in bodhi 19:48:16 * nirik
just finished rebooting his f14 vm... 19:48:22 <nirik> mcloaked: +1
19:48:34 <Viking-Ice> nirik if you want your component in fedora you
need to provide us with that info 19:48:43 <nirik> Viking-Ice: what do
you do if they don't? 19:48:51 <nirik> and what if they are horrible at
testing? 19:48:56 <Viking-Ice> dont introduce the component 19:49:12
<tflink> what do you do about the bad test cases that just introduce
noise? 19:49:30 <Southern_Gentlem> all of this is not addressing the
issues 19:49:32 <Viking-Ice> if the packager and or maintainer does not
know how to debug and test his own component <--- 19:49:51 <nirik>
Southern_Gentlem: suggestions? 19:50:01 <tflink> knowing how to debug
and being willing to write out test cases are not the same thing
19:50:22 <Viking-Ice> mcloaked, reporters have come and gone mostly
because they dont get responded to in bugzilla 19:50:42 <mcloaked> For
some kinds of test fails are quite tough to get useful debug info - eg
boot crash - how to get backtrace, install crash - how to get bactrace
- similar for serious kernel issues when they happen even if they are
rare. 19:50:56 <Southern_Gentlem> nirik,  other than breaking out
another list that has what needs testing nope 19:51:13 <nirik>
Southern_Gentlem: ok, will float that idea to the test list... or did
you want to? 19:51:17 <Southern_Gentlem> i see notification to proven
testers as the bigest problem 19:51:40 <nirik> we didn't really see
much more testing back when we spamed all proventesters I don't
think... 19:51:46 <nirik> but there was likely fewer back then.
19:52:04 <Southern_Gentlem> nirik,  go ahead but i see most of the test
list are testing devel and could care less about the currcent releases
or we wouldnt be in this shape 19:52:28 <Viking-Ice> yeah somepeople
have thrown updates-testers of that list 19:52:34 <mcloaked> Within
bodhi you can get a list of security packages such as at
https://admin.fedoraproject.org/updates/F14/security?_csrf_token=0834a0859819089ae1a83c6e8e96ed3b63a57934
19:52:38 <Viking-Ice> as in GA testers 19:52:45 <nirik> sure, for
mentioned reasons there's less interest in the stable releases. But
really there's not that many critpath/security ones. 19:53:01
<mcloaked> Might be nice if there was also a link to current critpath
package updates per Fedora version too? 19:53:17 <nirik> I really think
it wouldn't take too much for a group to process thru them... I think
it's reachable. 19:53:51 <Southern_Gentlem> nirik,  i think a most a
week email of list of packages waiting to be tested in the current
releases would be easy 19:53:57 <Viking-Ice> still reporters/testers
are faced with nonresponsive maintainers 19:54:04 <Viking-Ice> and EOL
19:54:07 <tflink> mcloaked: you mean other than
https://admin.fedoraproject.org/updates/critpath?unapproved=True&release=F16
19:54:34 <Viking-Ice> seriously when are we going to target the root of
the problem instead of working around it 19:54:41 <nirik>
Southern_Gentlem: well, emailing the updates-testing reports to another
list would be easy... 19:55:10 <Southern_Gentlem> nirik,  only when
stuff needs to be tested no status after that 19:55:11 <mcloaked>
tflink: Excellent - I was unaware of that construction!  I will
certainly use it in the future. 19:55:12 <nirik> Viking-Ice: because
there's no solution to that unless you intend to hire and task people
with full time development/qa? 19:55:27 <Viking-Ice> drop the ownership
model 19:55:34 <Viking-Ice> get the devs working together etc 19:55:49
<nirik> Southern_Gentlem: that would be a lot harder, as it would need
bodhi code changes. ;( 19:55:50 <Viking-Ice> necessary and needed
change 19:56:08 <nirik> Viking-Ice: beyond the scope of this meeting.
19:56:08 <mcloaked> tflink: It would be even nicer if that list of
unapproved critpath packages was a link on the left side of the bodhi
interface! 19:56:21 <nirik> Anyhow, 4 min left until next meeting...
19:56:29 <nirik> should we meet again next week at this time? another
time? not at all? 19:56:30 <tflink> I'm not convinced that would work
all that well, my experience is that when there's no ownership, nothing
gets done 19:56:38 <Viking-Ice> nirik, yeah bury the root cause once
more, nonresponsive maintainers are the root of alot of problems
19:56:48 <tflink> mcloaked: yeah, you have to click through a bunch of
stuff to find that link 19:56:52 <Southern_Gentlem> nirik,  and only
updates-testing for current releases not devel 19:57:02 <nirik>
Southern_Gentlem: devel has no updates-testing. ;) 19:57:19
<Southern_Gentlem> nirik,  what does f16 go to at the moment 19:57:19
<OldFart> I'm ok with a meeting next week....prefer an hour earlier.
19:57:25 * nirik waits for votes. will close the meeting in a minute or
two 19:57:30 <Southern_Gentlem> +1 19:57:34 <tflink> I'm also ok with
next week 19:57:41 <mcloaked> I believe I can make next week same time
19:57:46 <nirik> next week at 18? 19:57:50 <athmane> +1 19:57:55
<Viking-Ice> +1 19:57:57 <Cerlyn> + 19:58:04 <Southern_Gentlem> +1
19:58:11 <OldFart> +1 can't stay for the hour 19:58:12 <mcloaked> +!
19:58:16 <nirik> #agreed will meet next week at 18UTC. 19:58:31 <nirik>
Thanks for coming everyone... continue discussion over in #fedora-qa
and on list... 19:58:37 <nirik> #endmeeting
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
Url : http://lists.fedoraproject.org/pipermail/test/attachments/20110921/331f9eb3/attachment.bin

More information about the test mailing list