[Proposal] Test Day Guidelines

[Sumantro Mukherjee]

Hey All,

I would like to make a proposal for updating and fleshing out some of test day guideline.

To start with the wording on the Top of Test Day page:
If you come to this page before or after the test day is completed, your testing is still valuable, and you can use the information on this page to test, file any bugs you find at Bugzilla, and add your results to the results section. If this page is more than a month old when you arrive here, please check the current schedule and see if a similar but more recent Test Day is planned or has already happened.

# Proposal 1: We would also like people to *not* do testing if we are testing something which are compose sensitive and post-test day
chasing and finding the same bugs in the compose and filing dups won't make much sense. So, it will be best to reduce the time frame from a month to a week.
Also, check the current schedule for something fresher and ready requiring testing love.

# Proposal 2: We would love to see more coverage on the test cases, let's take Modularity and as an example, Contributors found bugs in Rpi 3 (ARM) which weren't the case with x86_64, Profile field in the test day results
page becomes very important and crucial. Maybe, we should avoid putting "Fedora-Server-dvd-xxxxxyyyyyyz.n.0.iso" which doesn't give much info about the testing environment or how it was executed. We might
want to flesh out the importance of that field and it will be very helpful in coming future.

# Proposal 3: Test Days have a specific team which takes on the task of curating the #fedora-test-day channel and these are mostly the people who are working on the feature being tested or developed for a significant amount of time.
This is done to ensure that all the contributors/participants can pin point folks who are playing the role they are designed for.It's in the best interest of contributors that they shouldn't edit this section as they misleading if done without proper knowledge.

Thoughts?

Thanks
/sumantrom

[Petr Šabata]

On Wed, Apr 11, 2018 at 03:23:21AM -0400, Sumantro Mukherjee wrote:
> Hey All,
>
> I would like to make a proposal for updating and fleshing out some of test day guideline.
>
> To start with the wording on the Top of Test Day page:
> If you come to this page before or after the test day is completed, your testing is still valuable, and you can use the information on this page to test, file any bugs you find at Bugzilla, and add your results to the results section. If this page is more than a month old when you arrive here, please check the current schedule and see if a similar but more recent Test Day is planned or has already happened.

+1
> # Proposal 1: We would also like people to *not* do testing if we are testing something which are compose sensitive and post-test day
> chasing and finding the same bugs in the compose and filing dups won't make much sense. So, it will be best to reduce the time frame from a month to a week.
> Also, check the current schedule for something fresher and ready requiring testing love.

+1
> # Proposal 2: We would love to see more coverage on the test cases, let's take Modularity and as an example, Contributors found bugs in Rpi 3 (ARM) which weren't the case with x86_64, Profile field in the test day results
> page becomes very important and crucial. Maybe, we should avoid putting "Fedora-Server-dvd-xxxxxyyyyyyz.n.0.iso" which doesn't give much info about the testing environment or how it was executed. We might
> want to flesh out the importance of that field and it will be very helpful in coming future.

That one probably affects all 32bit arches.  Fun.

+1
> # Proposal 3: Test Days have a specific team which takes on the task of curating the #fedora-test-day channel and these are mostly the people who are working on the feature being tested or developed for a significant amount of time.
> This is done to ensure that all the contributors/participants can pin point folks who are playing the role they are designed for.It's in the best interest of contributors that they shouldn't edit this section as they misleading if done without proper knowledge.

Yes, this really matters.

+1
> Thoughts?

Thanks you!

P

[Vipul Siddharth]

On Wed, Apr 11, 2018 at 12:53 PM Sumantro Mukherjee 
wrote:
> Hey All,
>
> I would like to make a proposal for updating and fleshing out some of test
> day guideline.
>
> To start with the wording on the Top of Test Day page:
> If you come to this page before or after the test day is completed, your
> testing is still valuable, and you can use the information on this page to
> test, file any bugs you find at Bugzilla, and add your results to the
> results section. If this page is more than a month old when you arrive
> here, please check the current schedule and see if a similar but more
> recent Test Day is planned or has already happened.
>
> # Proposal 1: We would also like people to *not* do testing if we are
> testing something which are compose sensitive and post-test day
> chasing and finding the same bugs in the compose and filing dups won't
> make much sense. So, it will be best to reduce the time frame from a month
> to a week.
> Also, check the current schedule for something fresher and ready requiring
> testing love.
>
> +1
> # Proposal 2: We would love to see more coverage on the test cases, let's
> take Modularity and as an example, Contributors found bugs in Rpi 3 (ARM)
> which weren't the case with x86_64, Profile field in the test day results
> page becomes very important and crucial. Maybe, we should avoid putting
> "Fedora-Server-dvd-xxxxxyyyyyyz.n.0.iso" which doesn't give much info about
> the testing environment or how it was executed. We might
> want to flesh out the importance of that field and it will be very helpful
> in coming future.

+1
> # Proposal 3: Test Days have a specific team which takes on the task of
> curating the #fedora-test-day channel and these are mostly the people who
> are working on the feature being tested or developed for a significant
> amount of time.
> This is done to ensure that all the contributors/participants can pin
> point folks who are playing the role they are designed for.It's in the best
> interest of contributors that they shouldn't edit this section as they
> misleading if done without proper knowledge.

+1
> Thoughts?

Mutual feelings. Should be implementable :)
Thanks
> Thanks
> /sumantrom
> _______________________________________________
> test mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

[Sinny Kumari]

> Hey All,
>
> I would like to make a proposal for updating and fleshing out some of test day guideline.
>
> To start with the wording on the Top of Test Day page:
> If you come to this page before or after the test day is completed, your testing is still
> valuable, and you can use the information on this page to test, file any bugs you find at
> Bugzilla, and add your results to the results section. If this page is more than a month
> old when you arrive here, please check the current schedule and see if a similar but more
> recent Test Day is planned or has already happened.

+1, makes sense.
> # Proposal 1: We would also like people to *not* do testing if we are testing something
> which are compose sensitive and post-test day
> chasing and finding the same bugs in the compose and filing dups won't make much
> sense. So, it will be best to reduce the time frame from a month to a week.
> Also, check the current schedule for something fresher and ready requiring testing love.

+1
> # Proposal 2: We would love to see more coverage on the test cases, let's take
> Modularity and as an example, Contributors found bugs in Rpi 3 (ARM) which weren't the
> case with x86_64, Profile field in the test day results
> page becomes very important and crucial. Maybe, we should avoid putting
> "Fedora-Server-dvd-xxxxxyyyyyyz.n.0.iso" which doesn't give much info about
> the testing environment or how it was executed. We might
> want to flesh out the importance of that field and it will be very helpful in coming
> future.

ISOs and other artifacts do contain arch value in their name. So, at least
on which architecture testing was done should be easy to identify from name.

But yeah, adding environment details like specific hardware 
used for testing will be good addition.
> # Proposal 3: Test Days have a specific team which takes on the task of curating the
> #fedora-test-day channel and these are mostly the people who are working on the feature
> being tested or developed for a significant amount of time.
> This is done to ensure that all the contributors/participants can pin point folks who are
> playing the role they are designed for.It's in the best interest of contributors that
> they shouldn't edit this section as they misleading if done without proper knowledge.

+1
> Thoughts?
>
> Thanks
> /sumantrom

[Adam Williamson]

On Wed, 2018-04-11 at 03:23 -0400, Sumantro Mukherjee wrote:
> Hey All,
>
> I would like to make a proposal for updating and fleshing out some of test day guideline.
>
> To start with the wording on the Top of Test Day page:
> If you come to this page before or after the test day is completed,
> your testing is still valuable, and you can use the information on
> this page to test, file any bugs you find at Bugzilla, and add your
> results to the results section. If this page is more than a month old
> when you arrive here, please check the current schedule and see if a
> similar but more recent Test Day is planned or has already happened.
> # Proposal 1: We would also like people to *not* do testing if we are
> testing something which are compose sensitive and post-test day
> chasing and finding the same bugs in the compose and filing dups
> won't make much sense. So, it will be best to reduce the time frame
> from a month to a week.
> Also, check the current schedule for something fresher and ready
> requiring testing love.

I picked 'a month' more or less arbitrarily, 'a week' is certainly an
OK choice too. Note that the template really is just a *template*, and
can (and should!) be adapted to each event as appropriate. So if *some
specific event* is very time sensitive, this text could be changed as
appropriate - to say 'a day' or whatever.
> # Proposal 2: We would love to see more coverage on the test cases,
> let's take Modularity and as an example, Contributors found bugs in
> Rpi 3 (ARM) which weren't the case with x86_64, Profile field in the
> test day results
> page becomes very important and crucial. Maybe, we should avoid
> putting "Fedora-Server-dvd-xxxxxyyyyyyz.n.0.iso" which doesn't give
> much info about the testing environment or how it was executed. We
> might
> want to flesh out the importance of that field and it will be very
> helpful in coming future.

I'm actually not quite sure what you're referring to here, or where the
"Fedora-Server-dvd-xxxxxyyyyyyz.n.0.iso" example comes from. Could you
provide a bit more context?

In general I'd certainly say that, again, the way in which result
reporting is set up should be carefully considered for each individual
test day; the template and SOP are only guides. You can set up a result
table in many different ways to capture results from different
combinations of environment factors...
> # Proposal 3: Test Days have a specific team which takes on the task
> of curating the #fedora-test-day channel and these are mostly the
> people who are working on the feature being tested or developed for a
> significant amount of time.
> This is done to ensure that all the contributors/participants can pin
> point folks who are playing the role they are designed for.It's in
> the best interest of contributors that they shouldn't edit this
> section as they misleading if done without proper knowledge.

Are you saying there should be a specifically-identified 'team' for
each individual test day, or an ongoing permanent 'team' for test days
as a whole? Either could potentially work, but I think the proposal
could do with a few more details...

[Charles-Antoine Couret]

Hi,
I have another proposal (or maybe I'm wrong about this).

Make an announcement (or a proposal) some days (1 week?) before the test 
day date. For example this week, I saw the announcement for the Cloud 
test day the day of the event. And the wiki page of this test day hasn't 
been created 1 week before.

I don't know if we can see the announcement somewhere else, personally I 
follow community blog + fedora mag + test announce for that.

Without this information it is more difficult to translate / forward the 
announcement (I make French articles every-time on my blog) on schedule. 
I understand that the tests could be executed some days later the 
official date but it is more relevant if we can have more results the 
dedicated day. And some test days required some preparations (download a 
big image or updates, create a virtual machine, or other things) and it 
is better if we can prepare that before this day.

I agree with the rest of guidlines.

Regards,
Charles-Antoine Couret

[Sumantro Mukherjee]

----- Original Message -----
> From: "Charles-Antoine Couret" 
> To: [e-mail]
> Sent: Friday, April 13, 2018 9:17:43 AM
> Subject: Re: [Proposal] Test Day Guidelines
>
> Hi,
> I have another proposal (or maybe I'm wrong about this).
>
> Make an announcement (or a proposal) some days (1 week?) before the test
> day date. For example this week, I saw the announcement for the Cloud
> test day the day of the event. And the wiki page of this test day hasn't
> been created 1 week before.
>
> I don't know if we can see the announcement somewhere else, personally I
> follow community blog + fedora mag + test announce for that.
>
> Without this information it is more difficult to translate / forward the
> announcement (I make French articles every-time on my blog) on schedule.
> I understand that the tests could be executed some days later the
> official date but it is more relevant if we can have more results the
> dedicated day. And some test days required some preparations (download a
> big image or updates, create a virtual machine, or other things) and it
> is better if we can prepare that before this day.
>
> I agree with the rest of guidlines.
>
> Regards,
> Charles-Antoine Couret
> _______________________________________________
> test mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

Hey Charles,

I know about the Cloud Test Day and we are really sorry, we couldn't come up with something days before.
The mag although had the annoucement [https://fedoramagazine.org/contribute-add-modularity-kernel-test-days/]
It's really good to hear that you have been helping us with a lot of things with posting the blog for contributors in French. Thanks for that! :)

The last bit, I totally agree that, most of us needs VMs and iso(s) are of big sizes, I would like to make sure we figure out a solution to 
address the situation.

Thanks
//sumantrom

[Sumantro Mukherjee]

----- Original Message -----
> From: "Adam Williamson" 
> To: "For testing and quality assurance of Fedora releases" 
> Sent: Thursday, April 12, 2018 3:57:33 AM
> Subject: Re: [Proposal] Test Day Guidelines
>
> On Wed, 2018-04-11 at 03:23 -0400, Sumantro Mukherjee wrote:
> > Hey All,
> >
> > I would like to make a proposal for updating and fleshing out some of test
> > day guideline.
> >
> > To start with the wording on the Top of Test Day page:
> > If you come to this page before or after the test day is completed,
> > your testing is still valuable, and you can use the information on
> > this page to test, file any bugs you find at Bugzilla, and add your
> > results to the results section. If this page is more than a month old
> > when you arrive here, please check the current schedule and see if a
> > similar but more recent Test Day is planned or has already happened.
> > # Proposal 1: We would also like people to *not* do testing if we are
> > testing something which are compose sensitive and post-test day
> > chasing and finding the same bugs in the compose and filing dups
> > won't make much sense. So, it will be best to reduce the time frame
> > from a month to a week.
> > Also, check the current schedule for something fresher and ready
> > requiring testing love.
>
> I picked 'a month' more or less arbitrarily, 'a week' is certainly an
> OK choice too. Note that the template really is just a *template*, and
> can (and should!) be adapted to each event as appropriate. So if *some
> specific event* is very time sensitive, this text could be changed as
> appropriate - to say 'a day' or whatever.

True. I will keep this in mind from next time :)
> > # Proposal 2: We would love to see more coverage on the test cases,
> > let's take Modularity and as an example, Contributors found bugs in
> > Rpi 3 (ARM) which weren't the case with x86_64, Profile field in the
> > test day results
> > page becomes very important and crucial. Maybe, we should avoid
> > putting "Fedora-Server-dvd-xxxxxyyyyyyz.n.0.iso" which doesn't give
> > much info about the testing environment or how it was executed. We
> > might
> > want to flesh out the importance of that field and it will be very
> > helpful in coming future.
>
> I'm actually not quite sure what you're referring to here, or where the
> "Fedora-Server-dvd-xxxxxyyyyyyz.n.0.iso" example comes from. Could you
> provide a bit more context?

The context where it stems from was, if you have a look at http://testdays.fedorainfracloud.org/events/38
we have mostly all testers, mentioning Fedora 28 Server on (KVM,ARM,x86_64,VMware,ppc64le) which gives
us a bit more idea if there is a failure. Just writing Fedora server dvd gives us absolutely no idea as to how the
test were executed. In one certain case during Modularity test day, a certain issue just came up which was specific to
ARM ... the profile also lets us know about the coverage across archs and various virtualizations. I felt, we should,
flesh out a bit more about the importance of the same.
> In general I'd certainly say that, again, the way in which result
> reporting is set up should be carefully considered for each individual
> test day; the template and SOP are only guides. You can set up a result
> table in many different ways to capture results from different
> combinations of environment factors...

yes, thats something I will look at :)
> > # Proposal 3: Test Days have a specific team which takes on the task
> > of curating the #fedora-test-day channel and these are mostly the
> > people who are working on the feature being tested or developed for a
> > significant amount of time.
> > This is done to ensure that all the contributors/participants can pin
> > point folks who are playing the role they are designed for.It's in
> > the best interest of contributors that they shouldn't edit this
> > section as they misleading if done without proper knowledge.
>
> Are you saying there should be a specifically-identified 'team' for
> each individual test day, or an ongoing permanent 'team' for test days
> as a whole? Either could potentially work, but I think the proposal
> could do with a few more details...

No, I am good with either, I am strictly against people editing the organizing list.
The team is defined and kept there to make sure the contributors know where to come and
whom to grab hold of , when something is breaking. Randomly editing the section renders to pupose moot.
Context:
[http://fedoraproject.org/wiki/Test_Day:2018-02-22_Kernel_4.15_Test_Day] 
[http://fedoraproject.org/wiki/Test_Day:2018-04-10_Add-On_Modularity_Test_Day]
> --
> Adam Williamson
> Fedora QA Community Monkey
> IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
> http://www.happyassassin.net
> _______________________________________________
> test mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

[Adam Williamson]

On Fri, 2018-04-13 at 02:12 -0400, Sumantro Mukherjee wrote:
> > I picked 'a month' more or less arbitrarily, 'a week' is certainly an
> > OK choice too. Note that the template really is just a *template*, and
> > can (and should!) be adapted to each event as appropriate. So if *some
> > specific event* is very time sensitive, this text could be changed as
> > appropriate - to say 'a day' or whatever.
>
> True. I will keep this in mind from next time :)

To be clear, if you wanna change the template to 'a week', that sounds
fine to me.
> > > # Proposal 2: We would love to see more coverage on the test cases,
> > > let's take Modularity and as an example, Contributors found bugs in
> > > Rpi 3 (ARM) which weren't the case with x86_64, Profile field in the
> > > test day results
> > > page becomes very important and crucial. Maybe, we should avoid
> > > putting "Fedora-Server-dvd-xxxxxyyyyyyz.n.0.iso" which doesn't give
> > > much info about the testing environment or how it was executed. We
> > > might
> > > want to flesh out the importance of that field and it will be very
> > > helpful in coming future.
> >
> > I'm actually not quite sure what you're referring to here, or where the
> > "Fedora-Server-dvd-xxxxxyyyyyyz.n.0.iso" example comes from. Could you
> > provide a bit more context?
>
> The context where it stems from was, if you have a look at http://tes
> tdays.fedorainfracloud.org/events/38
> we have mostly all testers, mentioning Fedora 28 Server on
> (KVM,ARM,x86_64,VMware,ppc64le) which gives
> us a bit more idea if there is a failure. Just writing Fedora server
> dvd gives us absolutely no idea as to how the
> test were executed. In one certain case during Modularity test day, a
> certain issue just came up which was specific to
> ARM ... the profile also lets us know about the coverage across archs
> and various virtualizations. I felt, we should,
> flesh out a bit more about the importance of the same.

OK, I see. The tricky part is it's a free text field, so all you can
really do is try to provide very specific instructions for filling it
out in each Test Day page, really clearly explaining what people should
include in it - at least, that's what I'd do.

I suppose we could also enhance the app and the metadata format so the
metadata could specify some 'hint text' for the profile field, or
something like that...
> > Are you saying there should be a specifically-identified 'team' for
> > each individual test day, or an ongoing permanent 'team' for test days
> > as a whole? Either could potentially work, but I think the proposal
> > could do with a few more details...
>
> No, I am good with either, I am strictly against people editing the organizing list.
> The team is defined and kept there to make sure the contributors know where to come and
> whom to grab hold of , when something is breaking. Randomly editing the section renders to pupose moot.
> Context:
> [http://fedoraproject.org/wiki/Test_Day:2018-02-22_Kernel_4.15_Test_Day] 
> [http://fedoraproject.org/wiki/Test_Day:2018-04-10_Add-On_Modularity_Test_Day]

OK, I see. So perhaps we should edit the template to include an
{{admon}} in that section asking people not to edit it unless they're
really part of the organizing team? Including some text that
specifically says "If you're just joining as a tester, you shouldn't
add yourself here"?

Thanks!