RATS tests still running against F17?

[Adam Williamson]

So I took a quick look at the resultsdb and noticed that the RATS tests
- rats_sanity and rats_install - still seem to be being run against F17,
which isn't much use. (And they invariably fail). Is there a plan to
switch them to Rawhide? Do we need some kind of release-transition SOP
for AutoQA tests, to ensure they're bumped from Rawhide to Branched and
then back to Rawhide after the stable release, in a timely manner?
Thanks!
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net

[Josef Skladanka]

IMHO, the repoinfo.conf needs to be changed, but I'm not sure - rats* tests go a bit
around me. I'll have a look at it today.

J.

----- Original Message -----
>  From: "Adam Williamson" <awilliam(a)redhat.com&gt;
>  To: autoqa-devel(a)lists.fedorahosted.org
>  Sent: Thursday, July 12, 2012 9:23:24 PM
>  Subject: RATS tests still running against F17?
>  
>  So I took a quick look at the resultsdb and noticed that the RATS
>  tests
>  - rats_sanity and rats_install - still seem to be being run against
>  F17,
>  which isn't much use. (And they invariably fail). Is there a plan to
>  switch them to Rawhide? Do we need some kind of release-transition
>  SOP
>  for AutoQA tests, to ensure they're bumped from Rawhide to Branched
>  and
>  then back to Rawhide after the stable release, in a timely manner?
>  Thanks!
>  --
>  Adam Williamson
>  Fedora QA Community Monkey
>  IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
>  http://www.happyassassin.net
>  
>  _______________________________________________
>  autoqa-devel mailing list
>  autoqa-devel(a)lists.fedorahosted.org
>  https://fedorahosted.org/mailman/listinfo/autoqa-devel
>

[Josef Skladanka]

> From what I've discovered: 
1) rats_sanity is currently
running against rawhide IMHO
   http://autoqa.fedoraproject.org/resultsdb/frontend/testrun?id_=359830
  
http://autoqa.fedoraproject.org/resultsdb/frontend/search?testcase=rats_s...
   
2) rats_install is not being scheduled at all. Last rats_install run is from 12/05/24
<http://autoqa.fedoraproject.org/resultsdb/frontend/search?testcase=rats_i...
IIRC Kamil ran the tests manualy for F17.

Does this answer your question, and is this the state you're satisfied with ATM?

J.

----- Original Message -----
>  From: "Josef Skladanka" <jskladan(a)redhat.com&gt;
>  To: "AutoQA development" <autoqa-devel(a)lists.fedorahosted.org&gt;
>  Sent: Tuesday, July 17, 2012 4:31:55 AM
>  Subject: Re: RATS tests still running against F17?
>  
>  IMHO, the repoinfo.conf needs to be changed, but I'm not sure - rats*
>  tests go a bit around me. I'll have a look at it today.
>  
>  J.
>  
>  ----- Original Message -----
>  > From: "Adam Williamson" <awilliam(a)redhat.com&gt;
>  > To: autoqa-devel(a)lists.fedorahosted.org
>  > Sent: Thursday, July 12, 2012 9:23:24 PM
>  > Subject: RATS tests still running against F17?
>  > 
>  > So I took a quick look at the resultsdb and noticed that the RATS
>  > tests
>  > - rats_sanity and rats_install - still seem to be being run against
>  > F17,
>  > which isn't much use. (And they invariably fail). Is there a plan
>  > to
>  > switch them to Rawhide? Do we need some kind of release-transition
>  > SOP
>  > for AutoQA tests, to ensure they're bumped from Rawhide to Branched
>  > and
>  > then back to Rawhide after the stable release, in a timely manner?
>  > Thanks!
>  > --
>  > Adam Williamson
>  > Fedora QA Community Monkey
>  > IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
>  > http://www.happyassassin.net
>  > 
>  > _______________________________________________
>  > autoqa-devel mailing list
>  > autoqa-devel(a)lists.fedorahosted.org
>  > https://fedorahosted.org/mailman/listinfo/autoqa-devel
>  > 
>  _______________________________________________
>  autoqa-devel mailing list
>  autoqa-devel(a)lists.fedorahosted.org
>  https://fedorahosted.org/mailman/listinfo/autoqa-devel
>

[James Laska]

On Mon, 2012-07-16 at 22:31 -0400, Josef Skladanka wrote:
>  IMHO, the repoinfo.conf needs to be changed, but I'm not sure -
> rats* tests go a bit around me. I'll have a look at it today. 
https://fedoraproject.org/wiki/How_to_update_AutoQA_repoinfo.conf

During release milestones, a ticket should be filed to request adjusting
repoinfo.conf.  If I recall, we had a reminder in the mass_branching_sop
[1] and end_of_life_SOP [2].  I'm not finding the more obvious link in
the appropriate final release SOP. 

Perhaps a repoinfo.conf ticket is needed in
https://fedoraproject.org/wiki/Stage_final_release_for_mirrors?  Is that
the proper SOP for capturing the steps needed to release a new Fedora?

Thanks,
James

[1]
https://fedoraproject.org/wiki/Mass_Branching_SOP#Update_AutoQA_repoinfo....
[2] https://fedoraproject.org/wiki/End_of_life_SOP#AutoQA

[Kamil Paral]

>  So I took a quick look at the resultsdb and noticed that the RATS
>  tests
>  - rats_sanity and rats_install - still seem to be being run against
>  F17,
>  which isn't much use. (And they invariably fail).  
Thanks for noticing this. It's a bug. rats_install is not being run, rats_sanity is,
and it's useless, correct.

>  Is there a plan to
>  switch them to Rawhide? Do we need some kind of release-transition
>  SOP
>  for AutoQA tests, to ensure they're bumped from Rawhide to Branched
>  and
>  then back to Rawhide after the stable release, in a timely manner? 
repoinfo.conf update should be enough, which I did. But the control file for rats_sanity
is broken, I'm working on a fix.

I intend to have it scheduled only against Branched, not Rawhide. Since we've started
branching Rawhide, we don't concentrate on Rawhide itself so much, and it's not as
critical as before. Also it would consume machine and maintenance resources I would rather
spend elsewhere. So I think our tests should target Branched first, and ignore Rawhide for
now. Feel free to object.

[Kamil Paral]

>  repoinfo.conf update should be enough, which I did. But the control
>  file for rats_sanity is broken, I'm working on a fix.
>  
>  I intend to have it scheduled only against Branched, not Rawhide.
>  Since we've started branching Rawhide, we don't concentrate on
>  Rawhide itself so much, and it's not as critical as before. Also it
>  would consume machine and maintenance resources I would rather spend
>  elsewhere. So I think our tests should target Branched first, and
>  ignore Rawhide for now. Feel free to object.
>   
I have disabled rats_install on production and staging server for the moment. The patch is
here:


diff --git a/tests/rats_sanity/control.autoqa b/tests/rats_sanity/control.autoqa
index 6c99181..a6a8e10 100644
--- a/tests/rats_sanity/control.autoqa
+++ b/tests/rats_sanity/control.autoqa
@@ -1,13 +1,16 @@
 # vim: set syntax=python
 # Notice: Most recent documentation is available at doc/control.autoqa.template.
 
-# we want to run rats_sanity just for post-repo-update 
+import autoqa.repoinfo
+
+# we want to run rats_sanity just for post-repo-update
 if event in ['post-repo-update']:
     execute = True
 
-# Only advertise rats_sanity for branched* and rawhide* tests
-if autoqa_args.has_key('reponame'):
-    name = autoqa_args['reponame'].lower()
-    if not (name.startswith('rawhide') or name.startswith('branched')):
+    # only run rats_sanity for Branched repo (i.e. composes enabled)
+    if autoqa_args.has_key('name'):
+        repo = autoqa.repoinfo.repoinfo.getrepo(autoqa_args['name'])
+        if not repo['composes']:
+            execute = False
+    else:
         execute = False
-


I'll commit tomorrow if there are no concerns.

[Kamil Paral]

>  I'll commit tomorrow if there are no concerns. 
Fix pushed to master and copied to both servers. rats_sanity should not be scheduled now,
but it should automatically start running once F18 is Branched.