[AutoQA] #427: Purge unneeded files in autotest results dir

[fedora-badges]

#427: Purge unneeded files in autotest results dir
-------------------------+-------------------------
  Reporter:  kparal      |      Owner:
      Type:  task        |     Status:  new
  Priority:  minor       |  Milestone:  Finger Food
 Component:  production  |   Keywords:
Blocked By:              |   Blocking:
-------------------------+-------------------------
 Currently we have a script that periodically traverses through autotest
 results dir and compresses too large files. Also we use tmpwatch to remove
 too old results. But we still sometimes have problems with disk space and
 number of inodes on our servers. Autotest stores large number of files in
 each result directory. By cleaning those we don't want we can save
 resources, speed up execution of maintenance scripts and potential
 debugging (like grepping through results).

 Let's create a script that would traverse the results directory
 periodically and remove unneeded files. It would also compress large
 files, so it would call or merge the existing script.

 If we look into an example results dir:
 http://autoqa-stg.fedoraproject.org/results/100310-autotest/qa07.qa/
 I believe we need to keep:
 {{{
 /control
 /debug/*.DEBUG
 /host_keyvals
 /keyval
 /rpmguard/debug/*.DEBUG ("rpmguard" is a dynamic test name here)
 /rpmguard/keyval
 /rpmguard/results
 /rpmguard/status
 /status.log
 /sysinfo/installed_packages
 /sysinfo/uname
 }}}

 The rest of it can be deleted.

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/427>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#427: Purge unneeded files in autotest results dir
------------------------+--------------------------
 Reporter:  kparal      |       Owner:
     Type:  task        |      Status:  new
 Priority:  minor       |   Milestone:  Finger Food
Component:  production  |  Resolution:
 Keywords:              |  Blocked By:
 Blocking:              |
------------------------+--------------------------
Description changed by kparal:

Old description:

>  Currently we have a script that periodically traverses through
> autotest
>  results dir and compresses too large files. Also we use tmpwatch to
>  remove too old results. But we still sometimes have problems with disk
>  space and number of inodes on our servers. Autotest stores large number
>  of files in each result directory. By cleaning those we don't want we can
>  save resources, speed up execution of maintenance scripts and potential
>  debugging (like grepping through results).
>
>  Let's create a script that would traverse the results directory
>  periodically and remove unneeded files. It would also compress large
>  files, so it would call or merge the existing script.
>
>  If we look into an example results dir:
>  http://autoqa-stg.fedoraproject.org/results/100310-autotest/qa07.qa/
>  I believe we need to keep:
>  {{{
>  /control
>  /debug/*.DEBUG
>  /host_keyvals
>  /keyval
>  /rpmguard/debug/*.DEBUG ("rpmguard" is a dynamic test name here)
>  /rpmguard/keyval
>  /rpmguard/results
>  /rpmguard/status
>  /status.log
>  /sysinfo/installed_packages
>  /sysinfo/uname
>  }}}
>
>  The rest of it can be deleted. 
New description:

 Currently we have a script that periodically traverses through autotest
 results dir and compresses too large files. Also we use tmpwatch to remove
 too old results. But we still sometimes have problems with disk space and
 number of inodes on our servers. Autotest stores large number of files in
 each result directory. By cleaning those we don't want we can save
 resources, speed up execution of maintenance scripts and potential
 debugging (like grepping through results).

 Let's create a script that would traverse the results directory
 periodically and remove unneeded files. It would also compress large
 files, so it would call or merge the existing script.

 If we look into an example results dir:
 http://autoqa-stg.fedoraproject.org/results/100310-autotest/qa07.qa/
 I believe we need to keep:
 {{{
 /control
 /debug
 /host_keyvals
 /keyval
 /<test_name>/debug/*.DEBUG
 /<test_name>/keyval
 /<test_name>/results
 /<test_name>/status
 /status.log
 /sysinfo/installed_packages
 /sysinfo/uname
 }}}

 The rest of it can be deleted.

--

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/427#comment:1>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#427: Purge unneeded files in autotest results dir
------------------------+--------------------------
 Reporter:  kparal      |       Owner:  jsedlak
     Type:  task        |      Status:  new
 Priority:  minor       |   Milestone:  Finger Food
Component:  production  |  Resolution:
 Keywords:              |  Blocked By:
 Blocking:              |
------------------------+--------------------------
Changes (by kparal):

 * owner:   => jsedlak


-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/427#comment:2>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#427: Purge unneeded files in autotest results dir
------------------------+--------------------------
 Reporter:  kparal      |       Owner:  jsedlak
     Type:  task        |      Status:  new
 Priority:  minor       |   Milestone:  Finger Food
Component:  production  |  Resolution:
 Keywords:              |  Blocked By:
 Blocking:              |
------------------------+--------------------------

Comment (by jsedlak):

 Requested script is in "cleanup" branch, in tools/cleanup_result_tree.py.

 Review and test it here:

https://fedorahosted.org/autoqa/browser/tools/cleanup_result_tree.py?rev=...

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/427#comment:3>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#427: Purge unneeded files in autotest results dir
------------------------+--------------------------
 Reporter:  kparal      |       Owner:  jsedlak
     Type:  task        |      Status:  new
 Priority:  minor       |   Milestone:  Finger Food
Component:  production  |  Resolution:
 Keywords:              |  Blocked By:
 Blocking:              |
------------------------+--------------------------

Comment (by kparal):

 Jan, I did some minor changes, please have a look whether I haven't broken
 it too much.

 Also, please replace argparse with optparse, we have RHEL 6 on our servers
 and there is only Python 2.6. Thanks.

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/427#comment:4>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#427: Purge unneeded files in autotest results dir
------------------------+--------------------------
 Reporter:  kparal      |       Owner:  jsedlak
     Type:  task        |      Status:  new
 Priority:  minor       |   Milestone:  Finger Food
Component:  production  |  Resolution:
 Keywords:              |  Blocked By:
 Blocking:              |
------------------------+--------------------------

Comment (by kparal):

 Deployed at autoqa-stg, if everything looks OK, we will put it on autoqa-
 production soon.

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/427#comment:5>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#427: Purge unneeded files in autotest results dir
------------------------+--------------------------
 Reporter:  kparal      |       Owner:  jsedlak
     Type:  task        |      Status:  new
 Priority:  minor       |   Milestone:  Finger Food
Component:  production  |  Resolution:
 Keywords:              |  Blocked By:
 Blocking:              |
------------------------+--------------------------

Comment (by kparal):

 Jan, currently if our gzipped file is larger than the threshold, we try to
 gzip it again, right? And again, and again.

 Please fix. I have disabled script execution for now.

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/427#comment:6>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#427: Purge unneeded files in autotest results dir
------------------------+----------------------
 Reporter:  kparal      |       Owner:  jsedlak
     Type:  task        |      Status:  closed
 Priority:  minor       |   Milestone:  0.9.0
Component:  production  |  Resolution:  fixed
 Keywords:              |  Blocked By:
 Blocking:              |
------------------------+----------------------
Changes (by jsedlak):

 * milestone:  Finger Food => 0.9.0
 * resolution:   => fixed
 * status:  new => closed


Comment:

 Required functionality pushed into master by commit d346b84e4483bb7.

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/427#comment:7>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#427: Purge unneeded files in autotest results dir
------------------------+----------------------
 Reporter:  kparal      |       Owner:  jsedlak
     Type:  task        |      Status:  closed
 Priority:  minor       |   Milestone:  0.9.0
Component:  production  |  Resolution:  fixed
 Keywords:              |  Blocked By:
 Blocking:              |
------------------------+----------------------

Comment (by kparal):

 Some numbers: The script reduced our results directory from 21 GB and 2.5
 million files on autoqa-prd to 10 GB and 1.2 million files. 50% reduction.
 Thanks Jan.

 I have set it up in cron both on autoqa-stg and autoqa-prd.

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/427#comment:8>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project