[AutoQA] #321: Store test logs for at least a month

[fedora-badges]

#321: Store test logs for at least a month
------------------------+---------------------------------------------------
 Reporter:  kparal      |       Owner:                   
     Type:  task        |      Status:  new              
 Priority:  major       |   Milestone:  Nice to have soon
Component:  production  |    Keywords:                   
------------------------+---------------------------------------------------
 Today is May 3rd and some test output from April 30th is already
 inaccessible
 (https://admin.fedoraproject.org/updates/zabbix-1.8.5-1.fc15). That is
 problematic for package maintainers and also for us when dealing with test
 issues. We should aim to store results for at least a month.

 James mentioned that log pruning was run every 15 days, but sometimes it
 had to run more often, because e.g. some depcheck result logs may take up
 to several hundred MBs. James, can you provide links to such logs in order
 for us to evaluate that issue? Also, you can provide some disk space
 statistics? E.g. how much data we usually generate per week, how much disk
 space we have available, etc.

 If we can't extend available disk space easily, let's aim for reducing
 logs size. Can we do some form of transparent filesystem compression? From
 my experience depcheck logs have extremely good compress ratio.

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/321>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#321: Store test logs for at least a month
------------------------+---------------------------------------------------
 Reporter:  kparal      |        Owner:                   
     Type:  task        |       Status:  new              
 Priority:  major       |    Milestone:  Nice to have soon
Component:  production  |   Resolution:                   
 Keywords:              |  
------------------------+---------------------------------------------------
Comment (by jlaska):

 tflink suspects that tmpwatch in RHEL5 isn't honoring the time argument of
 "15d".  Using a `find` command, we've confirmed that there are no result
 logs older than 1 day on the system.  Per Tim's suggestion, I've adjusted
 the tmpwatch command used in /etc/cron.d/autoqa to the following ...


 {{{
 0 13 * * * autotest /usr/sbin/tmpwatch -vv --dirmtime -umc -f 360
 /usr/share/autotest/results/ -X README
 }}}

 This should tell us the timestamp of files being removed (-vv) and only
 remove files older than 360 hours (15 days).  Hopefully with the above
 change, this will give us a good sense for what is happening.    That's
 certainly not expected.  Hopefully this is Investigation continues.

 I'd love to adjust to 30 days as a long-term goal, but it's just not
 practical given our current storage constraints.  Our storage needs are
 definitely longer sufficient since autoqa-0.4.4 increased the number of
 test cases, events ... and results.  As we start to collect more results
 longer than 1 day ... I'll gather some average per day numbers.

 We also have some jobs that have *enormous* test results (3.0G).  We'll
 also need to explore why those logs are obscenely huge.

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/321#comment:1>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#321: Store test logs for at least a month
------------------------+---------------------------------------------------
 Reporter:  kparal      |        Owner:                   
     Type:  task        |       Status:  new              
 Priority:  major       |    Milestone:  Nice to have soon
Component:  production  |   Resolution:                   
 Keywords:              |  
------------------------+---------------------------------------------------
Comment (by tflink):

 As an update, we have another job with a 1G debug log:
  * http://autoqa.fedoraproject.org/results/91524-autotest/172.16.0.19/

 I made a compressed tarball (159M) of the results directory if anyone is
 interested:
  *
 http://tflink.fedorapeople.org/autoqa/logsize_issue/91524_largelog_exampl...
  *
 http://tflink.fedorapeople.org/autoqa/logsize_issue/91524_largelog_exampl...

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/321#comment:2>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#321: Store test logs for at least a month
------------------------+---------------------------------------------------
 Reporter:  kparal      |        Owner:                   
     Type:  task        |       Status:  new              
 Priority:  major       |    Milestone:  Nice to have soon
Component:  production  |   Resolution:                   
 Keywords:              |  
------------------------+---------------------------------------------------
Comment (by kparal):

 The log tflink posted seems pretty straight-forward. Depcheck (or yum) got
 stuck in a loop and ran until some watchdog timer was reached/memory was
 exhausted/free disk space ran out (I don't know which one). We should
 provide some protection against those cases, either depcheck specific or
 general one. I'll create new ticket about that. The best would probably be
 to measure the size of the output log and crash the test if it exceeds
 some quota (like 10 or 100 MB). We can also automatically compress such
 logs (they will be useful just for us anyway).

 If we get rid of these enormous test logs, do we have enough disk space
 for storing our results for at least a month? (That's the limit I'd like
 to aim for.) What can we do else to improve it?

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/321#comment:3>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#321: Store test logs for at least a month
------------------------+---------------------------------------------------
 Reporter:  kparal      |        Owner:                   
     Type:  task        |       Status:  new              
 Priority:  major       |    Milestone:  Nice to have soon
Component:  production  |   Resolution:                   
 Keywords:              |  
------------------------+---------------------------------------------------
Comment (by jlaska):

 Replying to [comment:3 kparal]:
>  If we get rid of these enormous test logs, do we have enough disk
> space
>  
 for storing our results for at least a month? (That's the limit I'd like
 to aim for.) What can we do else to improve it?

 I don't know yet.  It's a function of the average number of tests run per
 day, average log size per test.  I'm pretty confident the amount of space
 we have now won't be sufficient for one month of logs.  However, I'd still
 like to crunch some numbers.

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/321#comment:4>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project

[fedora-badges]

#321: Store test logs for at least a month
------------------------+---------------------
 Reporter:  kparal      |       Owner:
     Type:  task        |      Status:  closed
 Priority:  major       |   Milestone:  0.8.0
Component:  production  |  Resolution:  fixed
 Keywords:              |
------------------------+---------------------
Changes (by kparal):

 * status:  new => closed
 * resolution:   => fixed
 * milestone:  Nice to have soon => 0.8.0


Comment:

 We currently store old logs for 2 months. That is about 40GB and half a
 million of files. We might consider lowering the limit to just 1 month,
 but we definitely solved this ticket.

-- 
Ticket URL: <https://fedorahosted.org/autoqa/ticket/321#comment:5>
AutoQA <http://autoqa.fedorahosted.org>
Automated QA project