Re: Production out of disk space [again]

[James Laska]

On Thu, 2011-05-05 at 07:42 -0600, Tim Flink wrote:
>  On 05/05/2011 07:34 AM, Kamil Paral wrote:
>  >> Hey gang,
>  >> 
>  >> Instead of resolving this issue manually by removing older test 
>  >> results, I thought others would might have some better ideas for
>  >> resolving the immediate issue of no free disk space.
>  >> 
>  >> I'm checking in with infrastructure to see if there are any free
>  >> disks that can be assigned to our guest to alleviate matters. In
>  >> the meantime, any recommendations to resolve this issue in the
>  >> short-term (today/tomorrow) so we can get jobs processing again?
>  > 
>  > Can we archive the large logs for further inspection and move it to
>  > another partition? That should help us going on quickly. Tomorrow I
>  > intend to write a cron script that will find large log files, gzip
>  > them and symlink the original files to them.
>  
>  Great minds think alike :)
>  
>  I don't think that we have another partition to move it to on that
>  machine since it's a VM, though. Is there another server we could move
>  them to? 
Not yet ... that'll be addressed "real soon now" though.

>  Another qnd option would be to move large logs to another publicly
>  available machine and replace the logs with html saying something like:
>  "This log has been moved to http://morediskspace/autoqa/jobnum/log" 
Can I just compress the existing logs that are over 100M for now?  Any
script suggestions for that?

>  I assume that the timeline for getting more disk space is still on
> the
>  order of next week, though? 
Yeah, that's the latest I have, and I'll probably add the realism
multiplier of x2.  But I'm reaching out to infrastructure now to explore
options with them also.

Thanks,
James