As a heads up, taskotron.stg is having some issues at the moment stemming from a full disk. It'll be a few weeks before production hits the same disk problem, so we have some time to fix things before it becomes really critical.
As an odd side note - this is the second time that we've been saved from a production outage because stg has been running longer than production :)
The timing of this isn't great for fixing it quickly as there's an outage for a critical infra system starting shortly and I don't think I can get it fixed before that starts.
I'll send out another email once it's been fixed.
Tim
On Fri, 25 Sep 2015 08:20:00 -0600 Tim Flink tflink@redhat.com wrote:
As a heads up, taskotron.stg is having some issues at the moment stemming from a full disk. It'll be a few weeks before production hits the same disk problem, so we have some time to fix things before it becomes really critical.
As an odd side note - this is the second time that we've been saved from a production outage because stg has been running longer than production :)
The timing of this isn't great for fixing it quickly as there's an outage for a critical infra system starting shortly and I don't think I can get it fixed before that starts.
I'll send out another email once it's been fixed.
As a short term solution, I've tarred up a bunch of old records to free up inodes on the stg buildmaster.
For a longer term solution, the master needs to be deployed differently and we have to figure out a data retention policy for the old logs and build data.
Tim
qa-devel@lists.fedoraproject.org
-
Tim Flink