It appears that we have lost one of our Xen servers for reasons not yet discovered. Systems affected seem to include bodhi, koji, account system, and possibly others.
The reason for the outage is still being investigated, as well as attempts to restore service. We're sorry for any inconvenience this may cause. More announcements will be made as services come back on-line.
On Sunday 20 January 2008, Jesse Keating wrote:
It appears that we have lost one of our Xen servers for reasons not yet discovered. Systems affected seem to include bodhi, koji, account system, and possibly others.
The reason for the outage is still being investigated, as well as attempts to restore service. We're sorry for any inconvenience this may cause. More announcements will be made as services come back on-line.
Service should now be restored to all services. Please let us know if you have issues.
We are still investigating the cause of the outage.
Dennis
Dennis Gilmore wrote:
On Sunday 20 January 2008, Jesse Keating wrote:
It appears that we have lost one of our Xen servers for reasons not yet discovered. Systems affected seem to include bodhi, koji, account system, and possibly others.
The reason for the outage is still being investigated, as well as attempts to restore service. We're sorry for any inconvenience this may cause. More announcements will be made as services come back on-line.
Service should now be restored to all services. Please let us know if you have issues.
We are still investigating the cause of the outage.
Dennis
Just for reference, Jesse's email is at 625am pst, and I know koji was unresponsive at 12am pst today up to 4am (thought hopefully you're able to trace that down a little better).
On Sun, 20 Jan 2008 11:58:38 -0800 Andrew Farris lordmorgul@gmail.com wrote:
Just for reference, Jesse's email is at 625am pst, and I know koji was unresponsive at 12am pst today up to 4am (thought hopefully you're able to trace that down a little better).
We've had multiple failures of the Xen host from what I can gather :/
On Sun, 2008-01-20 at 20:39 -0500, Jesse Keating wrote:
On Sun, 20 Jan 2008 11:58:38 -0800 Andrew Farris lordmorgul@gmail.com wrote:
Just for reference, Jesse's email is at 625am pst, and I know koji was unresponsive at 12am pst today up to 4am (thought hopefully you're able to trace that down a little better).
We've had multiple failures of the Xen host from what I can gather :/
And while koji responds now, it doesn't appear to build anything...
On Sun, 20 Jan 2008 20:52:58 -0500 Matthias Clasen mclasen@redhat.com wrote:
And while koji responds now, it doesn't appear to build anything...
I just fixed that. The builder daemon needed to be restarted after the prolonged hub outage. The builders are now picking up jobs again and will chew through the backlog. In fact 5 builds have just completed.
On Sunday, 20 January 2008 at 18:11, Dennis Gilmore wrote:
On Sunday 20 January 2008, Jesse Keating wrote:
It appears that we have lost one of our Xen servers for reasons not yet discovered. Systems affected seem to include bodhi, koji, account system, and possibly others.
The reason for the outage is still being investigated, as well as attempts to restore service. We're sorry for any inconvenience this may cause. More announcements will be made as services come back on-line.
Service should now be restored to all services. Please let us know if you have issues.
koji website is (still?) unresponsive as of 20:09 UTC.
Regards, R.
Dominik 'Rathann' Mierzejewski wrote:
On Sunday, 20 January 2008 at 18:11, Dennis Gilmore wrote:
On Sunday 20 January 2008, Jesse Keating wrote:
It appears that we have lost one of our Xen servers for reasons not yet discovered. Systems affected seem to include bodhi, koji, account system, and possibly others.
The reason for the outage is still being investigated, as well as attempts to restore service. We're sorry for any inconvenience this may cause. More announcements will be made as services come back on-line.
Service should now be restored to all services. Please let us know if you have issues.
koji website is (still?) unresponsive as of 20:09 UTC.
Regards, R.
Alive to west coast as of 12:38 pst.
-
Andrew Farris -
Dennis Gilmore -
Dominik 'Rathann' Mierzejewski -
Jesse Keating -
Matthias Clasen