[fedora-arm] Status Update: ARM Koji Build System Outage

[M Abed]

Hello Everyone!

As you are aware that we had *some* issues with the build farm due to a
power outage yesterday. We didn't have too many chances to send updates. So
here it is:

After the power outage, all the critical servers and the builders were up
and running. However due to the power outage there was some other
networking issues at Seneca. A short while later some of the systems
rebooted and we started dealing with it as soon as we noticed.

At that point some of the management servers were still down. There were
issues with Apache, SE Linux or Networking in one server or another. After
those issues were dealt with, there was the NFS Locking issue and it turned
out to be due to some Firewall rules. All that got fixed last night.

However, we are still not 100%. Some of the builders are still timing out
as of this morning and that issue is still under investigation. Please
consider the build farm to be out of commission till further notice. You
can submit jobs and that may or may not go through.

We are going to bring the farm back up as fast as we can.  We apologize for
the inconvenience and thank you for your patience.

Today *will* be a better day and we will try to provide updates
more frequently. If you have any question please reply to this email or
express it in the channel.


Sincerely


Max Abed
Fedora ARM Team
Seneca CDOT
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.fedoraproject.org/pipermail/arm/attachments/20120313/c703fdba/attachment.html>