Outage: Unplanned outage with netapp filers - 2012-01-25 10:00 UTC

[Kevin Fenzi]

Outage: Unplanned outage with netapp filers - 2012-01-25 10:00 UTC

 There was an unplanned outage in natapp filers in the phx2 location,
 which lasted approximately 5 hours. Due to failover, service impacting
 part of outage was very small.

 To convert UTC to your local time, take a look at
 http://fedoraproject.org/wiki/Infrastructure/UTCHowto
 or run:

 date -d '2012-01-25 10:00 UTC'

 Reason for outage:

 One of our pair of netapp filers failed at approx 10:00UTC. This
 caused a failover of resources to the other netapp and no service
 impact was seen. When the primary node was brought back on line it
 attempted to take resources back, but failed to do so correctly. This
 lead to a outage in nfs mounts on many affected Fedora servers. The
 issue was corrected and node brought back on line. A ticket is open
 with the vendor to provide a perm fix.

 Affected Services:

 Bodhi - https://admin.fedoraproject.org/updates/
 Docs - http://docs.fedoraproject.org/
 Fedora Account System - https://admin.fedoraproject.org/accounts/
 Fedora Community - https://admin.fedoraproject.org/community/
 Fedora Insight - https://insight.fedoraproject.org/
 Mirror List - https://mirrors.fedoraproject.org/
 Mirror Manager - https://admin.fedoraproject.org/mirrormanager/
 Package Database - https://admin.fedoraproject.org/pkgdb/
 Smolt - http://smolts.org/
 Start - http://start.fedoraproject.org/
 Wiki - http://fedoraproject.org/wiki/

 Unaffected Services:

 Ask Fedora - http://ask.fedoraproject.org/
 BFO - http://boot.fedoraproject.org/
 Buildsystem - http://koji.fedoraproject.org/
 GIT / Source Control
 DNS - ns1.fedoraproject.org, ns2.fedoraproject.org
 Email system
 Fedora Hosted - https://fedorahosted.org/
 Fedora People - http://fedorapeople.org/
 Main Website - http://fedoraproject.org/
 QA Services
 Secondary Architectures
 Torrent - http://torrent.fedoraproject.org/

 Ticket Link:

 Contact Information:

 Please join #fedora-admin in irc.freenode.net or add comments to the
 ticket for this outage above.