Outage: Unplanned outage with netapp filers - 2012-01-25 10:00 UTC
There was an unplanned outage in natapp filers in the phx2 location,
which lasted approximately 5 hours. Due to failover, service impacting
part of outage was very small.
To convert UTC to your local time, take a look at
http://fedoraproject.org/wiki/Infrastructure/UTCHowto
or run:
date -d '2012-01-25 10:00 UTC'
Reason for outage:
One of our pair of netapp filers failed at approx 10:00UTC. This
caused a failover of resources to the other netapp and no service
impact was seen. When the primary node was brought back on line it
attempted to take resources back, but failed to do so correctly. This
lead to a outage in nfs mounts on many affected Fedora servers. The
issue was corrected and node brought back on line. A ticket is open
with the vendor to provide a perm fix.
Affected Services:
Bodhi -
https://admin.fedoraproject.org/updates/
Docs -
http://docs.fedoraproject.org/
Fedora Account System -
https://admin.fedoraproject.org/accounts/
Fedora Community -
https://admin.fedoraproject.org/community/
Fedora Insight -
https://insight.fedoraproject.org/
Mirror List -
https://mirrors.fedoraproject.org/
Mirror Manager -
https://admin.fedoraproject.org/mirrormanager/
Package Database -
https://admin.fedoraproject.org/pkgdb/
Smolt -
http://smolts.org/
Start -
http://start.fedoraproject.org/
Wiki -
http://fedoraproject.org/wiki/
Unaffected Services:
Ask Fedora -
http://ask.fedoraproject.org/
BFO -
http://boot.fedoraproject.org/
Buildsystem -
http://koji.fedoraproject.org/
GIT / Source Control
DNS -
ns1.fedoraproject.org,
ns2.fedoraproject.org
Email system
Fedora Hosted -
https://fedorahosted.org/
Fedora People -
http://fedorapeople.org/
Main Website -
http://fedoraproject.org/
QA Services
Secondary Architectures
Torrent -
http://torrent.fedoraproject.org/
Ticket Link:
Contact Information:
Please join #fedora-admin in
irc.freenode.net or add comments to the
ticket for this outage above.