Outage: Fedora related services down 2010-12-15 22:00 UTC

[Stephen John Smoogen]

This outage is considered finished and closed at the moment.

Due to some change in network, DNS traffic was not being properly
moving between Fedora network and Red Hat filer. It appears that in
doing so the filer's NFS permission caches were being flooded with bad
lookups thus increasing CPU and lowering throughput.


On Wed, Dec 15, 2010 at 15:42, Stephen John Smoogen <smooge at gmail.com> wrote:
> Due to problems with some NFS in our PHX2 facility we are experiencing
> diminished capacity to several of our services. We are working with
> our provider and engineers on how to deal with this issue soon.
>
> To convert UTC to your local time, take a look at
> http://fedoraproject.org/wiki/Infrastructure/UTCHowto
> or run:
>
> date -d '2010-12-15 22:00 UTC'
>
> Reason for outage:
> NFS operations with filer are peaking below expected rates causing
> hangs on NFS clients.
>
> Affected Services:
>
> BFO - http://boot.fedoraproject.org/
> Buildsystem - http://koji.fedoraproject.org/
> CVS / Source Control
> Main Website - http://fedoraproject.org/
> Mirror List - https://mirrors.fedoraproject.org/
> Mirror Manager - https://admin.fedoraproject.org/mirrormanager/
> Package Database - https://admin.fedoraproject.org/pkgdb/
>
>
> Unaffected Services:
>
> Bodhi - https://admin.fedoraproject.org/updates/
> DNS - ns1.fedoraproject.org, ns2.fedoraproject.org
> Docs - http://docs.fedoraproject.org/
> Email system
> Fedora Account System - https://admin.fedoraproject.org/accounts/
> Fedora Community - https://admin.fedoraproject.org/community/
> Fedora Hosted - https://fedorahosted.org/
> Fedora People - http://fedorapeople.org/
> Fedora Talk - http://talk.fedoraproject.org/
> Smolt - http://smolts.org/
> Spins - http://spins.fedoraproject.org/
> Start - http://start.fedoraproject.org/
> Torrent - http://torrent.fedoraproject.org/
> Translation Services - http://translate.fedoraproject.org/
> Wiki - http://fedoraproject.org/wiki/
>
>
> Ticket Link:
>
> https://fedorahosted.org/fedora-infrastructure/ticket/2516
>
> Contact Information:
>
> Please join #fedora-admin in irc.freenode.net or respond to this email to
> track the status of this outage.
>
>
> --
> Stephen J Smoogen.
> "The core skill of innovators is error recovery, not failure avoidance."
> Randy Nelson, President of Pixar University.
> "Let us be kind, one to another, for most of us are fighting a hard
> battle." -- Ian MacLaren
>



-- 
Stephen J Smoogen.
"The core skill of innovators is error recovery, not failure avoidance."
Randy Nelson, President of Pixar University.
"Let us be kind, one to another, for most of us are fighting a hard
battle." -- Ian MacLaren