non-disclosure of infrastructure problem a management issue?

[Bj ø rn Tore Sund]

Nifty Fedora Mitch chose attack as the best defense:
> On Fri, Aug 22, 2008 at 10:36:21AM +1200, Clint Dilks wrote:
>> Bjoern Tore Sund wrote:
>>> It has now been a full week since the first announcement that Fedora
>>> had "infrastructure problems" and to stop updating systems.  Since
>>> then there has been two updates to the announcement, none of which
>>> have modified the "don't update" advice and noen of which has been
>>> specific as to the exact nature of the problems.  At one point we
>>> received a list of servers, but not services, which were back up and
>>> running.
>>>
>>> The University of Bergen has 500 linux clients running Fedora.  We
>>> average one reinstall/fresh install per day, often doing quite a lot
>>> more. Installs and reinstalls has had to stop completely, nightly
>>> updates have stopped, and until the nature of the problem is revealed
>>> we don't even know for certain whether it is safe for our IT staff to
>>> type admin passwords to our (RHEL-based, for the most part) servers
>>> from these work stations.
>
>With 500 clients ?

So far.  Got about 250 laptops coming into the system this autumn, as soon
as we have the setup and config regime properly structured and able to
handle it.  Should be ready sometime in September.

>Are you pulling updated from the internet or are
>you pulling from a local cache of "tested" updates.

I have often wished we had the manpower to do the latter.  Unfortunately, we
don't, so the local mirror is exactly that, a mirror.  One thing this
incident has taught us is to take regular backups of that mirror so that we
can roll back to a non-suspect version of the Fedora updates.  Didn't have
that before, really missed it the last couple of weeks.

>Are you using site specific kickstart config files that install local
>yum config files, ssh keys, sendmail setup and sudo config files so your admins
>can access the hosts without typing pass words?

Yes, to all.  Unfortunately that regime isn't 100% adhered to, which is
something we work on.  Equally unfortunately, we have had to give the
footwork guys sudo access to a limited set off commands.  Sudo with or
without passwords have different security implications, we've landed on
"with".

>What revision control of the config files?

Subversion.  Some distributed through nightly scripts using wget, some
through a commercial software package for server administration.

>I can see that the lack of updates would prove disconcerting
>but the inability to maintain day to day, another one just like
>yesterdays install seems fragile.

I'm sorry, but my English isn't good enough to parse that sentence
sufficiently to guess what you're trying to express.

>In business school there is a strategy of "owning your own
>dependencies".   The long term success stories in business include
>strong control of resources that they depend on.
>
>It is possible to manage yum and friends to allow only update packages
>resigned by your group at Bergan after testing them.

Indeed this is possible.  Unfortunately, we don't have the resources so we
are dependent on our Linux distro having those resources.  If I had
unlimited resources, this is not the only thing I would do differently.

>My last question -- what is the University of Bergin's written policy for
>this type and other risks.   Does university policy mandate the disclosure
>that you expect from RedHat.

It does, and we have.  Both when it has implicated our own users and when we
have uncovered compromised servers on our site being used for attacks
against other sites.

I'm sure your questions were part of a point you were making.  I trust that
you are happy with that point.  Me, I'm relieved that I finally have
concrete information on what has been happening and how it affects us.  In
the end I'm now more unhappy with RedHat than I am with Fedora - but that is
not a topic for this list.  At least Fedora told us _something_ was wrong.

-BT
-- 
Bjørn Tore Sund       Phone: 555-84894   Email:   bjorn.sund at it.uib.no
IT department         VIP:   81724       Support: http://bs.uib.no
Univ. of Bergen

When in fear and when in doubt, run in circles, scream and shout.