Deploying Red Hat Workstations

[Tom 'Needs A Hat' Mitchell]

On Tue, Mar 16, 2004 at 12:48:44PM -0500, Chris Purcell wrote:
> 
> We're in the process of deploying about 200 Fedora (or RHEL) workstations
> out to our remote offices scattered across the US.  I need a way that I
> can easily make changes to the workstations all at once.  I've been
> thinking of a hack that I would write (in Perl) that would go something
> like this...
> 
> 1) each workstation would execute a cron job daily that would download a
> script from our central server each day

Reconsider the reverse direction.

Have the central server push out to each host.  I assume that the
central management box is the best managed box you have.  Your
strategy as expressed gives 200+ boxes access to this central
resource.  That is not true with a push.  i.e. What if one of the 201
is hacked?

It is true that this is the reverse of the rhn update strategy but
their busines and goals sound different. Anyhow think clearly what the
risks and the access methods are.  Both push and pull will work....

> 2) that script would be executed by another cron job a few minutes later.
>  This script will contain any changes that I need to make.   If there
> aren't any updates for the day, then the script will be blank that day.

Avoid race conditions.  i.e. What if a download is slow, faile,
contains a typo or is interrupted and cron triggers the second
step. i.e. Will a partial download do bad stuff.  

Have a code design that will validate checksums even if it defaults to
true on day-one.

> Will this work?  There has to be something better than this out there.
> What do you guys do in this situation?   For example, we've deployed about
> 25 workstations out so far, but now I need to change this Perl script on
> each one of the machines.  What is the easiest way to push this script out
> besides SCP'ing it to each one individually.

Today, I would automate SCP from the server to a user account (not
root).  That account on each machine can have a ~/.ssh/known_hosts
file so you can have a shell loop do the transfers and not have to put
passwds in the scripts.  You will want to manage bandwidth and not
trigger all the transfers at once.  Remember that a cron job on all
200 machines could prove intertesting when you get the clocks
synchronized.

A push model facilitates testing and verification of changes.  Do 5%
on Monday, verify those 5%, Do 20% on Tuesday and verify.  Finish on
Wed.  Thursday Friday get ready for Monday, Sat/Sunday sleep on it.
Monday check and begin again.

With your pull strategy design can you stage the distribution of changes?
Can you push emergency changes on a new schedule?

Use different pseudo users for different functions when it makes sense.
Do not have a single user (root) doing all the work (minimize sudo?).

Inside your perl script you can also use wget to pull files (or sftp
to push/pull files).  Consider then reject having your script update
itself.  

Do a simple bubble chart with arrows that lets you track data flow and
authentication.

One path from the central office should be a "Look but don't touch"
account.  I woule call it an audit function (account/machine).

A different path from the central office should be a "Can fix and
repair stuff" root equivalent account (perhaps with no
~/.ssh/known_hosts file).

The goal of the bubble chart and arrows is to ensure that there is no
circle of validation where if one box is hacked all the others fall
over (or can be discovered for the same attack).

Next look at the bubble chart and cover each resource and building of
resources, then check that some sane backup plan is possible.  Today a
<$1000.00 laptop in a fire proof safe in a different building with
ssh-keys and passwds could facilitate recovery.  A fifty cent CDROM
.... in the fire safe?  What if the laptop is stolen?

Next put dollar signs (or Euros, etc.) on each bubble.  Data has value
as does the hardware.

-- 
	T o m  M i t c h e l l 
	/dev/null the ultimate in secure storage.