how about oneSIS? was: "Stateless Linux" project

[Josh England]

Hello all,

I'd like to introduce to the list a piece of software that
just happens to do exactly what this project sounds like its trying to
accomplish.  The software handles the root image of any number of nodes,
but allows for node/role specific behavior.  It allows you to deploy
a local cache of the root image onto a local disk, and a bootloader
if desired.  No matter what, though, the root filesystem is bit-for-bit
identical everywhere.  Deviations are handled by creating soft links
to/from a tmpfs ramdisk.  It really gives ultimate flexibility as to
how/where components of the filesystem should live (localdisk/ramdisk/NFS).

It can bootstrap the traditional NFSroot way (root=/dev/nfs) or use an
initrd to bootstrap in any number of creative ways.  The framework is
in place to be able to easily boot over any network filesystem
(including Lustre when it arrives), or over any network medium 
(including InfiniBand).  It uses an initrd 'template' to create new
initrds with very customizable behavior.

The project is called oneSIS (http://onesis.sourceforge.net).  We've used
it in production for running clusters for years, but I've just recently
(early this year) re-written it from scratch to make it cleaner and more
flexible.  Grab the CVS version from sourceforge and check it out.  It
makes everything the "Stateless linux" project is trying to do very, very
easy.

There is more information in the oneSIS software manual
(http://onesis.sourceforge.net/oneSIS-manual.pdf),  and I currently
have a paper being reviewed for submission, or I'd send it also.

-JE
-----------------------------------------------
Josh England
Sandia National Laboratory, Livermore, CA
Visualization and Scientific Computing
email: jjengla(a)sandia.gov
phone: [phone]



Miles Sabin wrote,
I think this is a fine idea. I have a few questions tho' ...

Outside of largish corporates where clones are common, there are smaller 
environments where machines are mostly (95%+, say) similar, but where 
there are machine-by-machine variations which fall outside the set of 
things which are covered by per-user homes or hardware auto- 
configuration. For instance, differences in which services should be 
running, their configuration, or minor kernel tweaks. Are such 
environments outside the scope of this proposal, or is there some plan 
to support per-machine or per-group/class deltas from a common 
baseline?

One thing that's always stopped me from bothering with NFS mounted homes 
is that for me a different machine quite often implies a different 
role, hence that the stuff that's appropriate to be in my home varies. 
This also tends to be a delta from a common baseline, so handling the 
variability by having distinct users for each role doesn't seem like 
quite the right answer. Caching doesn't seem likely to change this, and 
the "recreate Joe's workstation" scenario in the document seems to 
wander into this kind of territory. Do you have any thoughts about 
handling this?

Slightly related to the previous two questions ... My laptop moves 
between my home network, my work network, business-client networks, 
connected by dialup ISP, and disconnected. In each scenario the 
services running, their configuration, and what's appropriate for my 
home dir vary slightly. Even tho' this is pretty clearly beyond the 
scope of your proposal, there seems to be at least something in common: 
the setup is mostly the same, but there are some small deltas. Would it 
make sense to have a common mechanism to handle this?

I think all my questions boil down to: looks great for the 95% of stuff 
which stays the same, but how do you deal with the 5% of stuff which 
varies? And can you do it in a way which is 95% less hassle than 
treating the machines as completely separate entities?

Cheers,


Miles

[Havoc Pennington]

Hi,

On Tue, 2004-09-14 at 08:06 -0700, Josh England wrote:
>  The project is called oneSIS (http://onesis.sourceforge.net). 
> We've used
>  it in production for running clusters for years, but I've just recently
>  (early this year) re-written it from scratch to make it cleaner and more
>  flexible.  Grab the CVS version from sourceforge and check it out.  It
>  makes everything the "Stateless linux" project is trying to do very, very
>  easy. 
Reading the manual now, this looks great.

The changes we're making to Fedora should eliminate the rc script
changes you mention here:
http://onesis.sourceforge.net/oneSIS-manual/node22.html

This is very clever:
http://onesis.sourceforge.net/oneSIS-manual/node17.html

Anyway, I think it will take us some time to digest how this maps to the
desktop/server case in addition to clusters, and how to fit it together
with the work we have so far. Your thoughts on how to pull it all
together are welcome.

Havoc

[Josh England]

On Tue, 2004-09-14 at 18:07, Havoc Pennington wrote:
>  Anyway, I think it will take us some time to digest how this maps to
> the
>  desktop/server case in addition to clusters, and how to fit it together
>  with the work we have so far. Your thoughts on how to pull it all
>  together are welcome. 
Havoc,

Thanks for taking a look at oneSIS.  I'm certainly willing to discuss
how I can help it fit in with your ideas for Stateless Linux.

All of the abstractions in oneSIS are at the file level, so I believe it
can be used for many diskless-like environments besides clustering.  The
'compute/admin/etc' classes in the manual are completely arbitrary but
useful class names for a cluster system.  I have set up X on diskless
nodes before with only minor trickery, and it works just fine.  I think
a single INCLUDE file could be tailored to provide a working base system
capable of starting X, etc, on however many client nodes.

What kind of base functionality do you want out of the client nodes? 
What daemons would run...what kind of roles do you expect...what bits of
the root FS should live on a local disk..what should the bootstrap look
like?

In clustering, it is generally assumed that the whole cluster is on a
protected network, so oneSIS doesn't address any security concerns
directly, but I would love to hear ideas for how to improve that.

-JE

[John Hearns]

On Wed, 2004-09-15 at 07:55, Josh England wrote:

>  In clustering, it is generally assumed that the whole cluster is on
> a
>  protected network, so oneSIS doesn't address any security concerns
>  directly, but I would love to hear ideas for how to improve that. 
Agreed. There are more requests coming from users to have something
else than the traditional Beowulf 'master with two interfaces' network
architectures. Improvements in security will only help.