RFC: configuration via distributed database

[Jeff Darcy]

Many of the I/O parts of HekaFS rely on having consistent configuration
data, e.g. to ensure that servers export the same volumes with the same
credentials that clients are going to use.  Unfortunately, the way the
various hfs_* scripts manage configuration data (essentially by
forwarding user requests in their entirety to every node) has proven
tedious, inefficient, and error-prone.  In particular, many instances
have already been found of configuration data getting out of sync
between nodes, and there's no infrastructure for bringing them back
into sync after either an error or a failure.  That's why one of the
tasks for "HekaFS 1.1" has been to change the configuration subsystem
to use a distributed database instead.  Kaleb deserves the credit for
pointing out that we already have a pretty good distributed database
available to us - GlusterFS itself, which already has some of the
distribution and self-healing features we need.  Using a small private
filesystem also offers other advantages, such as easing configuration
backup or allowing the use of directories as containers for
multi-valued configuration variables.  There are still some details
that need to be worked out about how to configure that private
filesystem, hence this email.

Most of the code to deal with the configuration filesystem is in two
places: the hfs_utils module (e.g. open_db) and a new hfs_config
script.  This script would support the following operations:

	hfs_config serve
	Make the current node a server for the config filesystem.  You
	need at least one server for anything else to work; normally
	you'd have at least two or three to protect against node
	failures (with GlusterFS handling replication between them).

	hfs_config retire [node_name]
	Remove the named node, or the current node by default, from the
	list of config-filesystem servers.

	hfs_config init
	Populate the config filesystem with initial values.  Obviously
	there would have to be some protection against running this on
	an already-populated filesystem.

	hfs_config volfile
	Generate a client volfile for the config filesystem, e.g. to
	perform manual repair or actions for which no scripts exist
	(yet).

When it starts up, hekafsd would go through the following sequence:

	Check whether this node is supposed to be a config-filesystem
	server.  If so, generate a simple server volfile and spin up an
	instance of glusterfsd.

	Get the list of cluster members (currently from "gluster peer
	status"; in future it might be from a real cluster
	infrastructure).

	Query each member for its server status, using a curl request
	to hekafsd on that node.

	Construct a client volfile based on which members identify
	themselves as config-filesystem servers, including the local
	node if appropriate, and mount that.

	Verify the server list.  If a discrepancy is found (see below),
	take corrective action and re-mount if necessary.

The last step is necessary because a config-filesystem server might be
down while another node is querying for servers, and it would be
disastrous if this led to different nodes writing to different
overlapping subsets of servers.  The GlusterFS repair code would get
hopelessly confused, and the config filesystem would probably end up in
an unrecoverable state.  Therefore, each config-filesystem server
leaves a marker for itself *in the config filesystem*, to persist even
when that server is down.  (Needless to say, this marker should be
removed by "hfs_config retire" and the node_name option exists for the
specific purpose of removing a server that will never come back up.)
The list stored within the filesystem can then be checked against that
used for the initial mount.  If the two lists differ, then corrective
action can be taken before attempting the "real" mount.  Some examples
of corrective action might include:

	Manual sync from the "consensus" config to the local copy.

	Force a GlusterFS replica repair ("find ... | stat").

	Check versions/timestamps (maintained at the end of every user
	request) and perform more complex repair.

This approach - unlike the "broadcast operations" approach used
currently - should be robust against all but the most pathological
multiple-failure cases.  Now is the time to poke holes in it, if you're
so inclined.  Otherwise I'll begin implementing the infrastructure
described here, and converting scripts to use it.

[Pete Zaitcev]

On Tue, 6 Sep 2011 16:50:27 -0400
Jeff Darcy <jdarcy(a)redhat.com&gt; wrote:

>  Kaleb deserves the credit for
>  pointing out that we already have a pretty good distributed database
>  available to us - GlusterFS itself, which already has some of the
>  distribution and self-healing features we need. 
It is a clever idea if the chicken and egg problem gets resolved neatly.
But I'm just curious, did you guys think about using external
configuration management infrastructure, and if yes, what?

For example, we already rely on DNS (even if implicitly). How about
using the zone update feature to add-remove SRV records of servers
that have this "small private filesystem". They have port numbers.

Or, do not use private filesystem at all and roll our a Zookeeper
client (personally I hate both the complexity and verbosity of its API,
but I'm just asking).

Or, use rsyncd like in OpenStack (in Rackspace production really).

In short, I see what's approved but I wish to see what was rejected,
if you have the time to satisfy my curiosity.

>  When it starts up, hekafsd would go through [...] 
>  	Check whether this node is supposed to be a config-filesystem
>  	server.  If so, generate a simple server volfile and spin up an
>  	instance of glusterfsd. 
What does the "check" include, in more detail? Is there a seed
configuration file in /var/lib/hekafsd that "hfs_config init" creates?

-- Pete

[Jeff Darcy]

On Thu, 8 Sep 2011 13:29:32 -0600
Pete Zaitcev <zaitcev(a)redhat.com&gt; wrote:

>  On Tue, 6 Sep 2011 16:50:27 -0400
>  Jeff Darcy <jdarcy(a)redhat.com&gt; wrote:
>  
>  > Kaleb deserves the credit for
>  > pointing out that we already have a pretty good distributed database
>  > available to us - GlusterFS itself, which already has some of the
>  > distribution and self-healing features we need.
>  
>  It is a clever idea if the chicken and egg problem gets resolved
>  neatly. 
It's not quite a chicken and egg problem because HekaFS isn't using
itself.  It's just using GlusterFS - which is oblivious to HekaFS's
machinations - in a different way.

>  But I'm just curious, did you guys think about using external
>  configuration management infrastructure, and if yes, what?
>  
>  For example, we already rely on DNS (even if implicitly). How about
>  using the zone update feature to add-remove SRV records of servers
>  that have this "small private filesystem". They have port numbers. 
Relying on DNS in that way seems fraught with peril.  For one thing it
implies either a private DNS server or a complex security setup to
allow such arbitrary modification of records, and that will simply not
be acceptable to many users.  That bad decision was one of the main
obstacles to adoption of Hail, and I don't want it to be a similar kind
of barrier for HekaFS.

>  Or, do not use private filesystem at all and roll our a Zookeeper
>  client (personally I hate both the complexity and verbosity of its
>  API, but I'm just asking). 
I actually did think about using Zookeeper (or CLD).  On the one hand,
it would solve the config-consistency problem quite well.  On the
other hand, ...

* It would add a pretty nasty set of dependencies not only on ZK itself,
  but also on various libraries, the JVM itself, etc.

* It would increase startup/shutdown complexity and resource use, as
  there would be more daemons and ports to manage.

* Since we probably don't want to run ZK on every single server, we'd
  still need a way to identify which GlusterFS/HekaFS servers are also
  ZK servers . . . unless the ZK servers are separate, which would add
  even more complexity.

I'm not totally opposed to that approach, but it doesn't seem like the
incremental benefits outweigh the costs.  There are other distributed-DB
options that would pull in fewer dependencies and consume fewer
resources, but the other issues would remain and for every such option
I looked at (e.g. Voldemort, Riak) the result would have been the same.

>  Or, use rsyncd like in OpenStack (in Rackspace production really).
That's similar to what we do now.  The difference between broadcasting
high-level requests and broadcasting low-level data operations isn't
that much in that case.  It works OK in the normal cases, but leaves us
with the problem of whose copy is authoritative after a failure.
Building N-way consensus on top of two-way replication is exactly the
kind of thing I want to avoid; conflict resolution in the "weird" cases
is exactly what I want to push down to Someone Else's Code.

>  In short, I see what's approved but I wish to see what was
> rejected,
>  if you have the time to satisfy my curiosity.
>  
>  > When it starts up, hekafsd would go through [...]
>  
>  > 	Check whether this node is supposed to be a
>  > config-filesystem server.  If so, generate a simple server volfile
>  > and spin up an instance of glusterfsd.
>  
>  What does the "check" include, in more detail? Is there a seed
>  configuration file in /var/lib/hekafsd that "hfs_config init" creates? 
Correct, except that it's really more "hfs_config serve" that would do
what you describe; "hfs_config init" would populate the FS that's
shared between all of the servers.