BerkeleyDB and config files

[Michael A. Peters]

I'd like to suggest a change to the way Fedora Core does some things.
I'm not talking about "standard" linux config files like /etc/passwd or / 
etc/group or /etc/fstab (though the latter would be nice) - I'm talking  
about Fedora (and RH I suppose since Fedora has become a RH test bed)  
specific configuration files, such as /etc/security/console.perms and / 
etc/sysconfig/network-scripts/ifcfg-[iface] type stuff.

Rather than store them in flat files, which have different  
configuration parameters depending upon the file etc., use (maybe  
optionally) BerkeleyDB.

bdb is already needed for the rpm database, so a Fedora system will  
have bdb installed.

Both Python and Perl (as well as many other languages) already have  
good bdb interfaces, and both Python and Perl also have gtk+ bindings  
too.

If an embedded database was available for storing the configuration  
information, it would be simpler for packagers to script additions to  
those files, if necessary - IE the rpm for TiLP (software for talking  
to a Ti graphing calculator) could easily add (if needed) a <ticalc>  
class to console.perms with the necessary device node entries, and then  
add the needed permission definitions for the device class.

It also would allow gui administration for some configuration nodes for  
which there is not a gui. Kind of like how I can change some things in  
gnome using gconf-editor for which there isn't an existing control  
panel ready made for the task.

It also would make it easier for network administrators who set up box  
after box with needs to customize some of the default configurations to  
do so - after the install, a single documented script makes the needed  
entries/changes to the database and it's done.

[Iago Rubio]

On Thu, 2004-10-14 at 06:50, Michael A. Peters wrote:
>  I'd like to suggest a change to the way Fedora Core does some
> things.
>  I'm not talking about "standard" linux config files like /etc/passwd or / 
>  etc/group or /etc/fstab (though the latter would be nice) - I'm talking  
>  about Fedora (and RH I suppose since Fedora has become a RH test bed)  
>  specific configuration files, such as /etc/security/console.perms and / 
>  etc/sysconfig/network-scripts/ifcfg-[iface] type stuff.
>  
>  Rather than store them in flat files, which have different  
>  configuration parameters depending upon the file etc., use (maybe  
>  optionally) BerkeleyDB. 
Just to point that I don't like central configuration databases, and I
don't think it's the way to go.

They're single failure points for the whole system. 

In fact with current selinux improvements, you will not be able to have
a central database for all the system, but some smaller databases with
different security contexts, to manage MAC on different roles.

With this scenario, I don't see any improvements with this changes.

Flat files are pretty nice to manage/label.

>  bdb is already needed for the rpm database, so a Fedora system will 
>
>  have bdb installed.
>  
>  Both Python and Perl (as well as many other languages) already have  
>  good bdb interfaces, and both Python and Perl also have gtk+ bindings  
>  too. 
What's sure is any scripting language will support open/write/read
operations on flat files.

>  If an embedded database was available for storing the configuration 
>
>  information, it would be simpler for packagers to script additions to  
>  those files, if necessary - IE the rpm for TiLP (software for talking  
>  to a Ti graphing calculator) could easily add (if needed) a <ticalc>  
>  class to console.perms with the necessary device node entries, and then  
>  add the needed permission definitions for the device class.
>  
>  It also would allow gui administration for some configuration nodes for  
>  which there is not a gui. Kind of like how I can change some things in  
>  gnome using gconf-editor for which there isn't an existing control  
>  panel ready made for the task. 
You'll need a central database for this. You'll need to label this
central database, and you'll loose selinux capabilities to use different
security contexts per configuration file.

Or may be BerkletDB can assign different security contexts to different
database entries/tables ?
 
>  It also would make it easier for network administrators who set up
> box  
>  after box with needs to customize some of the default configurations to  
>  do so - after the install, a single documented script makes the needed  
>  entries/changes to the database and it's done. 
A plain python - or shell, or Perl, ... - script can do the job on flat
files also.
-- 
Iago Rubio         
- GPG Keyserv  * pgp.rediris.es id=0x909BD4DD

[Michael A. Peters]

On 10/14/2004 03:32:46 AM, Iago Rubio wrote:
>  On Thu, 2004-10-14 at 06:50, Michael A. Peters wrote:
>  > I'd like to suggest a change to the way Fedora Core does some
>  things.
>  > I'm not talking about "standard" linux config files like /etc/ 
>  passwd
>  or /
>  > etc/group or /etc/fstab (though the latter would be nice) - I'm
>  talking
>  > about Fedora (and RH I suppose since Fedora has become a RH test
>  bed)
>  > specific configuration files, such as /etc/security/console.perms
>  and /
>  > etc/sysconfig/network-scripts/ifcfg-[iface] type stuff.
>  >
>  > Rather than store them in flat files, which have different
>  > configuration parameters depending upon the file etc., use (maybe
>  > optionally) BerkeleyDB.
>  
>  Just to point that I don't like central configuration databases, and  
>  I
>  don't think it's the way to go.
>  
>  They're single failure points for the whole system. 
They don't need to be.
You can get redundance either through automated backups which are  
reverted to if corrupt, flat file backups, etc.

System would still boot if the database was corrupt - flat files are  
also single points of failure. Can you boot if /etc/fstab is crapped?

>  
>  In fact with current selinux improvements, you will not be able to
>  have
>  a central database for all the system, but some smaller databases  
>  with
>  different security contexts, to manage MAC on different roles.
>  
>  With this scenario, I don't see any improvements with this changes. 
The database can only be updated with utility xyz per selinux policy.
utility xyz can then have its own policy with respect to what/where/how  
it is called.

SELinux problem is not one that isn't easily solved.

>  
>  Flat files are pretty nice to manage/label.
>  
>  > bdb is already needed for the rpm database, so a Fedora system will
>  
>  > have bdb installed.
>  >
>  > Both Python and Perl (as well as many other languages) already have
>  
>  > good bdb interfaces, and both Python and Perl also have gtk+
>  bindings
>  > too.
>  
>  What's sure is any scripting language will support open/write/read
>  operations on flat files. 
Yes - but you then have to write a utility for each different type of  
config file, and then maintain those utilities. Otherwise - you end up  
with a system that is not friendly for the typical desktop user to  
administrate. With an embedded database, such admin tools are trivial -  
you don't have to learn how program xyz pases its config file etc.

>  
>  A plain python - or shell, or Perl, ... - script can do the job on
>  flat
>  files also. 
Look at the mess that the RH 5/6 configuration tool was - I forget what  
they called it (linuxconfig??), but it often did not work properly, and  
never worked properly with all things, because all the different flat  
files used different syntax in how the config files were used. Some  
were read by a C program, some were sourced by a shell script, etc.

That made it a nightmare to maintain the administration utility.
With a database, it is much easier. The value for the setting is  
requested from the database, or inserted into the database. It greatly  
simplifies how configuration information is stored and retrieved.

And an embedded database is easily backup up. I back up my rpm database  
daily (cron) - it hasn't failed on me in eons, but back when it did  
fail - it was trivial to go to a backup database.

An embedded database for system config files does not need to be  
terribly complex, and redundancy could easily be built into a wrapper  
app that handles the request for info. If there is a database integrity  
problem, it alerts the administrator and uses the last backup.

Clearly you would not want the database to contain data necessary to  
boot, such as your grub.conf info. But stuff like your yum  
repositories, network interface settings, console.perms data, etc. -  
that really would be better off (imho) in a single database.

I know people are put off by this because of what MS did with the  
registry. But that was MS. Gnome does exactly this - a database for  
gnome application settings, gconf - and it works extremely well, and is  
a hell of a lot nicer than config files.

[Jef Spaleta]

On Thu, 14 Oct 2004 22:12:51 -0700, Michael A. Peters <mpeters(a)mac.com&gt; wrote:
>  That made it a nightmare to maintain the administration utility.
>  With a database, it is much easier. The value for the setting is
>  requested from the database, or inserted into the database. It greatly
>  simplifies how configuration information is stored and retrieved. 
Specifically... Berkeley DB is an absolutely horrible idea if you need
to have multiple people changing multiple settings involving multiple
aspects of the system. Can Berkeley DB allow for multiple writes at
the same time? There is a price to be paid for centralization into a
single embedded database. I'm not extremely keen on the idea of
implementing a centralized embedded database for all the system
configs from networking to security policy on a system where there are
more than one administrator. And since I sort of get the feeling a lot
of RHEL deployments are large network, multiple adminstrator sorts of
things, I sort of doubt its in the best interest to steer development
into the berkeley db direction.   And I'm not particularly keen on the
idea of requiring moving to heavier multiple user database like
postgres to get around the problem of single client writing locks that
an embedded database would have. I honestly don't see any gain here at
all by over-centralization of disperate subsystem configfiles inside a
database. Optional database usage, sure. Requirement, no.

>  I know people are put off by this because of what MS did with the
>  registry. But that was MS. Gnome does exactly this - a database for
>  gnome application settings, gconf - and it works extremely well, and is
>  a hell of a lot nicer than config files. 
Cough... i think you misunderstand what Gconf is...and i think you'll
find a lively discussion if you care to look about the problems Gconf
has had. And frankly I wouldn't be all too shocked if Gconf gets
replaced with another attempt at this sort of thing anyways:
http://freedesktop.org/Standards/config-spec

GConf is NOT an embedded database. Its layer that sits between
applications and what could be a number of different configuration
storage backends. The commonly used backend isn't a 'database', its a
set of xml files. If GConf is the example of the better way to
approach the problem, then its not an example supports the argument
that you NEED a database backend like db4 at all.

You might be able to make a case for structured files like xml instead
of flat files, but the case specifically for a centralized berkeley db
falls short of the level of discussion thats going to be needed for
serious interest compared to previous long thread discussions
concerning 'registry' like behavior in the last 6 months. I personally
like the discussion pointing to the creation of libconfig that
implements an api for the sysconfig flat files instead of more
cumbersome solutions like a centralized db as a fedora/redhat
sysconfig specific stop gap measure before an appropriate api for
config files can be agreed on widely. But even then we get into
tradeoffs of how to build files that can be reasonably accessed with
and without advanced tools. Don't underestimate the power and
flexibility of being able to quickly open up a configuration file in
vi from a simplistic rescue environment and be able to get something
fixed, in your quest to build a file syntax and layout that is is to
build graphical tools on. There are tradeoffs invovled, and I'm not
even sure the benefits of even something simple like xml like
structured files strikes a balance to the ease of parsing the flat
files in vi when its needed.   I encourage you to look back to at
least July in the archives of this list and read up on similar
discussions.

-jef

[Nicolas Mailhot]

Le vendredi 15 octobre 2004 à 02:21 -0400, Jeff Spaleta a écrit :
>  Don't underestimate the power and
>  flexibility of being able to quickly open up a configuration file in
>  vi from a simplistic rescue environment and be able to get something
>  fixed, in your quest to build a file syntax and layout that is is to
>  build graphical tools on. There are tradeoffs invovled, and I'm not
>  even sure the benefits of even something simple like xml like
>  structured files strikes a balance to the ease of parsing the flat
>  files in vi when its needed.    
Just look at the fontconfig xml config files - perfectly accessible in
vi if you ask me. This is xml conf done right, not the junk that's
hidden behind gconf-editor. You've got about the same difference than
between a postfix and sendmail setup.

Ease of conf is not a tech issue only. If you don't care, the smartest
tech in the world won't fix your files for you (and the mess produced by
gconf does not mean conf files can't be done in xml)

Cheers,

-- 
Nicolas Mailhot

[Kyrre Ness Sjobak]

But frankly - you dont woryy about gnome settings being screwed in a
simplistic enviroment (read tomsrtb etc). You worry about getting it up
and running again.

fre, 15.10.2004 kl. 08.38 skrev Nicolas Mailhot:
>  Le vendredi 15 octobre 2004 à 02:21 -0400, Jeff Spaleta a écrit :
>  > Don't underestimate the power and
>  > flexibility of being able to quickly open up a configuration file in
>  > vi from a simplistic rescue environment and be able to get something
>  > fixed, in your quest to build a file syntax and layout that is is to
>  > build graphical tools on. There are tradeoffs invovled, and I'm not
>  > even sure the benefits of even something simple like xml like
>  > structured files strikes a balance to the ease of parsing the flat
>  > files in vi when its needed.   
>  
>  Just look at the fontconfig xml config files - perfectly accessible in
>  vi if you ask me. This is xml conf done right, not the junk that's
>  hidden behind gconf-editor. You've got about the same difference than
>  between a postfix and sendmail setup.
>  
>  Ease of conf is not a tech issue only. If you don't care, the smartest
>  tech in the world won't fix your files for you (and the mess produced by
>  gconf does not mean conf files can't be done in xml)
>  
>  Cheers,

[Remco Treffkorn]

On Thursday 14 October 2004 23:21, Jeff Spaleta wrote:
>  On Thu, 14 Oct 2004 22:12:51 -0700, Michael A. Peters
> <mpeters(a)mac.com&gt;  
wrote:
>  > That made it a nightmare to maintain the administration
> utility.
>  > With a database, it is much easier. The value for the setting is
>  > requested from the database, or inserted into the database. It greatly
>  > simplifies how configuration information is stored and retrieved.
>
>  Specifically... Berkeley DB is an absolutely horrible idea if you need
>  to have multiple people changing multiple settings involving multiple
>  aspects of the system. Can Berkeley DB allow for multiple writes at 
...

Look at the Elektra Project at http://elektra.sf.net. The author / maintainer 
has paid attention to fedback, and I think this solution is a viable 
compromise.

It fits in very well with Selinux. 

Solves the 'single point of failure' problem (to the extend it can be done).

Is 'vi' editable (and still maintains Selinux access control).

No additional dependencies (important for embedded).

Patches for some apps available.

Worth a read...

-- 
Remco Treffkorn (RT445)
HAM DC2XT
remco(a)rvt.com   [phone]

[Jef Spaleta]

On Fri, 15 Oct 2004 08:26:46 -0700, Remco Treffkorn <remco(a)rvt.com&gt; wrote:
>  Look at the Elektra Project at http://elektra.sf.net. The author /
> maintainer
>  has paid attention to fedback, and I think this solution is a viable
>  compromise. 
Its nice to see that project FINALLY changed its name to something
less prone to reactionary responses.  Long time readers of this list
have seen this project discussed before. I encourage you to look back
in this list to discussions starting in July of this year.
There was a long long thread and this project was discussed at
length... under its old name.

I'm not sure we need to rehash that discussion again, unless there are
some new developments that fedora developers need to be made aware of
from the linux registry project, i mean from the elektra project.


-jef

[Remco Treffkorn]

On Friday 15 October 2004 08:38, Jeff Spaleta wrote:
...
>  I'm not sure we need to rehash that discussion again, unless
> there are
>  some new developments that fedora developers need to be made aware of
>  from the linux registry project, i mean from the elektra project. 
The 'discussions' were severely hampered by the fact that people had 
prejudices because of a poor name choice.

Most participants had not even read the project info.

Nice sniper hit, though. Must be that special climate close to voting time ;-)

-- 
Remco Treffkorn (RT445)
HAM DC2XT
remco(a)rvt.com   [phone]

[Michael A. Peters]

On 10/15/2004 08:26:46 AM, Remco Treffkorn wrote:

>  
>  Look at the Elektra Project at http://elektra.sf.net. The author /
>  maintainer
>  has paid attention to fedback, and I think this solution is a viable
>  compromise. 
Thanks!
That's exactly the type of thing I think a desktop linux distro should  
have. I like what they are doing - will definitely watch it.

[Roberto Peon]

On Friday 15 October 2004 01:12 am, Michael A. Peters wrote:
<SNIP> (subject: redundancy)
>  They don't need to be.
>  You can get redundance either through automated backups which are
>  reverted to if corrupt, flat file backups, etc. 
Flat files can be backed up as well.

>  System would still boot if the database was corrupt - flat files are
>  also single points of failure. Can you boot if /etc/fstab is crapped? 

Configuration files are single point of failures for the processes that 
depend on their existance.

If all configuration is stored in one file, then it is more probable that
filesystem corruption will be able to take out that file (and thus all of your
configuration)


<SNIP> (subject: SE-Linux)
>  >
>  > What's sure is any scripting language will support open/write/read
>  > operations on flat files.
>
>  Yes - but you then have to write a utility for each different type of
>  config file, and then maintain those utilities. Otherwise - you end up
>  with a system that is not friendly for the typical desktop user to
>  administrate. With an embedded database, such admin tools are trivial -
>  you don't have to learn how program xyz pases its config file etc.
>
>  > A plain python - or shell, or Perl, ... - script can do the job on
>  > flat
>  > files also.
>
>  Look at the mess that the RH 5/6 configuration tool was - I forget what
>  they called it (linuxconfig??), but it often did not work properly, and
>  never worked properly with all things, because all the different flat
>  files used different syntax in how the config files were used. Some
>  were read by a C program, some were sourced by a shell script, etc.
>
>  That made it a nightmare to maintain the administration utility.
>  With a database, it is much easier. The value for the setting is
>  requested from the database, or inserted into the database. It greatly
>  simplifies how configuration information is stored and retrieved. 
A database does not necessarily make it easier, but it certainly makes it more
centralized.

A properly organized filesystem based approach, however should also be easy to
use.

>  And an embedded database is easily backup up. I back up my rpm
> database
>  daily (cron) - it hasn't failed on me in eons, but back when it did
>  fail - it was trivial to go to a backup database. 
A directory is easy to backup as well.

>  An embedded database for system config files does not need to be
>  terribly complex, and redundancy could easily be built into a wrapper
>  app that handles the request for info. If there is a database integrity
>  problem, it alerts the administrator and uses the last backup. 
Flat files don't need an additional wrapper to achieve this.

>  Clearly you would not want the database to contain data necessary to
>  boot, such as your grub.conf info. But stuff like your yum
>  repositories, network interface settings, console.perms data, etc. -
>  that really would be better off (imho) in a single database.
>
>  I know people are put off by this because of what MS did with the
>  registry. But that was MS. Gnome does exactly this - a database for
>  gnome application settings, gconf - and it works extremely well, and is
>  a hell of a lot nicer than config files. 
Properly done XML files are pretty darn legible, and by design are easily
extensible. 

Parsers for XML exist in many, if not most, scripting languages.
Since XML is text, it isn't all that hard to write a shell script to
modify a few key variables if necessary.

Databases have less tool coverage.

Also, one advantage that I might typically ascribe to databases, 
easy search (assuming, of course, that you have the right tool and you don't
have to know the database syntax, or write a program), is not likely to make
any difference, as 'grep' and 'find' are likely to suffice for any
configuration-setting searches.

Databases have one other advantage over flat files: diskspace usage.
A single file is going to be more efficent than a bunch of flat files.


Summary:

Database-based configuration approches have the following advantage:
1) more complete searching mechanisms (depending on the database)
2) better filesystem/filespace efficiency

Database-based configuration approaches suffer from:
1) increased chance of system failure
2) selinux+configuration database requires programmatic/script based
    workarounds 
3) less tool (scripting, bash, etc) support
4) VERY coarse-grained locking, so multiple simultaneous users suffer
5) Difficult to fix using vi (or some other text editor) in a 
   disaster-recovery situation


-Roberto JP