Fedora.next: I would like working configurations

Wed Jan 29 00:48:33 UTC 2014

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 01/28/2014 06:48 PM, Oron Peled wrote:
> 
> On Tuesday 28 January 2014 08:23:46 Stephen Gallagher wrote:
>> Forwarding this to server at lists.fedoraproject.org as well.
>> Responses inline.
> 
> As I'm not on the server list (yet), I replied to both lists. If
> "devel" should be off this thread, just remove it in your next
> reply and I'll understand the hint.
> 

It's fine to keep it on both lists (IMHO). I looped in server@ because
that's where most of the discussion is happening and I don't want
these good ideas getting lost.

>> I realize you address this below somewhat, but I want to call out
>> that what you're describing here is the *presentation layer* of 
>> configuration, not necessarily the configuration itself. In
>> general, the vast majority of applications keep their
>> configurations in 1) a set of text files, 2) a project-specific
>> database or 3) a combination of the two)
>> 
>> If projects provide a set of tools to manipulate this
>> configuration, that's separate from the configuration itself. As
>> we start talking about applying configuration to a system, we can
>> reuse existing tools where they make sense, but it's also
>> reasonable to say that we might choose to bypass them and use a
>> tool such as Augeaus or Puppet to apply the configuration we
>> want.
> 
> +1
> 
> My point was that creating a standard *model* does not prevent us 
> from choosing different mechanisms for the implementation. It
> simply hides those differences behind a standard interface[s].
> 
> Just like packagers may use different contents in their %build
> section but *every* package is still being built with the same
> rpmbuild command.
> 

I think we're saying the same thing here with different words (see below).

>> Yes, I agree. Simple package installation must not require 
>> interaction. We should define that as a fundamental tenet of our 
>> approach. We need it to always be possible to have unattended or 
>> scripted operation.
> 
> +1
> 
>> Yeah, obviously we'll need to maintain simple distinction about
>> how we maintain the data. Namespacing is one option. Another
>> might be to use an object-oriented API where we have objects
>> associated with each service (not necessarily package...) that
>> have configuration options specified as their attributes.
> 
> OO API is fine, as long as all classes present a unified
> *interface*.
> 
> About name-spacing/scoping: * If a package contain more than one
> service, the scoping may be more fine-grained/narrow than a package
> -- no problem.
> 
> * However, if the highest level scope is a package -- than each
> package may simply contain its configuration definitions. [I'm
> talking about the configuration keys+types, not their values].
> 
> * Advantages of such a design: - When a package is installed, its
> configuration options are exposed (simply because its configuration
> definition files are there)
> 
> - A package is still self-contained and fully describe its
> software (including the available configuration options).
> 

I would really like to avoid talking about "packages" when describing
Server Roles, because I feel it's very limiting. My personal goals
here are to describe *services* which may or may not map to a package.
For example, one package may offer multiple services. either unique
like the old inetd services or something like 389 DS or a Django app
which can have multiple instances of the same service running on
different ports.

Conversely, we can also have services like FreeIPA, which is comprised
of multiple packages that really should be configured together (though
that may boil down "under the hood" to managing the lower-level
services as well).

>> Extending the above, one of my thoughts on another thread on the 
>> Fedora Server mailing list is to use D-BUS for the configuration
>> API. Among other things, we have the D-BUS object types to work
>> with. This would mean that we can have service-specific objects
>> with appropriately-typed options.
> 
> Sounds like a great idea: * As you said we automatically gain all
> D-Bus goodies (including a fully developed type-system).
> 
> * Also, I think we can map it to some data-driven syntax.
> 
> Example: * For simplicity, let's assume each configuration item is
> a D-Bus property.
> 
> * Than a package may install a "configuration definition file": 
> /usr/share/system-config-data/<package>/foo.conf-defs: <D-Bus
> interface> <object> <property> <ro/rw/wo>
> 
> * So a generic configuration engine that read all these files, 
> should have no trouble accessing these properties, presenting them
> to the user (according to their types) and then inject the user
> choice over D-Bus.
> 
> * The packager would have to supply implementation for these
> objects, but we may ease this for most packages by supplying some
> common classes which can be easily inherited from. [obviously, this
> part is the toughest to get right]
> 

Agreed, but hopefully we can at least work on offering a few sensible
reference implementations first. A common class seems ideal, but
probably too much to bank on while we prove out the concept.

>>> * A package installation register its configuration options in
>>> a system-wide "database".
>> 
>> I'm not sure this makes sense. This would require us to maintain
>> data in two places, because zero existing packages would
>> *retrieve* their data from this location. We'd have a duplication
>> problem and we would also be responsible for bi-directional sync
>> (if someone hand-edited a configuration file, we'd need to read
>> it back into the database).
> 
> I suspect my choice of words ("database") wasn't so good.
> 
> I didn't mean storing the configuration *values*, only the
> available configuration options.
> 
> Perhaps calling it "catalog" would have made more sense.
> Basically, in "debconf" it contains a list of all possible
> "configuration entries". Each with: * A fully qualified name
> (including the package it belongs to). * Type info. * Available
> translations.
> 

Continuing with the D-BUS idea, I think we can probably get this
mostly for free in terms of D-BUS object introspection. As we drop
service plugins into the system, they'll come with introspection data
that should effectively provide this catalog. That's why I suggested
that the translations would be better at the API layer.

>> I think it makes more sense to build an API around something
>> like Augeaus and rely on being able to read and write the
>> configuration that the packages already understand.
> 
> Sure. If we can wrap Augeaus in some base class, than the
> implementation of most packages would be: * Inherit from this base
> class (via, to be designed, declarative syntax). * Pass our package
> specific data to this class constructor. Where the package-specific
> data would be a file containing relevant Augeaus code.
> 

Certainly worth putting in some research time to see if this will be
feasible, I'd say.

>>> * This also contains end-user visible text for each option 
>>> including i18n.
>> I think it would be better to handle this at the API layer,
>> rather than in a database.
> 
> There's a trade-off: * API layer -- more flexible. * Just i18n
> strings -- easier, may be given to translator teams, etc.
> 
> If you look at i18n -- the FOSS community already made this choice
> -- almost everybody model it as data strings (.desktop files,
> pot/po files, etc.)
> 

See above for comments about translations in the API layer.

>> Again, if we build a sensible API, it should be possible to
>> build whatever presentation layer we want to atop it. Of note,
>> we're planning to work with the Cockpit Project for our
>> reference implementation.
> 
> OK,
> 
>>> * All front ends need only understand the generic types of the 
>>> options. (I.e: they provide generic configuration UI widgets
>>> which aren't modified when new packages/options are added)
>>> 
>> 
>> I don't agree with this. Providing just a text entry box and a 
>> description isn't an effective UI technique, particularly if you
>> want to be able to produce guided or wizard-based operation. (For
>> example, if you have branching decisions during configuration).
> 
> * The basic widget depends on the richness of your type system. For
> example, debconf has a "multi-select" type and its typical 
> presentation is a drop-down list or something similar.
> 
> * But you are correct that even with rich widget collection, the 
> basic model is: "one-widget-at-a-time" which is somewhat limiting.
> 
> * Maybe someone can find some generic enhancement to the
> "one-widget" model?
> 
> However, a fully flexible UI model may present a lot of obstacles: 
> * Packagers would need to write/maintain non-trivial UI models. I
> suspect this won't be maintainable (unless most of it would rapidly
> flow to upstream... hmmm... chicken and egg problem, related to
> wide adoption).
> 
> * It would limit the available presentation formats. One of the
> huge benefits in "debconf" is that it can be used *everywhere*: -
> GUI - TUI (try to implement the Anaconda "spoke" interaction on a
> console...) - Regular text -- yes you can use it over plain old
> serial port (these days ignoring embedded systems doesn't sound
> like a good idea). - Preseed -- injecting (seeding) data
> beforehand. If you inject *some* but not all data -- How should it
> affect the interaction with complex UI ?
> 
> Maybe the right approach for Wizards and other complex UI would
> be: * Create *optional* complex UI separately (maybe include it in
> its own (foobar-server) package).
> 

I didn't describe this very well, because I agree with you here. What
I really meant was that a strict name/value system is probably not
sufficient, because options can be interdependent (and if we're
building this API, I'd like for us to do whatever we can to reduce the
likelihood of ending up with unusable configuration).

My view was that if we viewed configuration as objects with the option
for related child objects, we probably have enough information to
satisfy both cases. We can maintain the complex data through a limited
referential organization, but the objects themselves are very
straightforward. On the other hand, we also provide a hopefully
sufficient model that if someone wants to write the more complex GUI
atop it, they have a few additional clues from the API objects.

For a contrived example, suppose I want to stand up three separate
LDAP servers on a single machine using 389 DS. To me, it would make
sense to have a master object that identified the set of servers on
the machine and then each individual server would have its own
configuration object. While this is not impossible to describe in the
strict package namespace example, I think we can provide a more useful
example doing things this way.

> * This code would collect arbitrary info via arbitrary UI and
> would inject it (preseed) into the regular configuration items.
> 
> * Obviously, this "high-level" configuration UI would have tight
> coupling to the low-level configuration details (item names and
> types).
> 
> Other ideas?
> 
>>> * The debconf "API" allows fetching any option from the
>>> database and the values of these options are used to configure
>>> the actual software.
>>> 
>> 
>> See above for the sync problem with this.
> 
> As I explained, there's no problem since *values* are kept only
> once (inside the configured program native
> files/databases/whatever).
> 
> The only "database" I was talking about was describing *which*
> options exist, what are their types, etc.

Right, with that correction I understand what you mean. I think the
D-BUS introspection might well be the cheapest way to get this.