RFC: libosinfo: Library for virt OS/distro metadata 3

[Cole Robinson]

Hi all,

I've done the initial work for a new library, libosinfo (better name
recommendations appreciated). This library will provide OS meta data for use
in virt applications, replacing the dictionary we currently keep in virtinst.

This is based off of a post by Dan Berrange:

https://www.redhat.com/archives/et-mgmt-tools/2009-March/msg00028.html

The code can be fetched with:

git clone http://fedorapeople.org/~crobinso/osinfo/.git

Check out the TODO list for a simple roadmap.

http://fedorapeople.org/~crobinso/osinfo/TODO

The public API looks like:

/**
 * Values stored in the OS dictionary
 */
enum _os_value_type {
    OS_VALUE_NAME = 1,          /** Human readable family/distro... name */
    OS_VALUE_MEDIA_INSTALL_URL, /** URL to an install tree */
};
typedef enum _os_value_type os_value_t;

int     os_init();
void    os_close();

int     os_find_families        (char ***list);
int     os_find_distros         (const char *parent_id, char ***list);
int     os_find_releases        (const char *parent_id, char ***list);
int     os_find_updates         (const char *parent_id, char ***list);

int     os_lookup_value         (os_value_t value_type,
                                 const char *os_id,
                                 char **value);

The unique identifier for each distro is its 'id', which is a simple human
readable string, similar to values we use for virt-install --os-variant today.

The user will ask the API for available families/distros/releases/updates,
which will return a list of ids. We then pass an id to os_lookup_value to
actually retrieve data. The family/distro/... separation will likely be
removed pretty soon, in favor of an arbitrary hierarchy, where every OS
can have child OSes: no doubt hardcoding the family/distro/... split would
come back to bite us in the ass.

As an example, the following code will list the 'Name' of every id at
the'family' level, where name is full name of the OS (id = rhel5.3, name = Red
Hat Enterprise Linux 5.3):

char *value = NULL;
    char **namelist = NULL;
    int i, num_fams;

if (os_init() < 0)
        goto error;

if ((num_fams = os_list_families(&namelist)) < 0)
        goto error;

for (i = 0; i < num_fams; ++i) {
        if (os_lookup_value(OS_VALUE_NAME,
                            namelist[i],
                            &value) < 0)
            goto error;

printf("%s\n", value);
        free(value);
    }

error:
   free(namelist);
   free(value);
   os_close();

There is a simple tool called 'osinfo-tool' which allows listing an ASCII repr
of the OS hierarchy, and listing all values for an individual id (which is
pretty sparse at the moment since most of these values haven't been filled
in yet).

So, things that I'm interested in feedback on:

- How do we expect apps to list OS choices? Currently, virt-manager lists
  type (linux, windows, unix, etc.) and associated distros (Fedora 8, RHEL4,
  Debian Lenny, etc.). The linux/windows/unix info isn't represented in the
  xml (should it be?) so the best way seems to be:

Distro
    |
    --> Release
          |
          --> Update

Ex.

RHEL
    |
    -> RHEL5
        |
        -> 5.0
           5.1
           5.2

If we do away with the family/distro/... distinction, the user won't have
  much choice in the matter, but the 'family' concept (e.g. value of
  'Red Hat') isn't very useful to expose to a user.

- How should we handle derivatives like Scientific Linux + CentOS: should we
  expect users to understand they are based on RHEL, or give them explicit
  IDs?

- Querying for device values (supported buses, models, etc.). Dan's original
  proposal talks about this; to recommend a default with the best chance of
  actually working, we need to know:

- OS being installed
  - Virt type ('hvm' vs. 'xen')
  - Guest Architecture (i386, x86_64, ...)
  - Hypervisor (kvm, qemu, xen, vbox, ...)
  - Hypervisor version
  - Libvirt version

We would need to find the intersection of what the OS, the hypervisor,
  and libvirt support, and return what we decide is the best choice.

How to expose this in the API? We could simply have one long function

os_lookup_device_value(char *os_id, char *virt_type, char *arch, ...)

It works, but its pretty tedious, and I'm afraid that we would need
  even more info to make a correct choice in the future, and the above
  isn't flexible. We may also need some of the above info for other values
  (ACPI/APIC settings, returning a proper install url may depend on arch).
  Any suggestions?

- os_init and os_close: Any better ideas for this? os_init just parses the
  xml document, os_close frees it. We could run os_init with the first API
  call, but I think that makes it less clear that the user would then
  need to call os_close().

Any feedback appreciated.

Thanks,
Cole

[et-mgmt-tools] RFC: libosinfo: Library for virt OS/distro metadata 3

[John Levon]

On Sun, Jun 14, 2009 at 06:50:02PM -0400, Cole Robinson wrote:
> git clone http://fedorapeople.org/~crobinso/osinfo/.git

I'm not convinced by the "arbitrary hierarchy" thing: I just don't see
how that could possibly be useful. Surely it's either an entirely flat
namespace, or a shallow structured one like you have.

The flat namespace would be a set of keys and multiple values for each
key. One value would be "os type" (linux, windows, etc.). You'd most
likely have a "generic" entry still for fallback.

An alternative is to represent the hierarchy in the UNIX way, via the
filesystem:

/var/lib/osinfo/
/var/lib/osinfo/linux/info.xml
/var/lib/osinfo/linux/fedora/info.xml
/var/lib/osinfo/linux/fedora/8/info.xml

The fallback is pretty obvious then. Maybe it's over the top. The idea
of a single delivered XML file that users edit does trouble me though.
Maybe we at least have two files, one for customisations.
> int     os_init();
> void    os_close();

Always, always, always, pass back an opaque identifier in an API - you
never know when you'll need to track per-thread state. It's generally a
good idea to pass in a version define too.

The XML parsing itself should happen lazily. We might want to let the
user specify a file, for example.
> int     os_find_families        (char ***list);
> int     os_find_distros         (const char *parent_id, char ***list);
> int     os_find_releases        (const char *parent_id, char ***list);
> int     os_find_updates         (const char *parent_id, char ***list);

Regardless of the hierarchy question I hate the idea of exposing it in
the API like this.

There's really two (hopefully three eventually) things this library
needs to do: provide a list of everything it knows about so the user can
select it in a GUI or whatever, and provide configuration
recommendations given a particular set of values.

I don't think the library can make any assumptions about how the former
might look. It just needs to return some thing like:

struct osinfo {
    	const char *id;
    	nvpair_list_t values;
    };

Either allocated as an array (probably easiest) or in a list. It's then
up to the client to decide what hierarchy to actually use, and it's all
dependent on what values they pick out of the nv list.

(The third thing is to identify OS types based upon installation media
when possible.)
> If we do away with the family/distro/... distinction, the user won't have
>   much choice in the matter, but the 'family' concept (e.g. value of
>   'Red Hat') isn't very useful to expose to a user.

If you mean API user, think "icon".
> How should we handle derivatives like Scientific Linux + CentOS: should we
> expect users to understand they are based on RHEL, or give them explicit
> IDs?

Explicit IDs. You'd be surprised.
> We would need to find the intersection of what the OS, the hypervisor,
>   and libvirt support, and return what we decide is the best choice.
>
> How to expose this in the API? We could simply have one long function
>
> os_lookup_device_value(char *os_id, char *virt_type, char *arch, ...)

The API user should pass in an nvlist, where a set of the names are
defined and known about. The response needs to indicate whether it's a
preferred setting ("would like virtio") or a required one.

regards
john

[Richard W.M. Jones]

On Sun, Jun 14, 2009 at 06:50:02PM -0400, Cole Robinson wrote:
> The public API looks like:
>
> /**
> Values stored in the OS dictionary
>
> */
> enum _os_value_type {
>     OS_VALUE_NAME = 1,          /** Human readable family/distro... name */
>     OS_VALUE_MEDIA_INSTALL_URL, /** URL to an install tree */
> };
> typedef enum _os_value_type os_value_t;
>
> int     os_init();
> void    os_close();
>
> int     os_find_families        (char ***list);
> int     os_find_distros         (const char *parent_id, char ***list);
> int     os_find_releases        (const char *parent_id, char ***list);
> int     os_find_updates         (const char *parent_id, char ***list);
>
> int     os_lookup_value         (os_value_t value_type,
>                                  const char *os_id,
>                                  char **value);

There's a (little) overlap and a possible user in virt-inspector.

In virt-inspector we do things the other way around - we look inside
the guest for files like /etc/redhat-release and /etc/debian_version,
and parse those to determine the OS distro and release (also we parse
the registry to do the same for Windows).  The code for Linux is here:

http://git.et.redhat.com/?p=libguestfs.git;a=blob;f=inspector/virt-inspector...

It would be nice for virt-inspector to output ID strings which are
compatible with osinfo.

'Course we'll need Perl bindings.  Did you see this?

http://rwmj.wordpress.com/2009/04/20/generating-code/

Rich.

[Cole Robinson]

On 06/15/2009 08:15 AM, Richard W.M. Jones wrote:
> On Sun, Jun 14, 2009 at 06:50:02PM -0400, Cole Robinson wrote:
> > The public API looks like:
> >
> > /**
> > Values stored in the OS dictionary
> >
> > */
> > enum _os_value_type {
> >     OS_VALUE_NAME = 1,          /** Human readable family/distro... name */
> >     OS_VALUE_MEDIA_INSTALL_URL, /** URL to an install tree */
> > };
> > typedef enum _os_value_type os_value_t;
> >
> > int     os_init();
> > void    os_close();
> >
> > int     os_find_families        (char ***list);
> > int     os_find_distros         (const char *parent_id, char ***list);
> > int     os_find_releases        (const char *parent_id, char ***list);
> > int     os_find_updates         (const char *parent_id, char ***list);
> >
> > int     os_lookup_value         (os_value_t value_type,
> >                                  const char *os_id,
> >                                  char **value);
>
> There's a (little) overlap and a possible user in virt-inspector.
>
> In virt-inspector we do things the other way around - we look inside
> the guest for files like /etc/redhat-release and /etc/debian_version,
> and parse those to determine the OS distro and release (also we parse
> the registry to do the same for Windows).  The code for Linux is here:
>
> http://git.et.redhat.com/?p=libguestfs.git;a=blob;f=inspector/virt-inspector...
>
> It would be nice for virt-inspector to output ID strings which are
> compatible with osinfo.

Agreed, I was thinking something along the same lines.
> 'Course we'll need Perl bindings.  Did you see this?
>
> http://rwmj.wordpress.com/2009/04/20/generating-code/

Thanks, I'll take a look.

- Cole

[Daniel P. Berrange]

On Sun, Jun 14, 2009 at 06:50:02PM -0400, Cole Robinson wrote:
> The public API looks like:
>
> /**
> Values stored in the OS dictionary
>
> */
> enum _os_value_type {
>     OS_VALUE_NAME = 1,          /** Human readable family/distro... name */
>     OS_VALUE_MEDIA_INSTALL_URL, /** URL to an install tree */
> };
> typedef enum _os_value_type os_value_t;
>
> int     os_init();
> void    os_close();
>
> int     os_find_families        (char ***list);
> int     os_find_distros         (const char *parent_id, char ***list);
> int     os_find_releases        (const char *parent_id, char ***list);
> int     os_find_updates         (const char *parent_id, char ***list);
>
> int     os_lookup_value         (os_value_t value_type,
>                                  const char *os_id,
>                                  char **value);
>
> The unique identifier for each distro is its 'id', which is a simple human
> readable string, similar to values we use for virt-install --os-variant today.

As John suggested, I think we'd be safer having opaque structs for the
conceptual objects. One for the library itself, and another for an OS
distro.

Perhaps have an

'os_info_t'    as a handle for a library itself returned by os_init
   'os_distro_t'  as a handle for a single OS distro instance

os_info_t os_info_new()
    os_info_init(os_info_t *info, char *uri);   /* loads the XML data */

For OS distros I think we need APIs to:

- List all OS distros
 - Find OS distros, matching a specific set of properties
 - Read a property from an OS distro
 - Read all properties from an OS distro
 - List unique values for a property across all distros
> The user will ask the API for available families/distros/releases/updates,
> which will return a list of ids. We then pass an id to os_lookup_value to
> actually retrieve data. The family/distro/... separation will likely be
> removed pretty soon, in favor of an arbitrary hierarchy, where every OS
> can have child OSes: no doubt hardcoding the family/distro/... split would
> come back to bite us in the ass.

I agree, the fixed hierarchy I describe really doesn't seem very nice
looking back on it. The names I gave them are rather contrived and only
really map nicely onto RHEL/Fedora release process. I think we're better
off being more flexible and allowing for arbitrary relationships in the
data files and API. I don't think we neccessarily want to force a single
rooted tree structure here. The key important factor with the hierarchy
is the concept of sharing metadata.

I think we should take a hint from the way RDF works and define the API
and XML format as a flat list, but allow relationships to be defined,
and also allow tagging.

- Flat list of OS distros with their full name,  as defined by their
   vendor/distributor

"Red Hat Enterprise Linux 4.7"
     "Red Hat Enterprise Linux 5.0"
     "Fedora 10"
     "Fedora 10"
     "Debian Sarge"

- A 'derived' property. Allows derived distros to declare
    they should inherit metdata (eg Scientific Linux derives from 
    RHEL)

- A 'clone' property. Allows functionally identical rebuilds
   to declare they use exactly same metadata. (eg CentOS / RHEL)

- A 'upgrades' property. Allows to indicate 'Fedora 11' is the
   release following on from 'Fedora 10'.

- A 'publisher' property to give name of entity producing the
   distro eg  'Fedora Project', 'Red Hat', 'Microsoft'

- A 'kernel type' and 'kernel version' property, eg 'linux'
   and '2.6.26'.

Application UI might simulate a hierarchy by using the 'publisher'
property at first level, and then filtering the flat list of OS
distros at the 2nd level according to selected publisher. This
satisfies the key 'UI' reason for the hierarchy. The 'derived'
and 'clone'  allow for inheritance of metadata.
> So, things that I'm interested in feedback on:
>
> How do we expect apps to list OS choices? Currently, virt-manager lists
> type (linux, windows, unix, etc.) and associated distros (Fedora 8, RHEL4,
> Debian Lenny, etc.). The linux/windows/unix info isn't represented in the
> xml (should it be?) so the best way seems to be:
>
> Distro
>   |
>   --> Release
>         |
>         --> Update
>
> Ex.
>
> RHEL
>   |
>   -> RHEL5
>       |
>       -> 5.0
>          5.1
>          5.2
>
> If we do away with the family/distro/... distinction, the user won't have
> much choice in the matter, but the 'family' concept (e.g. value of
> 'Red Hat') isn't very useful to expose to a user.

We should try to avoid forcing one representation onto apps. I think the
flat OS list + sets of properties will allow apps to build a variety of
UI models for this, either search based, tree based or filter based.
> How should we handle derivatives like Scientific Linux + CentOS: should we
> expect users to understand they are based on RHEL, or give them explicit
> IDs?

They need explicit IDs, since they have unique download URLs that have
to be stored. The 'clone' and 'derived' properties will allow us to avoid
duplicating other metadata, and also allow apps to show/hide clones as
needed.
> Querying for device values (supported buses, models, etc.). Dan's original
> proposal talks about this; to recommend a default with the best chance of
> actually working, we need to know:
> OS being installed
> Virt type ('hvm' vs. 'xen')
> Guest Architecture (i386, x86_64, ...)
> Hypervisor (kvm, qemu, xen, vbox, ...)
> Hypervisor version
> Libvirt version
>
> We would need to find the intersection of what the OS, the hypervisor,
> and libvirt support, and return what we decide is the best choice.
>
> How to expose this in the API? We could simply have one long function
>
> os_lookup_device_value(char *os_id, char *virt_type, char *arch, ...)
>
> It works, but its pretty tedious, and I'm afraid that we would need
> even more info to make a correct choice in the future, and the above
> isn't flexible. We may also need some of the above info for other values
> (ACPI/APIC settings, returning a proper install url may depend on arch).
> Any suggestions?

The more I think about this, the more I think we should avoid any specific
named attributes in the API. Supported devices are just other types of
property we can associated with a distro, in addition to ones I already
listed earlier. This could be useful in the UI too, for example, if you
know the hypervisor requires support for 'Xen paravirt disk', then when
browsing OS, you can filter on this property just as you would with the
others.
> os_init and os_close: Any better ideas for this? os_init just parses the
> xml document, os_close frees it. We could run os_init with the first API
> call, but I think that makes it less clear that the user would then
> need to call os_close().

I think its good to keep the initializer explicit, and if you add an
opaque type representing a handle to the library, this will force apps
to caller it and track it.

Regards,
Daniel

[Daniel P. Berrange]

On Mon, Jun 15, 2009 at 08:22:16AM -0400, Cole Robinson wrote:
> On 06/15/2009 08:15 AM, Richard W.M. Jones wrote:
> > On Sun, Jun 14, 2009 at 06:50:02PM -0400, Cole Robinson wrote:
> > > The public API looks like:
> > >
> > > /**
> > > Values stored in the OS dictionary
> > >
> > > */
> > > enum _os_value_type {
> > >     OS_VALUE_NAME = 1,          /** Human readable family/distro... name */
> > >     OS_VALUE_MEDIA_INSTALL_URL, /** URL to an install tree */
> > > };
> > > typedef enum _os_value_type os_value_t;
> > >
> > > int     os_init();
> > > void    os_close();
> > >
> > > int     os_find_families        (char ***list);
> > > int     os_find_distros         (const char *parent_id, char ***list);
> > > int     os_find_releases        (const char *parent_id, char ***list);
> > > int     os_find_updates         (const char *parent_id, char ***list);
> > >
> > > int     os_lookup_value         (os_value_t value_type,
> > >                                  const char *os_id,
> > >                                  char **value);
> >
> > There's a (little) overlap and a possible user in virt-inspector.
> >
> > In virt-inspector we do things the other way around - we look inside
> > the guest for files like /etc/redhat-release and /etc/debian_version,
> > and parse those to determine the OS distro and release (also we parse
> > the registry to do the same for Windows).  The code for Linux is here:
> >
> > http://git.et.redhat.com/?p=libguestfs.git;a=blob;f=inspector/virt-inspector...
> >
> > It would be nice for virt-inspector to output ID strings which are
> > compatible with osinfo.
>
> Agreed, I was thinking something along the same lines.

If we keep to a flat list, then trivial answer here is to just use the
official distro name given by the vendor/distributor.

Daniel

[et-mgmt-tools] RFC: libosinfo: Library for virt OS/distro metadata 3

[Daniel P. Berrange]

On Mon, Jun 15, 2009 at 07:16:56AM -0400, John Levon wrote:
> On Sun, Jun 14, 2009 at 06:50:02PM -0400, Cole Robinson wrote:
> > git clone http://fedorapeople.org/~crobinso/osinfo/.git
>
> I'm not convinced by the "arbitrary hierarchy" thing: I just don't see
> how that could possibly be useful. Surely it's either an entirely flat
> namespace, or a shallow structured one like you have.
>
> The flat namespace would be a set of keys and multiple values for each
> key. One value would be "os type" (linux, windows, etc.). You'd most
> likely have a "generic" entry still for fallback.

I think this is the way to go. Flat list + properties, and allow apps
to build hiearchies on the fly as needed, based off properties.
> An alternative is to represent the hierarchy in the UNIX way, via the
> filesystem:
>
> /var/lib/osinfo/
> /var/lib/osinfo/linux/info.xml
> /var/lib/osinfo/linux/fedora/info.xml
> /var/lib/osinfo/linux/fedora/8/info.xml
>
> The fallback is pretty obvious then. Maybe it's over the top. The idea
> of a single delivered XML file that users edit does trouble me though.
> Maybe we at least have two files, one for customisations.

A single XML file would be pretty horrible for package upgrades.
With a flat list we can have 1 XML file per distro. If we allow
multiple search paths for XML files, we can have the stadnard
shipped ones in /usr/share/osinfo, and customized ones in /etc/osinfo
the latter overriding (or inheriting from) the former.
> > How should we handle derivatives like Scientific Linux + CentOS: should we
> > expect users to understand they are based on RHEL, or give them explicit
> > IDs?
>
> Explicit IDs. You'd be surprised.

Download/install URLs already require that we do separate IDs :-)

Daniel

[Cole Robinson]

Daniel P. Berrange wrote:
> On Sun, Jun 14, 2009 at 06:50:02PM -0400, Cole Robinson wrote:
> > The public API looks like:
> >
> > /**
> > Values stored in the OS dictionary
> >
> > */
> > enum _os_value_type {
> >     OS_VALUE_NAME = 1,          /** Human readable family/distro... name */
> >     OS_VALUE_MEDIA_INSTALL_URL, /** URL to an install tree */
> > };
> > typedef enum _os_value_type os_value_t;
> >
> > int     os_init();
> > void    os_close();
> >
> > int     os_find_families        (char ***list);
> > int     os_find_distros         (const char *parent_id, char ***list);
> > int     os_find_releases        (const char *parent_id, char ***list);
> > int     os_find_updates         (const char *parent_id, char ***list);
> >
> > int     os_lookup_value         (os_value_t value_type,
> >                                  const char *os_id,
> >                                  char **value);
> >
> > The unique identifier for each distro is its 'id', which is a simple human
> > readable string, similar to values we use for virt-install --os-variant today.
>
> As John suggested, I think we'd be safer having opaque structs for the
> conceptual objects. One for the library itself, and another for an OS
> distro.
>
> Perhaps have an
>
> 'os_info_t'    as a handle for a library itself returned by os_init
>    'os_distro_t'  as a handle for a single OS distro instance
> os_info_t os_info_new()
> os_info_init(os_info_t *info, char *uri);   /* loads the XML data */

Sounds good, though why have a separate os_info_new()? And I'd rather
have a separate API for initializing from a file, since that should be
uncommon enough that we don't need to force uri=NULL on most users.

int os_info_init(os_info_t **info)
int os_info_init_from_uri(os_info_t **info, char *uri)

Though it's a minor distinction for now.
> For OS distros I think we need APIs to:
> List all OS distros
> Find OS distros, matching a specific set of properties

What uses do you have in mind for this? Being able to say e.g. 'all
distros that support xen PV for IA64'?

How do you think an API call would look?
> Read a property from an OS distro
> Read all properties from an OS distro

Why would an API user want to do this? I don't see why we would need to
enable this specifically, rather than make the user do this iteratively.
> List unique values for a property across all distros

Not sure I fully understand this. Why would a user want this?
> > The user will ask the API for available families/distros/releases/updates,
> > which will return a list of ids. We then pass an id to os_lookup_value to
> > actually retrieve data. The family/distro/... separation will likely be
> > removed pretty soon, in favor of an arbitrary hierarchy, where every OS
> > can have child OSes: no doubt hardcoding the family/distro/... split would
> > come back to bite us in the ass.
>
> I agree, the fixed hierarchy I describe really doesn't seem very nice
> looking back on it. The names I gave them are rather contrived and only
> really map nicely onto RHEL/Fedora release process. I think we're better
> off being more flexible and allowing for arbitrary relationships in the
> data files and API. I don't think we neccessarily want to force a single
> rooted tree structure here. The key important factor with the hierarchy
> is the concept of sharing metadata.
>
> I think we should take a hint from the way RDF works and define the API
> and XML format as a flat list, but allow relationships to be defined,
> and also allow tagging.
>
> Flat list of OS distros with their full name,  as defined by their
> vendor/distributor
>
> "Red Hat Enterprise Linux 4.7"
>   "Red Hat Enterprise Linux 5.0"
>   "Fedora 10"
>   "Fedora 10"
>   "Debian Sarge"
>
> A 'derived' property. Allows derived distros to declare
>  they should inherit metdata (eg Scientific Linux derives from 
>  RHEL)
>
> A 'clone' property. Allows functionally identical rebuilds
> to declare they use exactly same metadata. (eg CentOS / RHEL)
>
> A 'upgrades' property. Allows to indicate 'Fedora 11' is the
> release following on from 'Fedora 10'.
>
> A 'publisher' property to give name of entity producing the
> distro eg  'Fedora Project', 'Red Hat', 'Microsoft'
>
> A 'kernel type' and 'kernel version' property, eg 'linux'
> and '2.6.26'.
>
> Application UI might simulate a hierarchy by using the 'publisher'
> property at first level, and then filtering the flat list of OS
> distros at the 2nd level according to selected publisher. This
> satisfies the key 'UI' reason for the hierarchy. The 'derived'
> and 'clone'  allow for inheritance of metadata.

I like this idea: certainly will give more flexibility.
> > So, things that I'm interested in feedback on:
> >
> > How do we expect apps to list OS choices? Currently, virt-manager lists
> > type (linux, windows, unix, etc.) and associated distros (Fedora 8, RHEL4,
> > Debian Lenny, etc.). The linux/windows/unix info isn't represented in the
> > xml (should it be?) so the best way seems to be:
> >
> > Distro
> >   |
> >   --> Release
> >         |
> >         --> Update
> >
> > Ex.
> >
> > RHEL
> >   |
> >   -> RHEL5
> >       |
> >       -> 5.0
> >          5.1
> >          5.2
> >
> > If we do away with the family/distro/... distinction, the user won't have
> > much choice in the matter, but the 'family' concept (e.g. value of
> > 'Red Hat') isn't very useful to expose to a user.
>
> We should try to avoid forcing one representation onto apps. I think the
> flat OS list + sets of properties will allow apps to build a variety of
> UI models for this, either search based, tree based or filter based.
> > How should we handle derivatives like Scientific Linux + CentOS: should we
> > expect users to understand they are based on RHEL, or give them explicit
> > IDs?
>
> They need explicit IDs, since they have unique download URLs that have
> to be stored. The 'clone' and 'derived' properties will allow us to avoid
> duplicating other metadata, and also allow apps to show/hide clones as
> needed.
> > Querying for device values (supported buses, models, etc.). Dan's original
> > proposal talks about this; to recommend a default with the best chance of
> > actually working, we need to know:
> > OS being installed
> > Virt type ('hvm' vs. 'xen')
> > Guest Architecture (i386, x86_64, ...)
> > Hypervisor (kvm, qemu, xen, vbox, ...)
> > Hypervisor version
> > Libvirt version
> >
> > We would need to find the intersection of what the OS, the hypervisor,
> > and libvirt support, and return what we decide is the best choice.
> >
> > How to expose this in the API? We could simply have one long function
> >
> > os_lookup_device_value(char *os_id, char *virt_type, char *arch, ...)
> >
> > It works, but its pretty tedious, and I'm afraid that we would need
> > even more info to make a correct choice in the future, and the above
> > isn't flexible. We may also need some of the above info for other values
> > (ACPI/APIC settings, returning a proper install url may depend on arch).
> > Any suggestions?
>
> The more I think about this, the more I think we should avoid any specific
> named attributes in the API. Supported devices are just other types of
> property we can associated with a distro, in addition to ones I already
> listed earlier. This could be useful in the UI too, for example, if you
> know the hypervisor requires support for 'Xen paravirt disk', then when
> browsing OS, you can filter on this property just as you would with the
> others.

If the app already knows their hypervisor requires 'Xen paravirt disk',
the above is fine, and we should facilitate filtering like that. However
I would like osinfo to save the user from having to know those details:
they should just be able to say 'I'm using xenpv on i386 for xen 1.2.3
and libvirt 4.5.6 with distro fedora10' and osinfo can return the
required info. So I still don't see how to solve the above problem.

Let's say we are installing winxp on qemu via libvirt: we want to know
the recommended sound device model:

winxp prefers ac97, es1370
qemu supports es1370, if >= 0.10.0, supports ac97
libvirt supports es1370, if >= 0.6.0 supports ac97

How do we solve this via the API?
> > os_init and os_close: Any better ideas for this? os_init just parses the
> > xml document, os_close frees it. We could run os_init with the first API
> > call, but I think that makes it less clear that the user would then
> > need to call os_close().
>
> I think its good to keep the initializer explicit, and if you add an
> opaque type representing a handle to the library, this will force apps
> to caller it and track it.

Sounds good.

Thanks a bunch!

- Cole

[Daniel P. Berrange]

On Mon, Jun 15, 2009 at 11:09:56AM -0400, Cole Robinson wrote:
> Daniel P. Berrange wrote:
> > On Sun, Jun 14, 2009 at 06:50:02PM -0400, Cole Robinson wrote:
> > For OS distros I think we need APIs to:
> > List all OS distros
> > Find OS distros, matching a specific set of properties
>
> What uses do you have in mind for this? Being able to say e.g. 'all
> distros that support xen PV for IA64'?
>
> How do you think an API call would look?

Just a function callback for doing the filtering. If the callback
returned 1 the distro object is kept, otherwise it is discarded.

typedef int (*osfilter-t)(osdistro_t *distro, void *opaque);
  osdistro_t **os_distro_find(osinfo_t, osfilter_t filter, void *opaque);
> > Read a property from an OS distro
> > Read all properties from an OS distro
>
> Why would an API user want to do this? I don't see why we would need to
> enable this specifically, rather than make the user do this iteratively.
> > List unique values for a property across all distros
>
> Not sure I fully understand this. Why would a user want this?

Imagine virt-manager building a 2 level hiearchy. For the first level
it wants to use this method

- Get a list of uninque vendor names among all known OS distros

eg

char **vendors = os_info_unique_properties(osinfo, "vendor");

Upon selecting a vendor, it then wants the earlier method

- Get a list of all OS distros for that vendor

eg

int vendorfilter(osdistro_t distro, void *opaque) {
         char *wantvendor = opaque;

char *gotvendor = os_distro_get_prop(distro, "vendor");
         if (gotvendor && STREQ(wantvendor, gotvendor)
              return 1;
         return 0;
      }

osdistro_t *distros = os_distro_find(osinfo, vendorfilter, "Red Hat");
> > > OS being installed
> > > Virt type ('hvm' vs. 'xen')
> > > Guest Architecture (i386, x86_64, ...)
> > > Hypervisor (kvm, qemu, xen, vbox, ...)
> > > Hypervisor version
> > > Libvirt version
> > >
> > > We would need to find the intersection of what the OS, the hypervisor,
> > >   and libvirt support, and return what we decide is the best choice.
> > >
> > > How to expose this in the API? We could simply have one long function
> > >
> > > os_lookup_device_value(char *os_id, char *virt_type, char *arch, ...)
> > >
> > > It works, but its pretty tedious, and I'm afraid that we would need
> > >   even more info to make a correct choice in the future, and the above
> > >   isn't flexible. We may also need some of the above info for other values
> > >   (ACPI/APIC settings, returning a proper install url may depend on arch).
> > >   Any suggestions?
> >
> > The more I think about this, the more I think we should avoid any specific
> > named attributes in the API. Supported devices are just other types of
> > property we can associated with a distro, in addition to ones I already
> > listed earlier. This could be useful in the UI too, for example, if you
> > know the hypervisor requires support for 'Xen paravirt disk', then when
> > browsing OS, you can filter on this property just as you would with the
> > others.
>
> If the app already knows their hypervisor requires 'Xen paravirt disk',
> the above is fine, and we should facilitate filtering like that. However
> I would like osinfo to save the user from having to know those details:
> they should just be able to say 'I'm using xenpv on i386 for xen 1.2.3
> and libvirt 4.5.6 with distro fedora10' and osinfo can return the
> required info. So I still don't see how to solve the above problem.
>
> Let's say we are installing winxp on qemu via libvirt: we want to know
> the recommended sound device model:
>
> winxp prefers ac97, es1370
> qemu supports es1370, if >= 0.10.0, supports ac97
> libvirt supports es1370, if >= 0.6.0 supports ac97
>
> How do we solve this via the API?

The latter 2 questions really say that libvirt needs to export more info
about what a driver has.

For the former, the OS info database needs to somehow provide a list of
drivers available for each OS. Obviously you can take the intersection,
but perhaps also define that they are listed in preferred order, "best"
first.

Perhaps we should be really ambitious and for each OS distro allow for a
full list of PCI device IDs it supports. For linux you can auto-generate
that from the kernel module metadata. You could even imagine that the 
distro provide an XML file in our format with this info in their release
media/trees.

Daniel

[et-mgmt-tools] RFC: libosinfo: Library for virt OS/distro metadata 3

[Cole Robinson]

John Levon wrote:
> On Sun, Jun 14, 2009 at 06:50:02PM -0400, Cole Robinson wrote:
> > git clone http://fedorapeople.org/~crobinso/osinfo/.git
>
> I'm not convinced by the "arbitrary hierarchy" thing: I just don't see
> how that could possibly be useful. Surely it's either an entirely flat
> namespace, or a shallow structured one like you have.
>
> The flat namespace would be a set of keys and multiple values for each
> key. One value would be "os type" (linux, windows, etc.). You'd most
> likely have a "generic" entry still for fallback.
>
> An alternative is to represent the hierarchy in the UNIX way, via the
> filesystem:
>
> /var/lib/osinfo/
> /var/lib/osinfo/linux/info.xml
> /var/lib/osinfo/linux/fedora/info.xml
> /var/lib/osinfo/linux/fedora/8/info.xml
>
> The fallback is pretty obvious then. Maybe it's over the top. The idea
> of a single delivered XML file that users edit does trouble me though.
> Maybe we at least have two files, one for customisations.

The single XML file approach certainly isn't the way forward. There are
numerous ways we can solve the 'let admin customize osinfo' problem, but
that can come after pinning down the API I think.
> > int     os_init();
> > void    os_close();
>
> Always, always, always, pass back an opaque identifier in an API - you
> never know when you'll need to track per-thread state. It's generally a
> good idea to pass in a version define too.

Sounds good, but I'm not sure what you mean by passing in a version define?
> The XML parsing itself should happen lazily. We might want to let the
> user specify a file, for example.
> > int     os_find_families        (char ***list);
> > int     os_find_distros         (const char *parent_id, char ***list);
> > int     os_find_releases        (const char *parent_id, char ***list);
> > int     os_find_updates         (const char *parent_id, char ***list);
>
> Regardless of the hierarchy question I hate the idea of exposing it in
> the API like this.
>
> There's really two (hopefully three eventually) things this library
> needs to do: provide a list of everything it knows about so the user can
> select it in a GUI or whatever, and provide configuration
> recommendations given a particular set of values.
>
> I don't think the library can make any assumptions about how the former
> might look. It just needs to return some thing like:
>
> struct osinfo {
>    	const char *id;
>    	nvpair_list_t values;
>    };
>
> Either allocated as an array (probably easiest) or in a list. It's then
> up to the client to decide what hierarchy to actually use, and it's all
> dependent on what values they pick out of the nv list.

Agreed, I think Dan's mail covered this pretty well.
> (The third thing is to identify OS types based upon installation media
> when possible.)

Agreed, I'd like to do all detection here in the future.
> > If we do away with the family/distro/... distinction, the user won't have
> >   much choice in the matter, but the 'family' concept (e.g. value of
> >   'Red Hat') isn't very useful to expose to a user.
>
> If you mean API user, think "icon".
> > How should we handle derivatives like Scientific Linux + CentOS: should we
> > expect users to understand they are based on RHEL, or give them explicit
> > IDs?
>
> Explicit IDs. You'd be surprised.
> > We would need to find the intersection of what the OS, the hypervisor,
> >   and libvirt support, and return what we decide is the best choice.
> >
> > How to expose this in the API? We could simply have one long function
> >
> > os_lookup_device_value(char *os_id, char *virt_type, char *arch, ...)
>
> The API user should pass in an nvlist, where a set of the names are
> defined and known about. The response needs to indicate whether it's a
> preferred setting ("would like virtio") or a required one.

I can see doing something like

os_info_set_install_prop(os_info_t info, int prop, char *propval)

So the API user might do:

os_info_set_install_prop(myinfo, OS_INSTALL_VIRT_TYPE, "hvm");
os_info_set_install_prop(myinfo, OS_INSTALL_ARCH, "x86_64");
os_info_set_install_prop(myinfo, OS_INSTALL_HV_TYPE, "kvm");

Then lookup device properties like you would any other prop.

Thanks!
- Cole

[John Levon]

On Mon, Jun 15, 2009 at 04:27:49PM +0100, Daniel P. Berrange wrote:
> > > Find OS distros, matching a specific set of properties
> >
> > What uses do you have in mind for this? Being able to say e.g. 'all
> > distros that support xen PV for IA64'?
> >
> > How do you think an API call would look?
>
> Just a function callback for doing the filtering. If the callback
> returned 1 the distro object is kept, otherwise it is discarded.
>
> typedef int (*osfilter-t)(osdistro_t *distro, void *opaque);
>   osdistro_t **os_distro_find(osinfo_t, osfilter_t filter, void *opaque);

I don't really like the idea of filtering. Instead it should just be a
callback API full stop. It's up to the client if they want to filter, or
find a specific entry, or whatever.

The lifetime rules are such that an entry exists whilst the handle is
open, so there's no issues with borrowing pointers in the callback etc.
> Imagine virt-manager building a 2 level hiearchy. For the first level
> it wants to use this method
> Get a list of uninque vendor names among all known OS distros

Pruning duplicates is surely something done in the client. It
really doesn't need library help...

regards
john

[et-mgmt-tools] RFC: libosinfo: Library for virt OS/distro metadata 3

[John Levon]

On Mon, Jun 15, 2009 at 11:28:09AM -0400, Cole Robinson wrote:
> > > int     os_init();
> > > void    os_close();
> >
> > Always, always, always, pass back an opaque identifier in an API - you
> > never know when you'll need to track per-thread state. It's generally a
> > good idea to pass in a version define too.
>
> Sounds good, but I'm not sure what you mean by passing in a version define?

osinfo.h:

...
#define OSINFO_VERSION 1
...

client.c:

#include <osinfo.h>

oi_handle_t os_init(OSINFO_VERSION);

This allows certain incompatible changes without having to rev the
soversion.

regards
john

[et-mgmt-tools] RFC: libosinfo: Library for virt OS/distro metadata 3

[John Levon]

On Mon, Jun 15, 2009 at 11:28:09AM -0400, Cole Robinson wrote:
> > The API user should pass in an nvlist, where a set of the names are
> > defined and known about. The response needs to indicate whether it's a
> > preferred setting ("would like virtio") or a required one.
>
> I can see doing something like
>
> os_info_set_install_prop(os_info_t info, int prop, char *propval)
>
> So the API user might do:
>
> os_info_set_install_prop(myinfo, OS_INSTALL_VIRT_TYPE, "hvm");
> os_info_set_install_prop(myinfo, OS_INSTALL_ARCH, "x86_64");
> os_info_set_install_prop(myinfo, OS_INSTALL_HV_TYPE, "kvm");

This isn't going to work as we most definitely have more than one value
of all of these settings.

Instead we need to pass in a list of "environments". Each one would
specify a particular combination of the values above (along with a
'preferred' setting methinks).

regards
john

[Daniel P. Berrange]

On Mon, Jun 15, 2009 at 11:50:18AM -0400, John Levon wrote:
> On Mon, Jun 15, 2009 at 04:27:49PM +0100, Daniel P. Berrange wrote:
> > > > Find OS distros, matching a specific set of properties
> > >
> > > What uses do you have in mind for this? Being able to say e.g. 'all
> > > distros that support xen PV for IA64'?
> > >
> > > How do you think an API call would look?
> >
> > Just a function callback for doing the filtering. If the callback
> > returned 1 the distro object is kept, otherwise it is discarded.
> >
> > typedef int (*osfilter-t)(osdistro_t *distro, void *opaque);
> >   osdistro_t **os_distro_find(osinfo_t, osfilter_t filter, void *opaque);
>
> I don't really like the idea of filtering. Instead it should just be a
> callback API full stop. It's up to the client if they want to filter, or
> find a specific entry, or whatever.

So you mean that it can just be a list iterator pattern

typedef int (*osdistro_iter)(osdistro_t *distro, void *opaque);
   int os_distro_iterate(osinfo_t, osdistro_iter iter, void *opaque);
> > Imagine virt-manager building a 2 level hiearchy. For the first level
> > it wants to use this method
> > Get a list of uninque vendor names among all known OS distros
>
> Pruning duplicates is surely something done in the client. It
> really doesn't need library help...

I was thinking that it may well be more efficient to let the library do it.
That said, this is an optimization we can add later if observed to be
neccessary in real world

Daniel

[John Levon]

On Mon, Jun 15, 2009 at 05:00:00PM +0100, Daniel P. Berrange wrote:
> > I don't really like the idea of filtering. Instead it should just be a
> > callback API full stop. It's up to the client if they want to filter, or
> > find a specific entry, or whatever.
>
> So you mean that it can just be a list iterator pattern
>
> typedef int (*osdistro_iter)(osdistro_t *distro, void *opaque);
>    int os_distro_iterate(osinfo_t, osdistro_iter iter, void *opaque);

Yep.

regards
john

[et-mgmt-tools] RFC: libosinfo: Library for virt OS/distro metadata 3

[Cole Robinson]

John Levon wrote:
> On Mon, Jun 15, 2009 at 11:28:09AM -0400, Cole Robinson wrote:
> > > The API user should pass in an nvlist, where a set of the names are
> > > defined and known about. The response needs to indicate whether it's a
> > > preferred setting ("would like virtio") or a required one.
> >
> > I can see doing something like
> >
> > os_info_set_install_prop(os_info_t info, int prop, char *propval)
> >
> > So the API user might do:
> >
> > os_info_set_install_prop(myinfo, OS_INSTALL_VIRT_TYPE, "hvm");
> > os_info_set_install_prop(myinfo, OS_INSTALL_ARCH, "x86_64");
> > os_info_set_install_prop(myinfo, OS_INSTALL_HV_TYPE, "kvm");
>
> This isn't going to work as we most definitely have more than one value
> of all of these settings.

The values the user sets are for what kind of guest they are installing
at that moment (x86_64 kvm in this case, i686 xen PV in another). That
teaches osinfo about our current setup, so it can give us valid
recommendations for device properties and whatnot. This could take place
after we already chose an OS ID.
> Instead we need to pass in a list of "environments". Each one would
> specify a particular combination of the values above (along with a
> 'preferred' setting methinks).

Maybe we can avoid an explicit 'preferred' concept, and just return a
list of supported values to the user. osinfo will put it's 'preferred'
choice as the first in the list, but if the user wants to differ, they
can choose from the other values in the list.

I guess this also raises the question of how we indicate 'return a
single value' vs. 'return a list' when fetching info.

Thanks,
Cole

[et-mgmt-tools] RFC: libosinfo: Library for virt OS/distro metadata 3

[John Levon]

On Mon, Jun 15, 2009 at 12:11:18PM -0400, Cole Robinson wrote:
> > > > The API user should pass in an nvlist, where a set of the names are
> > > > defined and known about. The response needs to indicate whether it's a
> > > > preferred setting ("would like virtio") or a required one.
> > >
> > > I can see doing something like
> > >
> > > os_info_set_install_prop(os_info_t info, int prop, char *propval)
> > >
> > > So the API user might do:
> > >
> > > os_info_set_install_prop(myinfo, OS_INSTALL_VIRT_TYPE, "hvm");
> > > os_info_set_install_prop(myinfo, OS_INSTALL_ARCH, "x86_64");
> > > os_info_set_install_prop(myinfo, OS_INSTALL_HV_TYPE, "kvm");
> >
> > This isn't going to work as we most definitely have more than one value
> > of all of these settings.
>
> The values the user sets are for what kind of guest they are installing
> at that moment (x86_64 kvm in this case, i686 xen PV in another).

That's backwards, though. I don't care about kvm or xen. I care about
installing a particular guest type, and want the library to tell me the
best method. To do that it needs to match guest needs against host
capabilities, and that implies the above properties need to be
multi-valued. There is no one "golden setup" even on a single system and
it would be a major mistake to presume there ever will be.
> > Instead we need to pass in a list of "environments". Each one would
> > specify a particular combination of the values above (along with a
> > 'preferred' setting methinks).
>
> Maybe we can avoid an explicit 'preferred' concept, and just return a
> list of supported values to the user. osinfo will put it's 'preferred'
> choice as the first in the list, but if the user wants to differ, they
> can choose from the other values in the list.

There may be more than one preferred setting ('kvm or xenpv, but I'd
avoid xenhvm'). Possibly we need a more nuanced notion.

regards
john

[et-mgmt-tools] RFC: libosinfo: Library for virt OS/distro metadata 3

[Cole Robinson]

John Levon wrote:
> On Mon, Jun 15, 2009 at 12:11:18PM -0400, Cole Robinson wrote:
> > > > > The API user should pass in an nvlist, where a set of the names are
> > > > > defined and known about. The response needs to indicate whether it's a
> > > > > preferred setting ("would like virtio") or a required one.
> > > >
> > > > I can see doing something like
> > > >
> > > > os_info_set_install_prop(os_info_t info, int prop, char *propval)
> > > >
> > > > So the API user might do:
> > > >
> > > > os_info_set_install_prop(myinfo, OS_INSTALL_VIRT_TYPE, "hvm");
> > > > os_info_set_install_prop(myinfo, OS_INSTALL_ARCH, "x86_64");
> > > > os_info_set_install_prop(myinfo, OS_INSTALL_HV_TYPE, "kvm");
> > >
> > > This isn't going to work as we most definitely have more than one value
> > > of all of these settings.
> >
> > The values the user sets are for what kind of guest they are installing
> > at that moment (x86_64 kvm in this case, i686 xen PV in another).
>
> That's backwards, though. I don't care about kvm or xen. I care about
> installing a particular guest type, and want the library to tell me the
> best method. To do that it needs to match guest needs against host
> capabilities, and that implies the above properties need to be
> multi-valued. There is no one "golden setup" even on a single system and
> it would be a major mistake to presume there ever will be.

No presumption here. In virt-manager, those above values are chosen by
the user (qemu vs. kvm vs. xenner, arch, xenpv vs. xenfv). I'm not
saying those above API calls would be hard coded, it would be the result of:

./virt-install --connect qemu:///system --arch x86_64 --virt-type kvm
--os-variant foobar ...

I hear you that it would be nice if the user could say 'here's the OS I
want, here's my host config, DO IT!', and to some degree
virt-manager/virt-install already plays that role, but at the osinfo
library it can come later and isn't a big priority at the moment. I'm
interested in just reaching parity with the current virtinst osdict
solution for now.
> > > Instead we need to pass in a list of "environments". Each one would
> > > specify a particular combination of the values above (along with a
> > > 'preferred' setting methinks).
> >
> > Maybe we can avoid an explicit 'preferred' concept, and just return a
> > list of supported values to the user. osinfo will put it's 'preferred'
> > choice as the first in the list, but if the user wants to differ, they
> > can choose from the other values in the list.
>
> There may be more than one preferred setting ('kvm or xenpv, but I'd
> avoid xenhvm'). Possibly we need a more nuanced notion.
>
> regards
> john

Thanks,
Cole

[et-mgmt-tools] RFC: libosinfo: Library for virt OS/distro metadata 3

[John Levon]

On Mon, Jun 15, 2009 at 12:51:49PM -0400, Cole Robinson wrote:
> > > The values the user sets are for what kind of guest they are installing
> > > at that moment (x86_64 kvm in this case, i686 xen PV in another).
> >
> > That's backwards, though. I don't care about kvm or xen. I care about
> > installing a particular guest type, and want the library to tell me the
> > best method. To do that it needs to match guest needs against host
> > capabilities, and that implies the above properties need to be
> > multi-valued. There is no one "golden setup" even on a single system and
> > it would be a major mistake to presume there ever will be.
>
> No presumption here. In virt-manager, those above values are chosen by
> the user (qemu vs. kvm vs. xenner, arch, xenpv vs. xenfv).

We aren't writing libvirtmanager though.
> I'm not saying those above API calls would be hard coded, it would be
> the result of:
>
> ./virt-install --connect qemu:///system --arch x86_64 --virt-type kvm
> --os-variant foobar ...
>
> I hear you that it would be nice if the user could say 'here's the OS I
> want, here's my host config, DO IT!', and to some degree
> virt-manager/virt-install already plays that role, but at the osinfo
> library it can come later

^^^^^^^^^^^^^^^^^

No, you're proposing an API which prevents it, that is, one value per
key (one hypervisor type, one arch, etc.). That's precisely my
complaint.

By all means make the current /implementation/ throw its hands up if
given more than one virtenv[1]. Just don't encode it into the API.

regards
john

[1] guest-arch+hypervisor-type+virt-type+... combo