VDSM Repository Reorganization

[Federico Simoncelli]

It is some time now that we are discussing an eventual repository 
reorganization for vdsm. In fact I'm sure that we all experienced
at least once the discomfort of having several modules scattered
around the tree.
The main goal of the reorganization would be to place the modules
in their proper location so that they can be used (imported) without
any special change (or hack) even when the code is executed inside
the development repository (e.g. tests).

Recently there has been an initial proposal about moving some of
these modules:

http://gerrit.ovirt.org/#/c/11858/

That spawned an interesting discussion that must involve the entire
community; in fact before starting any work we should try to converge
on a decision for the final repository structure in order to minimize
the discomfort for the contributors that will be forced to rebase
their pending gerrit patches. Even if the full reorganization won't
happen in a short time I think we should plan the entire structure
now and then eventually move only few relevant modules to their final
location.

To start the discussion I'm attaching here a sketch of the vdsm
repository structure that I envision:

.
|-- client
|   |-- [...]
|   `-- vdsClient.py
|-- common
|   |-- [...]
|   |-- betterPopen
|   |   `-- [...]
|   `-- vdsm
|       |-- [...]
|       `-- config.py
|-- contrib
|   |-- [...]
|   |-- nfs-check.py
|   `-- sos
|-- daemon
|   |-- [...]
|   |-- supervdsm.py
|   `-- vdsmd
`-- tool
    |-- [...]
    `-- vdsm-tool

This would allow any component (daemon, client, tool, etc...) to run
in place with the only addition of PYTHONPATH=$(top_srcdir)/common.
The tests would need also the additional component path but that is
a given since it's what they should be testing.

Once the components are in the proper place we can eventually start
using distutils inside the Makefile.am files in order to simplify the
installation process (and also as verification that our repository
structure is complying with the python standards).

Please get involved, share your feedback and proposals.
Thanks,
-- 
Federico

On Mon, Feb 11, 2013 at 12:17:39PM -0500, Federico Simoncelli wrote:
>  It is some time now that we are discussing an eventual repository
>  reorganization for vdsm. In fact I'm sure that we all experienced
>  at least once the discomfort of having several modules scattered
>  around the tree.
>  The main goal of the reorganization would be to place the modules
>  in their proper location so that they can be used (imported) without
>  any special change (or hack) even when the code is executed inside
>  the development repository (e.g. tests).
>
>  Recently there has been an initial proposal about moving some of
>  these modules:
>
>  http://gerrit.ovirt.org/#/c/11858/
>
>  That spawned an interesting discussion that must involve the entire
>  community; in fact before starting any work we should try to converge
>  on a decision for the final repository structure in order to minimize
>  the discomfort for the contributors that will be forced to rebase
>  their pending gerrit patches. Even if the full reorganization won't
>  happen in a short time I think we should plan the entire structure
>  now and then eventually move only few relevant modules to their final
>  location.
>
>  To start the discussion I'm attaching here a sketch of the vdsm
>  repository structure that I envision:
>
>  .
>  |-- client
>  |   |-- [...]
>  |   `-- vdsClient.py
>  |-- common
>  |   |-- [...]
>  |   |-- betterPopen
>  |   |   `-- [...]
>  |   `-- vdsm
>  |       |-- [...]
>  |       `-- config.py
>  |-- contrib
>  |   |-- [...]
>  |   |-- nfs-check.py
>  |   `-- sos
>  |-- daemon
>  |   |-- [...]
>  |   |-- supervdsm.py
>  |   `-- vdsmd
>  `-- tool
>      |-- [...]
>      `-- vdsm-tool 
Wondering if common should be named lib since you'll most likely want to
install it to the python lib dir anyway. Minor bikeshedding.

Why the distinction between client and tool? Do you expect other scripts
to be added and give them all their own directory? I'd expect that most
of their code would mostly live in vdsm and the actual executables are
thin wrappers. This would match up with the scripts dir distutils has.

Where do the hooks go?

>  This would allow any component (daemon, client, tool, etc...) to run
>  in place with the only addition of PYTHONPATH=$(top_srcdir)/common.
>  The tests would need also the additional component path but that is
>  a given since it's what they should be testing. 
+1 on fewer hacks to load the environment.

>  Once the components are in the proper place we can eventually start
>  using distutils inside the Makefile.am files in order to simplify the
>  installation process (and also as verification that our repository
>  structure is complying with the python standards). 
+1 IMHO use of auto* should be minimized when developing python.

[Dan Kenigsberg]

On Tue, Feb 12, 2013 at 01:58:01AM +0100, Ewoud Kohl van Wijngaarden wrote:
>  On Mon, Feb 11, 2013 at 12:17:39PM -0500, Federico Simoncelli wrote:
>  > It is some time now that we are discussing an eventual repository
>  > reorganization for vdsm. In fact I'm sure that we all experienced
>  > at least once the discomfort of having several modules scattered
>  > around the tree.
>  > The main goal of the reorganization would be to place the modules
>  > in their proper location so that they can be used (imported) without
>  > any special change (or hack) even when the code is executed inside
>  > the development repository (e.g. tests).
>  >
>  > Recently there has been an initial proposal about moving some of
>  > these modules:
>  >
>  > http://gerrit.ovirt.org/#/c/11858/
>  >
>  > That spawned an interesting discussion that must involve the entire
>  > community; in fact before starting any work we should try to converge
>  > on a decision for the final repository structure in order to minimize
>  > the discomfort for the contributors that will be forced to rebase
>  > their pending gerrit patches. Even if the full reorganization won't
>  > happen in a short time I think we should plan the entire structure
>  > now and then eventually move only few relevant modules to their final
>  > location.
>  >
>  > To start the discussion I'm attaching here a sketch of the vdsm
>  > repository structure that I envision:
>  >
>  > .
>  > |-- client
>  > |   |-- [...]
>  > |   `-- vdsClient.py
>  > |-- common
>  > |   |-- [...]
>  > |   |-- betterPopen
>  > |   |   `-- [...]
>  > |   `-- vdsm
>  > |       |-- [...]
>  > |       `-- config.py
>  > |-- contrib
>  > |   |-- [...]
>  > |   |-- nfs-check.py
>  > |   `-- sos
>  > |-- daemon
>  > |   |-- [...]
>  > |   |-- supervdsm.py
>  > |   `-- vdsmd
>  > `-- tool
>  >     |-- [...]
>  >     `-- vdsm-tool
>  
>  Wondering if common should be named lib since you'll most likely want to
>  install it to the python lib dir anyway. Minor bikeshedding.
>  
>  Why the distinction between client and tool? Do you expect other scripts
>  to be added and give them all their own directory? I'd expect that most
>  of their code would mostly live in vdsm and the actual executables are
>  thin wrappers. This would match up with the scripts dir distutils has. 
vdsm-tool is to be shipped with the server only. It is a tool that helps
the admin (and the sysv/systemd service) configure the host for vdsm.
It's code is part of common (or lib) only to make it easier to use it
with the default sys.path.

>  
>  Where do the hooks go? 
They, too, are part of the daemon, but I think they can stay as a
top-level directory as they are now.

>  
>  > This would allow any component (daemon, client, tool, etc...) to run
>  > in place with the only addition of PYTHONPATH=$(top_srcdir)/common.
>  > The tests would need also the additional component path but that is
>  > a given since it's what they should be testing.
>  
>  +1 on fewer hacks to load the environment.
>  
>  > Once the components are in the proper place we can eventually start
>  > using distutils inside the Makefile.am files in order to simplify the
>  > installation process (and also as verification that our repository
>  > structure is complying with the python standards).
>  
>  +1 IMHO use of auto* should be minimized when developing python.
>  _______________________________________________
>  vdsm-devel mailing list
>  vdsm-devel(a)lists.fedorahosted.org
>  https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel

On Mon, Feb 11, 2013 at 12:17:39PM -0500, Federico Simoncelli wrote:
>  It is some time now that we are discussing an eventual repository 
>  reorganization for vdsm. In fact I'm sure that we all experienced
>  at least once the discomfort of having several modules scattered
>  around the tree.
>  The main goal of the reorganization would be to place the modules
>  in their proper location so that they can be used (imported) without
>  any special change (or hack) even when the code is executed inside
>  the development repository (e.g. tests).
>  
>  Recently there has been an initial proposal about moving some of
>  these modules:
>  
>  http://gerrit.ovirt.org/#/c/11858/
>  
>  That spawned an interesting discussion that must involve the entire
>  community; in fact before starting any work we should try to converge
>  on a decision for the final repository structure in order to minimize
>  the discomfort for the contributors that will be forced to rebase
>  their pending gerrit patches. Even if the full reorganization won't
>  happen in a short time I think we should plan the entire structure
>  now and then eventually move only few relevant modules to their final
>  location.
>  
>  To start the discussion I'm attaching here a sketch of the vdsm
>  repository structure that I envision:
>  
>  .
>  |-- client
>  |   |-- [...]
>  |   `-- vdsClient.py
>  |-- common
>  |   |-- [...]
>  |   |-- betterPopen
>  |   |   `-- [...]
>  |   `-- vdsm
>  |       |-- [...]
>  |       `-- config.py
>  |-- contrib
>  |   |-- [...]
>  |   |-- nfs-check.py
>  |   `-- sos
>  |-- daemon
>  |   |-- [...]
>  |   |-- supervdsm.py
>  |   `-- vdsmd
>  `-- tool
>      |-- [...]
>      `-- vdsm-tool 
The schema file vdsmapi-schema.json (as well as the python module to parse it)
are needed by the server and clients.  Initially I'd think it should be
installed in 'common', but a client does not need things like betterPopen.  Any
recommendation on where the schema/API definition should live?

-- 
Adam Litke <agl(a)us.ibm.com&gt;
IBM Linux Technology Center

[Federico Simoncelli]

----- Original Message -----
>  From: "Adam Litke" <agl(a)us.ibm.com&gt;
>  To: "Federico Simoncelli" <fsimonce(a)redhat.com&gt;
>  Cc: "VDSM Project Development" <vdsm-devel(a)lists.fedorahosted.org&gt;,
> "Dan Kenigsberg" <danken(a)redhat.com&gt;, "Saggi
>  Mizrahi" <smizrahi(a)redhat.com&gt;, "Vinzenz Feenstra"
> <vfeenstr(a)redhat.com&gt;, "Ayal Baron" <abaron(a)redhat.com&gt;
>  Sent: Tuesday, February 12, 2013 9:08:09 PM
>  Subject: Re: VDSM Repository Reorganization
>  
>  On Mon, Feb 11, 2013 at 12:17:39PM -0500, Federico Simoncelli wrote:
>  > It is some time now that we are discussing an eventual repository
>  > reorganization for vdsm. In fact I'm sure that we all experienced
>  > at least once the discomfort of having several modules scattered
>  > around the tree.
>  > The main goal of the reorganization would be to place the modules
>  > in their proper location so that they can be used (imported)
>  > without
>  > any special change (or hack) even when the code is executed inside
>  > the development repository (e.g. tests).
>  > 
>  > Recently there has been an initial proposal about moving some of
>  > these modules:
>  > 
>  > http://gerrit.ovirt.org/#/c/11858/
>  > 
>  > That spawned an interesting discussion that must involve the entire
>  > community; in fact before starting any work we should try to
>  > converge
>  > on a decision for the final repository structure in order to
>  > minimize
>  > the discomfort for the contributors that will be forced to rebase
>  > their pending gerrit patches. Even if the full reorganization won't
>  > happen in a short time I think we should plan the entire structure
>  > now and then eventually move only few relevant modules to their
>  > final
>  > location.
>  > 
>  > To start the discussion I'm attaching here a sketch of the vdsm
>  > repository structure that I envision:
>  > 
>  > .
>  > |-- client
>  > |   |-- [...]
>  > |   `-- vdsClient.py
>  > |-- common
>  > |   |-- [...]
>  > |   |-- betterPopen
>  > |   |   `-- [...]
>  > |   `-- vdsm
>  > |       |-- [...]
>  > |       `-- config.py
>  > |-- contrib
>  > |   |-- [...]
>  > |   |-- nfs-check.py
>  > |   `-- sos
>  > |-- daemon
>  > |   |-- [...]
>  > |   |-- supervdsm.py
>  > |   `-- vdsmd
>  > `-- tool
>  >     |-- [...]
>  >     `-- vdsm-tool
>  
>  The schema file vdsmapi-schema.json (as well as the python module to
>  parse it) are needed by the server and clients.  Initially I'd think it
>  should be installed in 'common', but a client does not need things like
>  betterPopen. Any recommendation on where the schema/API definition
>  should live? 
The problem is that betterPopen is misplaced as it's used only by the
daemon.

Anyway let's not mix the three different aspects:

- repository structure
- installation location
- packaging

That said in my opinion it can remain in "common" (which can be renamed to
"lib" as Ewoud suggested) and be installed in the site-lib *but* be
packaged only with the daemon (while we're waiting for it to become fully
independent and be moved out of the repo).

I suppose that the code dealing with the schema will be shared by both the
client and the daemon and it will be placed in "common" (or "lib")
and
the schema itself can go there too.

To be honest I started envisioning client/common/daemon/tool as pure-python
directories (as they are once installed) so maybe the schema could be
placed somewhere else with other miscellaneous files (./schema? ./config?),
but I wouldn't stress over this if it's impractical.

-- 
Federico

First lets illustrate the current repository structure:
 - tests
 - vdsm
  - storage
  - gluster
  - betterPopen
   ... and more unused directories 
 - vdsm_api
 - vdsm_cli
 - vdsm_hooks
 - vdsm_log
 - vdsm_reg
 - vdsm_tools
 - vds_bootstrap
 - contrib

As I understand your proposal, you added 'common' and 'deamon' and left
the rest the way it is.
So vdsm can be renamed to vdsm_core and can include sub folders for deamon and common\lib
python files.
It'll be something like:
- vdsm_core
  - deamon
    - vdsmd 
    .. - more deamon scripts
  - lib - independent tools -> can also be part of vdsm_tools, as it doesn't part
of the vdsm_core
    - utils.py
    - betterPopen
    - dmidecode    
    ...  - more unrelated utils that vdsm uses
  - core
    - vdsm
    - API.py
    - clientIF.py
    - config.py defines.py constants.py ... should be part of vdsm_core
    - libvirtconnection.py
    ...  core files of vdsm
  - storage
    ... as it now without files that can be and should be moved to lib folder (e.g.
threadPool, task, taskManager and inc..)

the rest we'll stay the same.

about the installation location it something else..
In my opinion this arrangement is nice to have, but not a must - we'll still need to
set PYTHONPATH when we run from different locations (what you called making an hack), so
as I see it, it just for comfort.. no? Maybe I missed something..

(And tests can be split to subfolders for each part - core\tests lib\tests and inc..)

----- Original Message -----
From: "Federico Simoncelli" <fsimonce(a)redhat.com&gt;
To: "Adam Litke" <agl(a)us.ibm.com&gt;
Cc: "VDSM Project Development" <vdsm-devel(a)lists.fedorahosted.org&gt;
Sent: Wednesday, February 13, 2013 1:06:55 AM
Subject: Re: [vdsm] VDSM Repository Reorganization



----- Original Message -----
>  From: "Adam Litke" <agl(a)us.ibm.com&gt;
>  To: "Federico Simoncelli" <fsimonce(a)redhat.com&gt;
>  Cc: "VDSM Project Development" <vdsm-devel(a)lists.fedorahosted.org&gt;,
> "Dan Kenigsberg" <danken(a)redhat.com&gt;, "Saggi
>  Mizrahi" <smizrahi(a)redhat.com&gt;, "Vinzenz Feenstra"
> <vfeenstr(a)redhat.com&gt;, "Ayal Baron" <abaron(a)redhat.com&gt;
>  Sent: Tuesday, February 12, 2013 9:08:09 PM
>  Subject: Re: VDSM Repository Reorganization
>  
>  On Mon, Feb 11, 2013 at 12:17:39PM -0500, Federico Simoncelli wrote:
>  > It is some time now that we are discussing an eventual repository
>  > reorganization for vdsm. In fact I'm sure that we all experienced
>  > at least once the discomfort of having several modules scattered
>  > around the tree.
>  > The main goal of the reorganization would be to place the modules
>  > in their proper location so that they can be used (imported)
>  > without
>  > any special change (or hack) even when the code is executed inside
>  > the development repository (e.g. tests).
>  > 
>  > Recently there has been an initial proposal about moving some of
>  > these modules:
>  > 
>  > http://gerrit.ovirt.org/#/c/11858/
>  > 
>  > That spawned an interesting discussion that must involve the entire
>  > community; in fact before starting any work we should try to
>  > converge
>  > on a decision for the final repository structure in order to
>  > minimize
>  > the discomfort for the contributors that will be forced to rebase
>  > their pending gerrit patches. Even if the full reorganization won't
>  > happen in a short time I think we should plan the entire structure
>  > now and then eventually move only few relevant modules to their
>  > final
>  > location.
>  > 
>  > To start the discussion I'm attaching here a sketch of the vdsm
>  > repository structure that I envision:
>  > 
>  > .
>  > |-- client
>  > |   |-- [...]
>  > |   `-- vdsClient.py
>  > |-- common
>  > |   |-- [...]
>  > |   |-- betterPopen
>  > |   |   `-- [...]
>  > |   `-- vdsm
>  > |       |-- [...]
>  > |       `-- config.py
>  > |-- contrib
>  > |   |-- [...]
>  > |   |-- nfs-check.py
>  > |   `-- sos
>  > |-- daemon
>  > |   |-- [...]
>  > |   |-- supervdsm.py
>  > |   `-- vdsmd
>  > `-- tool
>  >     |-- [...]
>  >     `-- vdsm-tool
>  
>  The schema file vdsmapi-schema.json (as well as the python module to
>  parse it) are needed by the server and clients.  Initially I'd think it
>  should be installed in 'common', but a client does not need things like
>  betterPopen. Any recommendation on where the schema/API definition
>  should live? 
The problem is that betterPopen is misplaced as it's used only by the
daemon.

Anyway let's not mix the three different aspects:

- repository structure
- installation location
- packaging

That said in my opinion it can remain in "common" (which can be renamed to
"lib" as Ewoud suggested) and be installed in the site-lib *but* be
packaged only with the daemon (while we're waiting for it to become fully
independent and be moved out of the repo).

I suppose that the code dealing with the schema will be shared by both the
client and the daemon and it will be placed in "common" (or "lib")
and
the schema itself can go there too.

To be honest I started envisioning client/common/daemon/tool as pure-python
directories (as they are once installed) so maybe the schema could be
placed somewhere else with other miscellaneous files (./schema? ./config?),
but I wouldn't stress over this if it's impractical.

-- 
Federico
_______________________________________________
vdsm-devel mailing list
vdsm-devel(a)lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel

[Vinzenz Feenstra]

On 02/13/2013 03:08 PM, Yaniv Bronheim wrote:
>  First lets illustrate the current repository structure:
>    - tests
>    - vdsm
>     - storage
>     - gluster
>     - betterPopen
>      ... and more unused directories
>    - vdsm_api
>    - vdsm_cli
>    - vdsm_hooks
>    - vdsm_log
>    - vdsm_reg
>    - vdsm_tools
>    - vds_bootstrap
>    - contrib
>
>  As I understand your proposal, you added 'common' and 'deamon' and left
> the rest the way it is.
>  So vdsm can be renamed to vdsm_core and can include sub folders for deamon and common\lib
> python files.
>  It'll be something like:
>  - vdsm_core
>     - deamon
>       - vdsmd
>       .. - more deamon scripts
>     - lib - independent tools -> can also be part of vdsm_tools, as it doesn't part
> of the vdsm_core
>       - utils.py
>       - betterPopen
>       - dmidecode
>       ...  - more unrelated utils that vdsm uses
>     - core
>       - vdsm
>       - API.py
>       - clientIF.py
>       - config.py defines.py constants.py ... should be part of vdsm_core
>       - libvirtconnection.py
>       ...  core files of vdsm
>     - storage
>       ... as it now without files that can be and should be moved to lib folder (e.g.
> threadPool, task, taskManager and inc..)
>
>  the rest we'll stay the same.
>
>  about the installation location it something else..
>  In my opinion this arrangement is nice to have, but not a must - we'll still need to
> set PYTHONPATH when we run from different locations (what you called making an hack), so
> as I see it, it just for comfort.. no? Maybe I missed something..
>
>  (And tests can be split to subfolders for each part - core\tests lib\tests and inc..)
>
>  ----- Original Message -----
>  From: "Federico Simoncelli" <fsimonce(a)redhat.com&gt;
>  To: "Adam Litke" <agl(a)us.ibm.com&gt;
>  Cc: "VDSM Project Development" <vdsm-devel(a)lists.fedorahosted.org&gt;
>  Sent: Wednesday, February 13, 2013 1:06:55 AM
>  Subject: Re: [vdsm] VDSM Repository Reorganization
>
>
>
>  ----- Original Message -----
> > From: "Adam Litke" <agl(a)us.ibm.com&gt;
> > To: "Federico Simoncelli" <fsimonce(a)redhat.com&gt;
> > Cc: "VDSM Project Development" <vdsm-devel(a)lists.fedorahosted.org&gt;,
> "Dan Kenigsberg" <danken(a)redhat.com&gt;, "Saggi
> > Mizrahi" <smizrahi(a)redhat.com&gt;, "Vinzenz Feenstra"
> <vfeenstr(a)redhat.com&gt;, "Ayal Baron" <abaron(a)redhat.com&gt;
> > Sent: Tuesday, February 12, 2013 9:08:09 PM
> > Subject: Re: VDSM Repository Reorganization
> >
> > On Mon, Feb 11, 2013 at 12:17:39PM -0500, Federico Simoncelli wrote:
> >> It is some time now that we are discussing an eventual repository
> >> reorganization for vdsm. In fact I'm sure that we all experienced
> >> at least once the discomfort of having several modules scattered
> >> around the tree.
> >> The main goal of the reorganization would be to place the modules
> >> in their proper location so that they can be used (imported)
> >> without
> >> any special change (or hack) even when the code is executed inside
> >> the development repository (e.g. tests).
> >>
> >> Recently there has been an initial proposal about moving some of
> >> these modules:
> >>
> >> http://gerrit.ovirt.org/#/c/11858/
> >>
> >> That spawned an interesting discussion that must involve the entire
> >> community; in fact before starting any work we should try to
> >> converge
> >> on a decision for the final repository structure in order to
> >> minimize
> >> the discomfort for the contributors that will be forced to rebase
> >> their pending gerrit patches. Even if the full reorganization won't
> >> happen in a short time I think we should plan the entire structure
> >> now and then eventually move only few relevant modules to their
> >> final
> >> location.
> >>
> >> To start the discussion I'm attaching here a sketch of the vdsm
> >> repository structure that I envision:
> >>
> >> .
> >> |-- client
> >> |   |-- [...]
> >> |   `-- vdsClient.py
> >> |-- common
> >> |   |-- [...]
> >> |   |-- betterPopen
> >> |   |   `-- [...]
> >> |   `-- vdsm
> >> |       |-- [...]
> >> |       `-- config.py
> >> |-- contrib
> >> |   |-- [...]
> >> |   |-- nfs-check.py
> >> |   `-- sos
> >> |-- daemon
> >> |   |-- [...]
> >> |   |-- supervdsm.py
> >> |   `-- vdsmd
> >> `-- tool
> >>      |-- [...]
> >>      `-- vdsm-tool
> > The schema file vdsmapi-schema.json (as well as the python module to
> > parse it) are needed by the server and clients.  Initially I'd think it
> > should be installed in 'common', but a client does not need things like
> > betterPopen. Any recommendation on where the schema/API definition
> > should live?
>  The problem is that betterPopen is misplaced as it's used only by the
>  daemon.
>
>  Anyway let's not mix the three different aspects: 
+1
>  - repository structure 
Well that's what is supposed to be the topic.
>  - installation location
>  - packaging 
Those are supposed to be off topic considerint $SUBJECT
>  That said in my opinion it can remain in "common" (which can be renamed to
>  "lib" as Ewoud suggested) and be installed in the site-lib *but* be
>  packaged only with the daemon (while we're waiting for it to become fully
>  independent and be moved out of the repo). 
Where such things could be put in a
folder 'extra' for now. (As an example)
>  I suppose that the code dealing with the schema will be shared by both the
>  client and the daemon and it will be placed in "common" (or "lib")
> and
>  the schema itself can go there too.
>
>  To be honest I started envisioning client/common/daemon/tool as pure-python
>  directories (as they are once installed) so maybe the schema could be
>  placed somewhere else with other miscellaneous files (./schema? ./config?),
>  but I wouldn't stress over this if it's impractical. 
I would like to see
also pure python directories.


A rough quick n dirty overview how I could imagine it to look like:

topdir/
         build-aux/*
         contrib/*
         deployment/
                     bootstrap/
                     log/
                     reg/
         doc/
             vdsm
             api
             cli
         hooks/*
         m4/*
         tests/*
         scripts/
                 vdsm/*
                 storage/*
                 network/*
         vdsm/
                 api/
                     json/
                         jsonrpc/
                         BindingJsonRpc.py
                     xml/
                         BindingXMLRPC.py
                     Bridge.py
                     vdsmapi-schema.json
                     process-schema.py
                     vdsmapi.py
                 cli/
                     vdsClient.py
                     VdsClientGluster.py
                 extra/
                     betterPopen/
                 shared/
                     vdsm/
                         SecureXMLRPCServer.py
                         config.py.in
                         constants.py.in
                         utils.py
                         vdscli.py.in*
                         utils.py
                         tool/
                 daemon/
                     storage/*
                     network/*
                     vdsmd/
                         vdsm
                         *


That said, I need to assert that I am not yet fully aware of all 
connections between the files and I guess the maintainers will know best 
about their respective parts of responsibility.

I'd like to add that it'd be probably a good idea to start having 
network and storage split off the main daemon for maintenance reasons, 
even if they interface each other.
I personally even could see a scenario where network and storage are 
working as standalone daemons. However that's off topic for know I guess.

-- 
Regards,

Vinzenz Feenstra | Senior Software Engineer
RedHat Engineering Virtualization R & D
Phone: [phone]
IRC: vfeenstr or evilissimo

Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com

[Vinzenz Feenstra]

Hi everyone,

It would be nice to come to an agreement any time soon. I would like to 
apply all the change as soon as possible.
I would not like to see this go into the depth of a dead hole.

Thanks.

On 02/13/2013 03:54 PM, Vinzenz Feenstra wrote:
>  On 02/13/2013 03:08 PM, Yaniv Bronheim wrote:
> > First lets illustrate the current repository structure:
> >   - tests
> >   - vdsm
> >    - storage
> >    - gluster
> >    - betterPopen
> >     ... and more unused directories
> >   - vdsm_api
> >   - vdsm_cli
> >   - vdsm_hooks
> >   - vdsm_log
> >   - vdsm_reg
> >   - vdsm_tools
> >   - vds_bootstrap
> >   - contrib
> >
> > As I understand your proposal, you added 'common' and 'deamon' and 
> > left the rest the way it is.
> > So vdsm can be renamed to vdsm_core and can include sub folders for 
> > deamon and common\lib python files.
> > It'll be something like:
> > - vdsm_core
> >    - deamon
> >      - vdsmd
> >      .. - more deamon scripts
> >    - lib - independent tools -> can also be part of vdsm_tools, as it 
> > doesn't part of the vdsm_core
> >      - utils.py
> >      - betterPopen
> >      - dmidecode
> >      ...  - more unrelated utils that vdsm uses
> >    - core
> >      - vdsm
> >      - API.py
> >      - clientIF.py
> >      - config.py defines.py constants.py ... should be part of vdsm_core
> >      - libvirtconnection.py
> >      ...  core files of vdsm
> >    - storage
> >      ... as it now without files that can be and should be moved to 
> > lib folder (e.g. threadPool, task, taskManager and inc..)
> >
> > the rest we'll stay the same.
> >
> > about the installation location it something else..
> > In my opinion this arrangement is nice to have, but not a must - 
> > we'll still need to set PYTHONPATH when we run from different 
> > locations (what you called making an hack), so as I see it, it just 
> > for comfort.. no? Maybe I missed something..
> >
> > (And tests can be split to subfolders for each part - core\tests 
> > lib\tests and inc..)
> >
> > ----- Original Message -----
> > From: "Federico Simoncelli" <fsimonce(a)redhat.com&gt;
> > To: "Adam Litke" <agl(a)us.ibm.com&gt;
> > Cc: "VDSM Project Development" <vdsm-devel(a)lists.fedorahosted.org&gt;
> > Sent: Wednesday, February 13, 2013 1:06:55 AM
> > Subject: Re: [vdsm] VDSM Repository Reorganization
> >
> >
> >
> > ----- Original Message -----
> >> From: "Adam Litke" <agl(a)us.ibm.com&gt;
> >> To: "Federico Simoncelli" <fsimonce(a)redhat.com&gt;
> >> Cc: "VDSM Project Development"
> <vdsm-devel(a)lists.fedorahosted.org&gt;, 
> >> "Dan Kenigsberg" <danken(a)redhat.com&gt;, "Saggi
> >> Mizrahi" <smizrahi(a)redhat.com&gt;, "Vinzenz Feenstra" 
> >> <vfeenstr(a)redhat.com&gt;, "Ayal Baron" <abaron(a)redhat.com&gt;
> >> Sent: Tuesday, February 12, 2013 9:08:09 PM
> >> Subject: Re: VDSM Repository Reorganization
> >>
> >> On Mon, Feb 11, 2013 at 12:17:39PM -0500, Federico Simoncelli wrote:
> >>> It is some time now that we are discussing an eventual repository
> >>> reorganization for vdsm. In fact I'm sure that we all experienced
> >>> at least once the discomfort of having several modules scattered
> >>> around the tree.
> >>> The main goal of the reorganization would be to place the modules
> >>> in their proper location so that they can be used (imported)
> >>> without
> >>> any special change (or hack) even when the code is executed inside
> >>> the development repository (e.g. tests).
> >>>
> >>> Recently there has been an initial proposal about moving some of
> >>> these modules:
> >>>
> >>> http://gerrit.ovirt.org/#/c/11858/
> >>>
> >>> That spawned an interesting discussion that must involve the entire
> >>> community; in fact before starting any work we should try to
> >>> converge
> >>> on a decision for the final repository structure in order to
> >>> minimize
> >>> the discomfort for the contributors that will be forced to rebase
> >>> their pending gerrit patches. Even if the full reorganization won't
> >>> happen in a short time I think we should plan the entire structure
> >>> now and then eventually move only few relevant modules to their
> >>> final
> >>> location.
> >>>
> >>> To start the discussion I'm attaching here a sketch of the vdsm
> >>> repository structure that I envision:
> >>>
> >>> .
> >>> |-- client
> >>> |   |-- [...]
> >>> |   `-- vdsClient.py
> >>> |-- common
> >>> |   |-- [...]
> >>> |   |-- betterPopen
> >>> |   |   `-- [...]
> >>> |   `-- vdsm
> >>> |       |-- [...]
> >>> |       `-- config.py
> >>> |-- contrib
> >>> |   |-- [...]
> >>> |   |-- nfs-check.py
> >>> |   `-- sos
> >>> |-- daemon
> >>> |   |-- [...]
> >>> |   |-- supervdsm.py
> >>> |   `-- vdsmd
> >>> `-- tool
> >>>      |-- [...]
> >>>      `-- vdsm-tool
> >> The schema file vdsmapi-schema.json (as well as the python module to
> >> parse it) are needed by the server and clients.  Initially I'd think it
> >> should be installed in 'common', but a client does not need things like
> >> betterPopen. Any recommendation on where the schema/API definition
> >> should live?
> > The problem is that betterPopen is misplaced as it's used only by the
> > daemon.
> >
> > Anyway let's not mix the three different aspects:
>  +1
> >
> > - repository structure
>  Well that's what is supposed to be the topic.
> > - installation location
> > - packaging
>  Those are supposed to be off topic considerint $SUBJECT
> >
> > That said in my opinion it can remain in "common" (which can be 
> > renamed to
> > "lib" as Ewoud suggested) and be installed in the site-lib *but* be
> > packaged only with the daemon (while we're waiting for it to become 
> > fully
> > independent and be moved out of the repo).
>  Where such things could be put in a folder 'extra' for now. (As an 
>  example)
> >
> > I suppose that the code dealing with the schema will be shared by 
> > both the
> > client and the daemon and it will be placed in "common" (or
> "lib") and
> > the schema itself can go there too.
> >
> > To be honest I started envisioning client/common/daemon/tool as 
> > pure-python
> > directories (as they are once installed) so maybe the schema could be
> > placed somewhere else with other miscellaneous files (./schema? 
> > ./config?),
> > but I wouldn't stress over this if it's impractical.
>  I would like to see also pure python directories.
>
>
>  A rough quick n dirty overview how I could imagine it to look like:
>
>  topdir/
>          build-aux/*
>          contrib/*
>          deployment/
>                      bootstrap/
>                      log/
>                      reg/
>          doc/
>              vdsm
>              api
>              cli
>          hooks/*
>          m4/*
>          tests/*
>          scripts/
>                  vdsm/*
>                  storage/*
>                  network/*
>          vdsm/
>                  api/
>                      json/
>                          jsonrpc/
>                          BindingJsonRpc.py
>                      xml/
>                          BindingXMLRPC.py
>                      Bridge.py
>                      vdsmapi-schema.json
>                      process-schema.py
>                      vdsmapi.py
>                  cli/
>                      vdsClient.py
>                      VdsClientGluster.py
>                  extra/
>                      betterPopen/
>                  shared/
>                      vdsm/
>                          SecureXMLRPCServer.py
>                          config.py.in
>                          constants.py.in
>                          utils.py
>                          vdscli.py.in*
>                          utils.py
>                          tool/
>                  daemon/
>                      storage/*
>                      network/*
>                      vdsmd/
>                          vdsm
>                          *
>
>
>  That said, I need to assert that I am not yet fully aware of all 
>  connections between the files and I guess the maintainers will know 
>  best about their respective parts of responsibility.
>
>  I'd like to add that it'd be probably a good idea to start having 
>  network and storage split off the main daemon for maintenance reasons, 
>  even if they interface each other.
>  I personally even could see a scenario where network and storage are 
>  working as standalone daemons. However that's off topic for know I guess.
>  
-- 
Regards,

Vinzenz Feenstra | Senior Software Engineer
RedHat Engineering Virtualization R & D
Phone: [phone]
IRC: vfeenstr or evilissimo

Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com

[Dave Neary]

Hi Vinzenz,

On 02/18/2013 05:43 PM, Vinzenz Feenstra wrote:
>  It would be nice to come to an agreement any time soon. I would like
> to
>  apply all the change as soon as possible.
>  I would not like to see this go into the depth of a dead hole. 
The absence of feedback typically means one of 6 things:

* No-one read it
* No-one understood it
* No-one cared
* No-one is paying attention to you any more
* It's so ridiculous it's unworthy of comment
* Everyone who read it agrees

I suggest you assume the last one, and proceed until someone starts 
shouting at you ;-)

Thanks,
Dave.
-- 
Dave Neary - Community Action and Impact
Open Source and Standards, Red Hat - http://community.redhat.com
Ph: [phone] / Cell: [phone]

于 2013-2-13 22:54, Vinzenz Feenstra 写道:
>  On 02/13/2013 03:08 PM, Yaniv Bronheim wrote:
> > First lets illustrate the current repository structure:
> >   - tests
> >   - vdsm
> >    - storage
> >    - gluster
> >    - betterPopen
> >     ... and more unused directories
> >   - vdsm_api
> >   - vdsm_cli
> >   - vdsm_hooks
> >   - vdsm_log
> >   - vdsm_reg
> >   - vdsm_tools
> >   - vds_bootstrap
> >   - contrib
> >
> > As I understand your proposal, you added 'common' and 'deamon' and 
> > left the rest the way it is.
> > So vdsm can be renamed to vdsm_core and can include sub folders for 
> > deamon and common\lib python files.
> > It'll be something like:
> > - vdsm_core
> >    - deamon
> >      - vdsmd
> >      .. - more deamon scripts
> >    - lib - independent tools -> can also be part of vdsm_tools, as it 
> > doesn't part of the vdsm_core
> >      - utils.py
> >      - betterPopen
> >      - dmidecode
> >      ...  - more unrelated utils that vdsm uses
> >    - core
> >      - vdsm
> >      - API.py
> >      - clientIF.py
> >      - config.py defines.py constants.py ... should be part of vdsm_core
> >      - libvirtconnection.py
> >      ...  core files of vdsm
> >    - storage
> >      ... as it now without files that can be and should be moved to 
> > lib folder (e.g. threadPool, task, taskManager and inc..)
> >
> > the rest we'll stay the same.
> >
> > about the installation location it something else..
> > In my opinion this arrangement is nice to have, but not a must - 
> > we'll still need to set PYTHONPATH when we run from different 
> > locations (what you called making an hack), so as I see it, it just 
> > for comfort.. no? Maybe I missed something..
> >
> > (And tests can be split to subfolders for each part - core\tests 
> > lib\tests and inc..)
> >
> > ----- Original Message -----
> > From: "Federico Simoncelli" <fsimonce(a)redhat.com&gt;
> > To: "Adam Litke" <agl(a)us.ibm.com&gt;
> > Cc: "VDSM Project Development" <vdsm-devel(a)lists.fedorahosted.org&gt;
> > Sent: Wednesday, February 13, 2013 1:06:55 AM
> > Subject: Re: [vdsm] VDSM Repository Reorganization
> >
> >
> >
> > ----- Original Message -----
> >> From: "Adam Litke" <agl(a)us.ibm.com&gt;
> >> To: "Federico Simoncelli" <fsimonce(a)redhat.com&gt;
> >> Cc: "VDSM Project Development"
> <vdsm-devel(a)lists.fedorahosted.org&gt;, 
> >> "Dan Kenigsberg" <danken(a)redhat.com&gt;, "Saggi
> >> Mizrahi" <smizrahi(a)redhat.com&gt;, "Vinzenz Feenstra" 
> >> <vfeenstr(a)redhat.com&gt;, "Ayal Baron" <abaron(a)redhat.com&gt;
> >> Sent: Tuesday, February 12, 2013 9:08:09 PM
> >> Subject: Re: VDSM Repository Reorganization
> >>
> >> On Mon, Feb 11, 2013 at 12:17:39PM -0500, Federico Simoncelli wrote:
> >>> It is some time now that we are discussing an eventual repository
> >>> reorganization for vdsm. In fact I'm sure that we all experienced
> >>> at least once the discomfort of having several modules scattered
> >>> around the tree.
> >>> The main goal of the reorganization would be to place the modules
> >>> in their proper location so that they can be used (imported)
> >>> without
> >>> any special change (or hack) even when the code is executed inside
> >>> the development repository (e.g. tests).
> >>>
> >>> Recently there has been an initial proposal about moving some of
> >>> these modules:
> >>>
> >>> http://gerrit.ovirt.org/#/c/11858/
> >>>
> >>> That spawned an interesting discussion that must involve the entire
> >>> community; in fact before starting any work we should try to
> >>> converge
> >>> on a decision for the final repository structure in order to
> >>> minimize
> >>> the discomfort for the contributors that will be forced to rebase
> >>> their pending gerrit patches. Even if the full reorganization won't
> >>> happen in a short time I think we should plan the entire structure
> >>> now and then eventually move only few relevant modules to their
> >>> final
> >>> location.
> >>>
> >>> To start the discussion I'm attaching here a sketch of the vdsm
> >>> repository structure that I envision:
> >>>
> >>> .
> >>> |-- client
> >>> |   |-- [...]
> >>> |   `-- vdsClient.py
> >>> |-- common
> >>> |   |-- [...]
> >>> |   |-- betterPopen
> >>> |   |   `-- [...]
> >>> |   `-- vdsm
> >>> |       |-- [...]
> >>> |       `-- config.py
> >>> |-- contrib
> >>> |   |-- [...]
> >>> |   |-- nfs-check.py
> >>> |   `-- sos
> >>> |-- daemon
> >>> |   |-- [...]
> >>> |   |-- supervdsm.py
> >>> |   `-- vdsmd
> >>> `-- tool
> >>>      |-- [...]
> >>>      `-- vdsm-tool
> >> The schema file vdsmapi-schema.json (as well as the python module to
> >> parse it) are needed by the server and clients.  Initially I'd think it
> >> should be installed in 'common', but a client does not need things like
> >> betterPopen. Any recommendation on where the schema/API definition
> >> should live?
> > The problem is that betterPopen is misplaced as it's used only by the
> > daemon.
> >
> > Anyway let's not mix the three different aspects:
>  +1
> >
> > - repository structure
>  Well that's what is supposed to be the topic.
> > - installation location
> > - packaging
>  Those are supposed to be off topic considerint $SUBJECT
> >
> > That said in my opinion it can remain in "common" (which can be 
> > renamed to
> > "lib" as Ewoud suggested) and be installed in the site-lib *but* be
> > packaged only with the daemon (while we're waiting for it to become 
> > fully
> > independent and be moved out of the repo).
>  Where such things could be put in a folder 'extra' for now. (As an 
>  example)
> >
> > I suppose that the code dealing with the schema will be shared by 
> > both the
> > client and the daemon and it will be placed in "common" (or
> "lib") and
> > the schema itself can go there too.
> >
> > To be honest I started envisioning client/common/daemon/tool as 
> > pure-python
> > directories (as they are once installed) so maybe the schema could be
> > placed somewhere else with other miscellaneous files (./schema? 
> > ./config?),
> > but I wouldn't stress over this if it's impractical.
>  I would like to see also pure python directories.
>
>
>  A rough quick n dirty overview how I could imagine it to look like:
>
>  topdir/
>          build-aux/*
>          contrib/*
>          deployment/
>                      bootstrap/
>                      log/
>                      reg/
>          doc/
>              vdsm
>              api
>              cli
>          hooks/*
>          m4/*
>          tests/*
>          scripts/
>                  vdsm/*
>                  storage/*
>                  network/*
>          vdsm/
>                  api/
>                      json/
>                          jsonrpc/
>                          BindingJsonRpc.py
>                      xml/
>                          BindingXMLRPC.py
>                      Bridge.py
>                      vdsmapi-schema.json
>                      process-schema.py
>                      vdsmapi.py
>                  cli/
>                      vdsClient.py
>                      VdsClientGluster.py
>                  extra/
>                      betterPopen/
>                  shared/
>                      vdsm/
>                          SecureXMLRPCServer.py
>                          config.py.in
>                          constants.py.in
>                          utils.py
>                          vdscli.py.in*
>                          utils.py
>                          tool/
>                  daemon/
>                      storage/*
>                      network/*
>                      vdsmd/
>                          vdsm
>                          *
>  

I would like to make all of the service under vdsm/daemon directory only 
reference the files under vdsm/shared.  And back-reference from 
shared/vdsm to vdsm/daemon is not allowed.  In current VDSM repository, 
the files under vdsm/storage and the files under vdsm are 
cross-referenced on each other,  that is a disaster to the effort trying 
to make stand-alone service from storage  service outside of vdsmd.  In 
addition, the files under vdsm/cli should also only depend on the files 
under vdsm/shared.

BTW: Another service under vdsm/daemon should be supervdsm, so the 
daemon should like:
...
    daemon/
      storage/*
      network/*
      vdsmd/
      supervdsmd/
...


>  That said, I need to assert that I am not yet fully aware of all 
>  connections between the files and I guess the maintainers will know 
>  best about their respective parts of responsibility.
>
>  I'd like to add that it'd be probably a good idea to start having 
>  network and storage split off the main daemon for maintenance reasons, 
>  even if they interface each other.
>  I personally even could see a scenario where network and storage are 
>  working as standalone daemons. However that's off topic for know I guess.
>  

-- 
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: [phone]  Tieline: 9051626 E-mail: shuming(a)cn.ibm.com or
shuming(a)linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193,
PRC

----- Original Message -----
>  From: "Adam Litke" <agl(a)us.ibm.com&gt;
>  To: "Federico Simoncelli" <fsimonce(a)redhat.com&gt;
>  Cc: "VDSM Project Development" <vdsm-devel(a)lists.fedorahosted.org&gt;,
> "Dan Kenigsberg" <danken(a)redhat.com&gt;, "Saggi
>  Mizrahi" <smizrahi(a)redhat.com&gt;, "Vinzenz Feenstra"
> <vfeenstr(a)redhat.com&gt;, "Ayal Baron" <abaron(a)redhat.com&gt;
>  Sent: Tuesday, February 12, 2013 3:08:09 PM
>  Subject: Re: VDSM Repository Reorganization
>  
>  On Mon, Feb 11, 2013 at 12:17:39PM -0500, Federico Simoncelli wrote:
>  > It is some time now that we are discussing an eventual repository
>  > reorganization for vdsm. In fact I'm sure that we all experienced
>  > at least once the discomfort of having several modules scattered
>  > around the tree.
>  > The main goal of the reorganization would be to place the modules
>  > in their proper location so that they can be used (imported)
>  > without
>  > any special change (or hack) even when the code is executed inside
>  > the development repository (e.g. tests).
>  > 
>  > Recently there has been an initial proposal about moving some of
>  > these modules:
>  > 
>  > http://gerrit.ovirt.org/#/c/11858/
>  > 
>  > That spawned an interesting discussion that must involve the entire
>  > community; in fact before starting any work we should try to
>  > converge
>  > on a decision for the final repository structure in order to
>  > minimize
>  > the discomfort for the contributors that will be forced to rebase
>  > their pending gerrit patches. Even if the full reorganization won't
>  > happen in a short time I think we should plan the entire structure
>  > now and then eventually move only few relevant modules to their
>  > final
>  > location.
>  > 
>  > To start the discussion I'm attaching here a sketch of the vdsm
>  > repository structure that I envision:
>  > 
>  > .
>  > |-- client
>  > |   |-- [...]
>  > |   `-- vdsClient.py
>  > |-- common
>  > |   |-- [...]
>  > |   |-- betterPopen
>  > |   |   `-- [...]
>  > |   `-- vdsm
>  > |       |-- [...]
>  > |       `-- config.py
>  > |-- contrib
>  > |   |-- [...]
>  > |   |-- nfs-check.py
>  > |   `-- sos
>  > |-- daemon
>  > |   |-- [...]
>  > |   |-- supervdsm.py
>  > |   `-- vdsmd
>  > `-- tool
>  >     |-- [...]
>  >     `-- vdsm-tool
>  
>  The schema file vdsmapi-schema.json (as well as the python module to
>  parse it)
>  are needed by the server and clients.  Initially I'd think it should
>  be
>  installed in 'common', but a client does not need things like
>  betterPopen.  Any
>  recommendation on where the schema/API definition should live? 
Well they both
should have the file but when installed both should have
their own version of the file depending on the version installed of the
client or the server. This is so that vdsm-cli doesn't depend on vdsm
or vice-versa. You can't have them share the file since if one is installed
with a version of the schema where the schema syntax changed the client\server
will fail to parse the schema.

As for development, I think the least bad solution is to put it in contrib
with symlinks that have relative paths.
|--daemon
|  |-- [...]
|  `-- vdsmapi-schema.json -> ../contrib/vdsmapi-schema.json
|--client
|  |-- [...]
|  `-- vdsmapi-schema.json -> ../contrib/vdsmapi-schema.json
|--contrib
|  |-- [...]
|  `-- vdsmapi-schema.json
:
.

Git knows how to handle symlinks and symlinks are relative to the location of
the symlink.

We could also just select the daemon or the client folder and put the real
file there and a symlink in the other but I feel it's like choosing which
one of your children is the main user of a schema file.
>  
>  --
>  Adam Litke <agl(a)us.ibm.com&gt;
>  IBM Linux Technology Center
>  
>

[Federico Simoncelli]

----- Original Message -----
>  From: "Saggi Mizrahi" <smizrahi(a)redhat.com&gt;
>  To: "Adam Litke" <agl(a)us.ibm.com&gt;
>  Cc: "VDSM Project Development" <vdsm-devel(a)lists.fedorahosted.org&gt;,
> "Dan Kenigsberg" <danken(a)redhat.com&gt;, "Vinzenz
>  Feenstra" <vfeenstr(a)redhat.com&gt;, "Ayal Baron"
> <abaron(a)redhat.com&gt;, "Federico Simoncelli" <fsimonce(a)redhat.com&gt;
>  Sent: Monday, February 18, 2013 8:50:30 PM
>  Subject: Re: VDSM Repository Reorganization
>  
>  ----- Original Message -----
>  > From: "Adam Litke" <agl(a)us.ibm.com&gt;
>  > To: "Federico Simoncelli" <fsimonce(a)redhat.com&gt;
>  > Cc: "VDSM Project Development" <vdsm-devel(a)lists.fedorahosted.org&gt;,
>  > "Dan Kenigsberg" <danken(a)redhat.com&gt;, "Saggi
>  > Mizrahi" <smizrahi(a)redhat.com&gt;, "Vinzenz Feenstra"
>  > <vfeenstr(a)redhat.com&gt;, "Ayal Baron" <abaron(a)redhat.com&gt;
>  > Sent: Tuesday, February 12, 2013 3:08:09 PM
>  > Subject: Re: VDSM Repository Reorganization
>  > 
>  > On Mon, Feb 11, 2013 at 12:17:39PM -0500, Federico Simoncelli
>  > wrote:
>  > > It is some time now that we are discussing an eventual repository
>  > > reorganization for vdsm. In fact I'm sure that we all experienced
>  > > at least once the discomfort of having several modules scattered
>  > > around the tree.
>  > > The main goal of the reorganization would be to place the modules
>  > > in their proper location so that they can be used (imported)
>  > > without
>  > > any special change (or hack) even when the code is executed
>  > > inside
>  > > the development repository (e.g. tests).
>  > > 
>  > > Recently there has been an initial proposal about moving some of
>  > > these modules:
>  > > 
>  > > http://gerrit.ovirt.org/#/c/11858/
>  > > 
>  > > That spawned an interesting discussion that must involve the
>  > > entire
>  > > community; in fact before starting any work we should try to
>  > > converge
>  > > on a decision for the final repository structure in order to
>  > > minimize
>  > > the discomfort for the contributors that will be forced to rebase
>  > > their pending gerrit patches. Even if the full reorganization
>  > > won't
>  > > happen in a short time I think we should plan the entire
>  > > structure
>  > > now and then eventually move only few relevant modules to their
>  > > final
>  > > location.
>  > > 
>  > > To start the discussion I'm attaching here a sketch of the vdsm
>  > > repository structure that I envision:
>  > > 
>  > > .
>  > > |-- client
>  > > |   |-- [...]
>  > > |   `-- vdsClient.py
>  > > |-- common
>  > > |   |-- [...]
>  > > |   |-- betterPopen
>  > > |   |   `-- [...]
>  > > |   `-- vdsm
>  > > |       |-- [...]
>  > > |       `-- config.py
>  > > |-- contrib
>  > > |   |-- [...]
>  > > |   |-- nfs-check.py
>  > > |   `-- sos
>  > > |-- daemon
>  > > |   |-- [...]
>  > > |   |-- supervdsm.py
>  > > |   `-- vdsmd
>  > > `-- tool
>  > >     |-- [...]
>  > >     `-- vdsm-tool
>  > 
>  > The schema file vdsmapi-schema.json (as well as the python module
>  > to parse it)
>  > are needed by the server and clients.  Initially I'd think it
>  > should be
>  > installed in 'common', but a client does not need things like
>  > betterPopen.  Any
>  > recommendation on where the schema/API definition should live?
>  
>  Well they both should have the file but when installed both should have
>  their own version of the file depending on the version installed of the
>  client or the server. This is so that vdsm-cli doesn't depend on vdsm
>  or vice-versa. You can't have them share the file since if one is installed
>  with a version of the schema where the schema syntax changed the client\server
>  will fail to parse the schema. 
I'm not sure what's the purpose of having different versions of the
client/server on the same machine. The software repository is one and
it should provide both (as they're built from the same source).
This is the standard way of delivering client/server applications in
all the distributions. We can change that but we must have a good reason.

>  As for development, I think the least bad solution is to put it in
> contrib
>  with symlinks that have relative paths.
>  
>  |--daemon
>  |  |-- [...]
>  |  `-- vdsmapi-schema.json -> ../contrib/vdsmapi-schema.json
>  |--client
>  |  |-- [...]
>  |  `-- vdsmapi-schema.json -> ../contrib/vdsmapi-schema.json
>  |--contrib
>  |  |-- [...]
>  |  `-- vdsmapi-schema.json
>  :
>  .
>  
>  Git knows how to handle symlinks and symlinks are relative to the
>  location of the symlink. 
+1.

-- 
Federico

----- Original Message -----
>  From: "Federico Simoncelli" <fsimonce(a)redhat.com&gt;
>  To: "Saggi Mizrahi" <smizrahi(a)redhat.com&gt;
>  Cc: "VDSM Project Development" <vdsm-devel(a)lists.fedorahosted.org&gt;,
> "Dan Kenigsberg" <danken(a)redhat.com&gt;, "Vinzenz
>  Feenstra" <vfeenstr(a)redhat.com&gt;, "Ayal Baron"
> <abaron(a)redhat.com&gt;, "Adam Litke" <agl(a)us.ibm.com&gt;
>  Sent: Tuesday, February 19, 2013 11:27:59 AM
>  Subject: Re: VDSM Repository Reorganization
>  
>  ----- Original Message -----
>  > From: "Saggi Mizrahi" <smizrahi(a)redhat.com&gt;
>  > To: "Adam Litke" <agl(a)us.ibm.com&gt;
>  > Cc: "VDSM Project Development" <vdsm-devel(a)lists.fedorahosted.org&gt;,
>  > "Dan Kenigsberg" <danken(a)redhat.com&gt;, "Vinzenz
>  > Feenstra" <vfeenstr(a)redhat.com&gt;, "Ayal Baron"
> <abaron(a)redhat.com&gt;,
>  > "Federico Simoncelli" <fsimonce(a)redhat.com&gt;
>  > Sent: Monday, February 18, 2013 8:50:30 PM
>  > Subject: Re: VDSM Repository Reorganization
>  > 
>  > ----- Original Message -----
>  > > From: "Adam Litke" <agl(a)us.ibm.com&gt;
>  > > To: "Federico Simoncelli" <fsimonce(a)redhat.com&gt;
>  > > Cc: "VDSM Project Development"
>  > > <vdsm-devel(a)lists.fedorahosted.org&gt;,
>  > > "Dan Kenigsberg" <danken(a)redhat.com&gt;, "Saggi
>  > > Mizrahi" <smizrahi(a)redhat.com&gt;, "Vinzenz Feenstra"
>  > > <vfeenstr(a)redhat.com&gt;, "Ayal Baron" <abaron(a)redhat.com&gt;
>  > > Sent: Tuesday, February 12, 2013 3:08:09 PM
>  > > Subject: Re: VDSM Repository Reorganization
>  > > 
>  > > On Mon, Feb 11, 2013 at 12:17:39PM -0500, Federico Simoncelli
>  > > wrote:
>  > > > It is some time now that we are discussing an eventual
>  > > > repository
>  > > > reorganization for vdsm. In fact I'm sure that we all
>  > > > experienced
>  > > > at least once the discomfort of having several modules
>  > > > scattered
>  > > > around the tree.
>  > > > The main goal of the reorganization would be to place the
>  > > > modules
>  > > > in their proper location so that they can be used (imported)
>  > > > without
>  > > > any special change (or hack) even when the code is executed
>  > > > inside
>  > > > the development repository (e.g. tests).
>  > > > 
>  > > > Recently there has been an initial proposal about moving some
>  > > > of
>  > > > these modules:
>  > > > 
>  > > > http://gerrit.ovirt.org/#/c/11858/
>  > > > 
>  > > > That spawned an interesting discussion that must involve the
>  > > > entire
>  > > > community; in fact before starting any work we should try to
>  > > > converge
>  > > > on a decision for the final repository structure in order to
>  > > > minimize
>  > > > the discomfort for the contributors that will be forced to
>  > > > rebase
>  > > > their pending gerrit patches. Even if the full reorganization
>  > > > won't
>  > > > happen in a short time I think we should plan the entire
>  > > > structure
>  > > > now and then eventually move only few relevant modules to their
>  > > > final
>  > > > location.
>  > > > 
>  > > > To start the discussion I'm attaching here a sketch of the vdsm
>  > > > repository structure that I envision:
>  > > > 
>  > > > .
>  > > > |-- client
>  > > > |   |-- [...]
>  > > > |   `-- vdsClient.py
>  > > > |-- common
>  > > > |   |-- [...]
>  > > > |   |-- betterPopen
>  > > > |   |   `-- [...]
>  > > > |   `-- vdsm
>  > > > |       |-- [...]
>  > > > |       `-- config.py
>  > > > |-- contrib
>  > > > |   |-- [...]
>  > > > |   |-- nfs-check.py
>  > > > |   `-- sos
>  > > > |-- daemon
>  > > > |   |-- [...]
>  > > > |   |-- supervdsm.py
>  > > > |   `-- vdsmd
>  > > > `-- tool
>  > > >     |-- [...]
>  > > >     `-- vdsm-tool
>  > > 
>  > > The schema file vdsmapi-schema.json (as well as the python module
>  > > to parse it)
>  > > are needed by the server and clients.  Initially I'd think it
>  > > should be
>  > > installed in 'common', but a client does not need things like
>  > > betterPopen.  Any
>  > > recommendation on where the schema/API definition should live?
>  > 
>  > Well they both should have the file but when installed both should
>  > have
>  > their own version of the file depending on the version installed of
>  > the
>  > client or the server. This is so that vdsm-cli doesn't depend on
>  > vdsm
>  > or vice-versa. You can't have them share the file since if one is
>  > installed
>  > with a version of the schema where the schema syntax changed the
>  > client\server
>  > will fail to parse the schema.
>  
>  I'm not sure what's the purpose of having different versions of the
>  client/server on the same machine. The software repository is one and
>  it should provide both (as they're built from the same source).
>  This is the standard way of delivering client/server applications in
>  all the distributions. We can change that but we must have a good
>  reason. 
There isn't really a reason. But, as I said, you don't want them
to
depend on each other or have the schema in it's own rpm.
This means that you have to distribute them separately.

I also want to allow to update the client on a host without updating
the server. This is because you may want to have a script that works
across the cluster but not update all the hosts.

Now, even though you will use only old methods, the schema itself
might become unparsable by old implementations.

>  
>  > As for development, I think the least bad solution is to put it in
>  > contrib
>  > with symlinks that have relative paths.
>  > 
>  > |--daemon
>  > |  |-- [...]
>  > |  `-- vdsmapi-schema.json -> ../contrib/vdsmapi-schema.json
>  > |--client
>  > |  |-- [...]
>  > |  `-- vdsmapi-schema.json -> ../contrib/vdsmapi-schema.json
>  > |--contrib
>  > |  |-- [...]
>  > |  `-- vdsmapi-schema.json
>  > :
>  > .
>  > 
>  > Git knows how to handle symlinks and symlinks are relative to the
>  > location of the symlink.
>  
>  +1.
>  
>  --
>  Federico
>

On Tue, Feb 19, 2013 at 03:53:46PM -0500, Saggi Mizrahi wrote:
>  > I'm not sure what's the purpose of having different
> versions of the
>  > client/server on the same machine. The software repository is one and
>  > it should provide both (as they're built from the same source).
>  > This is the standard way of delivering client/server applications in
>  > all the distributions. We can change that but we must have a good
>  > reason.
>  There isn't really a reason. But, as I said, you don't want them to
>  depend on each other or have the schema in it's own rpm.
>  This means that you have to distribute them separately.
>  
>  I also want to allow to update the client on a host without updating
>  the server. This is because you may want to have a script that works
>  across the cluster but not update all the hosts.
>  
>  Now, even though you will use only old methods, the schema itself
>  might become unparsable by old implementations. 
This should never happen.  Right now each "symbol" in the schema is represented
by a single OrderedDict and the parsing code just loads the schema file into the
a list of these dicts.  Once loaded, the vdsmapi module categorizes symbols
according to the top-level keys.  Unrecognized symbol types are simply skipped.

-- 
Adam Litke <agl(a)us.ibm.com&gt;
IBM Linux Technology Center

In order to make progress on the file reorg, I want to summarize the discussion
and propose that a consensus has been reached regarding placement of the schema
file.

The current code has a routine find_schema() that can locate the schema file in
the development source tree or in an installed location.  Therefore, it only
needs to appear in the source tree in a single location and we will not need any
symlinks for this purpose.  Recently, the API handling code (schema and parsing
module) have been split into their own rpm.  This should solve the installation
problem since any entity that needs access to the schema and parser should
simply install the vdsm-api rpm.

-- 
Adam Litke <agl(a)us.ibm.com&gt;
IBM Linux Technology Center