Python Packages + Multiple Sources

[BJ Dierkes]

Hello all,

I've been trying to figure the best way to handle this situation for the better part
of two days... and so I figured I'd bring it on list and hopefully I can clarify this
all.  I have several projects that relate to this question, however I will reference
'cement' as an example.  Basically, cement is a CLI Application Framework for
Python [1].  I am also the upstream maintainer, so I have the ability to modify upstream
or downstream.  Currently, cement consists of three separate sources:

	cement
	cement.devtools
	cement.test

All three pieces follow each release meaning, when 0.8.12 (current stable) was released...
new tarbals were released for all three.  The reason for separate tarbals is primarily for
maintaining releases via PyPi [2].  I need all three pieces to be separate so that users
can 'easy_install cement', without pulling in a dozen dependencies for
cement.devtools or cement.test.  I don't have the luxury of creating
'subpackages' in PyPi, so I have to break up the sources.  

That said, in the Fedora world... it is a bit excessive to maintain three separate RPM
packages (for all Fedora/EPEL releases)... when all three packages are related and could
easily role up under a single package, with subpackages.  I have the same issue with
'rosendale' [3] which is/will be a set of plugins for applications built on
cement.  The same situation exists, in PyPi I need separate source tarbals because users
need to be able to 'easy_install rosendale.some_plugin'.  In Fedora world,
maintaining a single package set for 'rosendale' would be ideal and make more
sense because I can do sub packages for each plugin and not have to maintain 20 different
package sets.

What I would like to see is if this type of situation would lend itself to making an
exception to the FPG regarding 'one source per package'.  I assume the section
'Bundling of multiple projects' [4] is relevant, though it is pretty vague.  I
guess what I'm looking for is for someone with more time in the community to give some
advice on this situation.  Ideally, I would like to be able to maintain a single package
set for say 'cement', but with Source0 (cement), Source1 (cement.devtools),
Source2 (cement.test).

Or would it be recommended to have a separate tarbal like
'cement-all-0.8.12.tar.gz' which would include all parts of the project, and use
that as Source0?

Thanks for your time.

References:

[1] http://builtoncement.org
[2] http://pypi.python.org
[3]http://builtoncement.org/rosendale/1.0/doc/
[4] http://fedoraproject.org/wiki/PackagingGuidelines#Bundling_of_multiple_pr...

---
derks

[Jef Spaleta]

On Wed, Dec 8, 2010 at 2:39 AM, BJ Dierkes <wdierkes(a)5dollarwhitebox.org&gt; wrote:
>  Hello all,
>  
Just to be clear... PyPI has an implied "one source" requirement
embedded in its repository structure and you have optimized your
upstream project release structure to meet PyPI's implied requirement.

Question does PyPI handle dependency tracking?

If I easy_install cement.devtools   does cement get installed automagically?


-jef

[Stanislav Ochotnicky]

On 12/08/2010 04:25 AM, Jeff Spaleta wrote:
>  On Wed, Dec 8, 2010 at 2:39 AM, BJ Dierkes
> <wdierkes(a)5dollarwhitebox.org&gt; wrote:
> > Hello all,
> >
>  
>  Just to be clear... PyPI has an implied "one source" requirement
>  embedded in its repository structure and you have optimized your
>  upstream project release structure to meet PyPI's implied requirement.
>  
>  Question does PyPI handle dependency tracking?
>  
>  If I easy_install cement.devtools   does cement get installed automagically? 
Yes, setup function from python setuptools has named argument
'install_requires' that causes dependencies to be pulled in during
install time. Assuming cement.devtools has install_requires = ['cement']
this will work as you'd expect

-- 
Stanislav Ochotnicky <sochotnicky(a)redhat.com&gt;
Associate Software Engineer - Base Operating Systems Brno

PGP: 71A1677C
Red Hat Inc.                               http://cz.redhat.com

[BJ Dierkes]

On Dec 8, 2010, at 2:55 AM, Stanislav Ochotnicky wrote:

>  On 12/08/2010 04:25 AM, Jeff Spaleta wrote:
> > On Wed, Dec 8, 2010 at 2:39 AM, BJ Dierkes <wdierkes(a)5dollarwhitebox.org&gt;
> wrote:
> >> Hello all,
> >> 
> > 
> > Just to be clear... PyPI has an implied "one source" requirement
> > embedded in its repository structure and you have optimized your
> > upstream project release structure to meet PyPI's implied requirement.
> > 
> > Question does PyPI handle dependency tracking?
> > 
> > If I easy_install cement.devtools   does cement get installed automagically?
>  
>  Yes, setup function from python setuptools has named argument
>  'install_requires' that causes dependencies to be pulled in during
>  install time. Assuming cement.devtools has install_requires = ['cement']
>  this will work as you'd expect
>   
That is exactly right.  

---
derks

[Jef Spaleta]

On Wed, Dec 8, 2010 at 6:06 PM, BJ Dierkes <wdierkes(a)5dollarwhitebox.org&gt; wrote:
>  That is exactly right. 
reading over the instructions on the pypi page for cement.devtools
explicitly tells people to easy_install cement prior to
easy_install'ing cement.devtools, so I wanted clarification as to
whether that was necessasry.


-jef

[BJ Dierkes]

On Dec 8, 2010, at 1:53 PM, Jeff Spaleta wrote:

>  On Wed, Dec 8, 2010 at 6:06 PM, BJ Dierkes
> <wdierkes(a)5dollarwhitebox.org&gt; wrote:
> > That is exactly right.
>  
>  reading over the instructions on the pypi page for cement.devtools
>  explicitly tells people to easy_install cement prior to
>  easy_install'ing cement.devtools, so I wanted clarification as to
>  whether that was necessasry.
>   
I've updated this to be more accurate, thank you.

---
derks

[Till Maas]

On Tue, Dec 07, 2010 at 08:39:50PM -0600, BJ Dierkes wrote:

>  All three pieces follow each release meaning, when 0.8.12 (current
> stable) was released... new tarbals were released for all three.  The reason for separate
> tarbals is primarily for maintaining releases via PyPi [2].  I need all three pieces to be
> separate so that users can 'easy_install cement', without pulling in a dozen
> dependencies for cement.devtools or cement.test.  I don't have the luxury of creating
> 'subpackages' in PyPi, so I have to break up the sources.   
>  What I would like to see is if this type of situation would lend
> itself to making an exception to the FPG regarding 'one source per package'.  I
> assume the section 'Bundling of multiple projects' [4] is relevant, though it is
> pretty vague.  I guess what I'm looking for is for someone with more time in the
> community to give some advice on this situation.  Ideally, I would like to be able to
> maintain a single package set for say 'cement', but with Source0 (cement), Source1
> (cement.devtools), Source2 (cement.test). 
To quote the guidelines:
| Fedora packages should make every effort to avoid having multiple,
| separate, upstream projects bundled together in a single package.

Since all tarballs belong to the same upstream project, there is nothing
wrong with using all three in one SPEC.

>  Or would it be recommended to have a separate tarbal like
> 'cement-all-0.8.12.tar.gz' which would include all parts of the project, and use
> that as Source0? 
I do not see any additional value for Fedora when doing this.

Regards
Till

>  [4]
> http://fedoraproject.org/wiki/PackagingGuidelines#Bundling_of_multiple_pr...

[Toshio Kuratomi]

On Wed, Dec 08, 2010 at 09:29:10PM +0100, Till Maas wrote:
>  On Tue, Dec 07, 2010 at 08:39:50PM -0600, BJ Dierkes wrote:
>  
>  > All three pieces follow each release meaning, when 0.8.12 (current stable) was
> released... new tarbals were released for all three.  The reason for separate tarbals is
> primarily for maintaining releases via PyPi [2].  I need all three pieces to be separate
> so that users can 'easy_install cement', without pulling in a dozen dependencies
> for cement.devtools or cement.test.  I don't have the luxury of creating
> 'subpackages' in PyPi, so I have to break up the sources.  
>  
>  > What I would like to see is if this type of situation would lend itself to making an
> exception to the FPG regarding 'one source per package'.  I assume the section
> 'Bundling of multiple projects' [4] is relevant, though it is pretty vague.  I
> guess what I'm looking for is for someone with more time in the community to give some
> advice on this situation.  Ideally, I would like to be able to maintain a single package
> set for say 'cement', but with Source0 (cement), Source1 (cement.devtools),
> Source2 (cement.test).
>  
>  To quote the guidelines:
>  | Fedora packages should make every effort to avoid having multiple,
>  | separate, upstream projects bundled together in a single package.
>  
>  Since all tarballs belong to the same upstream project, there is nothing
>  wrong with using all three in one SPEC.
>  
>  > Or would it be recommended to have a separate tarbal like
> 'cement-all-0.8.12.tar.gz' which would include all parts of the project, and use
> that as Source0?
>  
>  I do not see any additional value for Fedora when doing this. 
Separate packages do have benefits for users.  Here's an example:

Let's say that I'm doing new development on cement-0.9 atm but someone
reports a security issue that needs attention on the current stable release
of cement.devtools-0.8.12.  If I was upstream, I'd be very tempted to only
release a new cement.devtools-0.8.12.1 and not bother with a cement-0.8.12.1
version (with no changes other than the version bump) since 1) I want to get
the security update out ASAP and 2) Why inconvenience my users with
downloading cement again when only users of cement.devtools need to do
something.

For packagers, whether the upstream decided to release cement-0.8.12.1 or
not, you have the same choice as long as you use separate srpms.  You can
say... hmmm... upstream's changelog for cement is "* Keep version number in
sync with cement.devtools release" I think I'll skip this update to cement
so that users don't have to download the new package for no benefit.  if you
use on srpm, a change to any of the tarballs is going to mean you need to
update for all of them.

So if you're going to draft a change to the Fedora Guideline, I'd phrase it
as "If you are the upstream for the package and you are always going to
release updates of the tarballs simultaneously (even if there's only changes
to some of the tarballs) you may include multiple tarballs in the same
package."  I don't know that I'm convinced that I'd vote for that but
I think that's the approach that would make the most sense.

-Toshio