[RFC] RPM's Python dependency generator

[Igor Gnatenko]

Hi,

in short, it reads egg metadata and can generate Provides (which we
already do now), Requires (which I want to talk about) and Recommends
(which I don't care atm).

Let's take simple package -- aiohttp.
https://bugzilla.redhat.com/show_bug.cgi?id=1381750

As you can see, since some version multidict requirement was bumped to
> = 2.0 and async_timeout requirement was added. Currently we specify

all requirements during initial package and usually without versions
which is breaking after some time.

So, let's try it (I will skip unrelated parts).
Before:
python(abi) = 3.5
python3-chardet
python3-multidict
After:
python(abi) = 3.5
python3.5dist(async-timeout)
python3.5dist(chardet)
python3.5dist(multidict) >= 2.0

Without more complicated packages (MNE, nipy, nilearn, moss) it's
getting much more harder since I have there 10+ deps.

We can't really track all changes in upstream code, so if we will
enable dependency generator, it will do this work for us. Note that we
can't just enable it in RPM, because it will break a lot of packages
due to:
* missing requires in egg metadata
* extra requires in egg metadata (e.g. windows-modules)

I would propose plan:
1. Create macro which will enable/disable depgen (e.g %python_enable_depgen)
2. Start enabling depgen and porting things (somehow reuse
portingdb.xyz probably?) and submitting patches upstream
3. In 1-2 releases I hope we can use it for major amount of packages
4. Enable depgen by default in RPM, add disabling depgen for remaining packages

Neal, you can share how Mageia did that as well and feel free to comment this ;)

Thoughts?

[Tomas Orsava]

On 11/30/2016 08:04 AM, Igor Gnatenko wrote:
> Hi,
>
> in short, it reads egg metadata and can generate Provides (which we
> already do now), Requires (which I want to talk about) and Recommends
> (which I don't care atm).
>
> Let's take simple package -- aiohttp.
> https://bugzilla.redhat.com/show_bug.cgi?id=1381750
>
> As you can see, since some version multidict requirement was bumped to
> > = 2.0 and async_timeout requirement was added. Currently we specify
>
> all requirements during initial package and usually without versions
> which is breaking after some time.
>
> So, let's try it (I will skip unrelated parts).
> Before:
> python(abi) = 3.5
> python3-chardet
> python3-multidict
> After:
> python(abi) = 3.5
> python3.5dist(async-timeout)
> python3.5dist(chardet)
> python3.5dist(multidict) >= 2.0
>
> Without more complicated packages (MNE, nipy, nilearn, moss) it's
> getting much more harder since I have there 10+ deps.
>
> We can't really track all changes in upstream code, so if we will
> enable dependency generator, it will do this work for us. Note that we
> can't just enable it in RPM, because it will break a lot of packages
> due to:
> missing requires in egg metadata
> extra requires in egg metadata (e.g. windows-modules)
>
> I would propose plan:
> Create macro which will enable/disable depgen (e.g %python_enable_depgen)

That would be cool!
> Start enabling depgen and porting things (somehow reuse
>
> portingdb.xyz probably?) and submitting patches upstream

While portingdb is open source and you can definitely set up a new 
instance, I think that would be an overkill. Whether a package supports 
Python 3 or not is highly relevant to other packages, since they depend 
on one another.
However, whether a package manually lists its Requires or uses a script 
to do that is not relevant to other packages, therefore I don't see the 
benefit of a portingdb portal here.
> In 1-2 releases I hope we can use it for major amount of packages
> Enable depgen by default in RPM, add disabling depgen for remaining packages

I don't think the depgen should be enabled by default, at least not in 
the foreseeable future. IIRC it's not that well implemented—e.g. I 
believe it doesn't read requirements.txt for example (but I might be wrong).
There will be a lot of cases where the generated requirements are 
incomplete, or contain unnecessary entries, etc. I think it should 
remain an opt-in.

Regards,
Tomas

[Neal Gompa]

On Wed, Nov 30, 2016 at 8:41 AM, Tomas Orsava  wrote:
> I don't think the depgen should be enabled by default, at least not in the
> foreseeable future. IIRC it's not that well implemented—e.g. I believe it
> doesn't read requirements.txt for example (but I might be wrong).
> There will be a lot of cases where the generated requirements are
> incomplete, or contain unnecessary entries, etc. I think it should remain an
> opt-in.

According to various Python people, we're not actually supposed to
read requirements.txt. That file is explicitly designed for people to
individualized deployments. The proper place to get this information
is from the egg-info/dist-info data, which is what we read. The fact
that some people abuse requirements.txt and have it read in by their
setup.py is beside the point. Whatever the setup.py (thus
pip/easy_install/etc.) says it needs, the generator will dutifully
report.

[Tomas Orsava]

On 11/30/2016 02:44 PM, Neal Gompa wrote:
> On Wed, Nov 30, 2016 at 8:41 AM, Tomas Orsava  wrote:
> > I don't think the depgen should be enabled by default, at least not in the
> > foreseeable future. IIRC it's not that well implemented—e.g. I believe it
> > doesn't read requirements.txt for example (but I might be wrong).
> > There will be a lot of cases where the generated requirements are
> > incomplete, or contain unnecessary entries, etc. I think it should remain an
> > opt-in.
>
> According to various Python people, we're not actually supposed to
> read requirements.txt. That file is explicitly designed for people to
> individualized deployments. The proper place to get this information
> is from the egg-info/dist-info data, which is what we read. The fact
> that some people abuse requirements.txt and have it read in by their
> setup.py is beside the point. Whatever the setup.py (thus
> pip/easy_install/etc.) says it needs, the generator will dutifully
> report.

The fact remains in too many cases it will need to be manually adjusted, 
it won't be foolproof.
Therefore I argue it would be better for it to be an opt-in so that new 
packagers don't immediately have to jump in into debugging a depgen they 
have no clue how really works.

[Igor Gnatenko]

On Wed, Nov 30, 2016 at 2:53 PM, Tomas Orsava  wrote:
> On 11/30/2016 02:44 PM, Neal Gompa wrote:
> > On Wed, Nov 30, 2016 at 8:41 AM, Tomas Orsava  wrote:
> > > I don't think the depgen should be enabled by default, at least not in
> > > the
> > > foreseeable future. IIRC it's not that well implemented—e.g. I believe it
> > > doesn't read requirements.txt for example (but I might be wrong).
> > > There will be a lot of cases where the generated requirements are
> > > incomplete, or contain unnecessary entries, etc. I think it should remain
> > > an
> > > opt-in.
> >
> > According to various Python people, we're not actually supposed to
> > read requirements.txt. That file is explicitly designed for people to
> > individualized deployments. The proper place to get this information
> > is from the egg-info/dist-info data, which is what we read. The fact
> > that some people abuse requirements.txt and have it read in by their
> > setup.py is beside the point. Whatever the setup.py (thus
> > pip/easy_install/etc.) says it needs, the generator will dutifully
> > report.
>
> The fact remains in too many cases it will need to be manually adjusted, it
> won't be foolproof.
> Therefore I argue it would be better for it to be an opt-in so that new
> packagers don't immediately have to jump in into debugging a depgen they
> have no clue how really works.

We'll see how it will go. we have depgen for pkgconfig, libraries,
etc. for many years and people don't go and debug it immediately, but
for many of packages it will help a lot. Anyhow, we'll see after
couple of releases.

Neal suggested to have:
%__python_requires
%{_rpmconfigdir}/%{?pythondistdeps_enable:pythondistdeps.py}%{!?pythondistdeps_enable:pythondeps.sh}
--requires
in python.attr inside RPM.

I tested it and it just works once I specify `%global
pythondistdeps_enable 1` in spec. Can you help me to get this
included? With RPM part it's clear how to get this, but updating
guidelines and other stuff...
> python-devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]

[Neal Gompa]

On Thu, Dec 1, 2016 at 8:36 AM, Igor Gnatenko  wrote:
> On Wed, Nov 30, 2016 at 2:53 PM, Tomas Orsava  wrote:
> > On 11/30/2016 02:44 PM, Neal Gompa wrote:
> > > On Wed, Nov 30, 2016 at 8:41 AM, Tomas Orsava  wrote:
> > > > I don't think the depgen should be enabled by default, at least not in
> > > > the
> > > > foreseeable future. IIRC it's not that well implemented—e.g. I believe it
> > > > doesn't read requirements.txt for example (but I might be wrong).
> > > > There will be a lot of cases where the generated requirements are
> > > > incomplete, or contain unnecessary entries, etc. I think it should remain
> > > > an
> > > > opt-in.
> > >
> > > According to various Python people, we're not actually supposed to
> > > read requirements.txt. That file is explicitly designed for people to
> > > individualized deployments. The proper place to get this information
> > > is from the egg-info/dist-info data, which is what we read. The fact
> > > that some people abuse requirements.txt and have it read in by their
> > > setup.py is beside the point. Whatever the setup.py (thus
> > > pip/easy_install/etc.) says it needs, the generator will dutifully
> > > report.
> >
> > The fact remains in too many cases it will need to be manually adjusted, it
> > won't be foolproof.
> > Therefore I argue it would be better for it to be an opt-in so that new
> > packagers don't immediately have to jump in into debugging a depgen they
> > have no clue how really works.
>
> We'll see how it will go. we have depgen for pkgconfig, libraries,
> etc. for many years and people don't go and debug it immediately, but
> for many of packages it will help a lot. Anyhow, we'll see after
> couple of releases.
>
> Neal suggested to have:
> %__python_requires
> %{_rpmconfigdir}/%{?pythondistdeps_enable:pythondistdeps.py}%{!?pythondistdeps_enable:pythondeps.sh}
> --requires
> in python.attr inside RPM.
>
> I tested it and it just works once I specify `%global
> pythondistdeps_enable 1` in spec. Can you help me to get this
> included? With RPM part it's clear how to get this, but updating
> guidelines and other stuff...

This will also drastically simplify the work of tools like pyp2rpm,
since instead of having to do crazy processing of module names, it can
just use the appropriate pypi provides/requires. If the macro template
enables the requires generator, it won't even need to specify requires
at all, as they'll be generated from the wheel data anyway.

[Petr Viktorin]

On 12/01/2016 02:42 PM, Neal Gompa wrote:
> On Thu, Dec 1, 2016 at 8:36 AM, Igor Gnatenko  wrote:
> > On Wed, Nov 30, 2016 at 2:53 PM, Tomas Orsava  wrote:
> > > On 11/30/2016 02:44 PM, Neal Gompa wrote:
> > > > On Wed, Nov 30, 2016 at 8:41 AM, Tomas Orsava  wrote:
> > > > > I don't think the depgen should be enabled by default, at least not in
> > > > > the
> > > > > foreseeable future. IIRC it's not that well implemented—e.g. I believe it
> > > > > doesn't read requirements.txt for example (but I might be wrong).
> > > > > There will be a lot of cases where the generated requirements are
> > > > > incomplete, or contain unnecessary entries, etc. I think it should remain
> > > > > an
> > > > > opt-in.
> > > >
> > > > According to various Python people, we're not actually supposed to
> > > > read requirements.txt. That file is explicitly designed for people to
> > > > individualized deployments. The proper place to get this information
> > > > is from the egg-info/dist-info data, which is what we read. The fact
> > > > that some people abuse requirements.txt and have it read in by their
> > > > setup.py is beside the point. Whatever the setup.py (thus
> > > > pip/easy_install/etc.) says it needs, the generator will dutifully
> > > > report.
> > >
> > > The fact remains in too many cases it will need to be manually adjusted, it
> > > won't be foolproof.
> > > Therefore I argue it would be better for it to be an opt-in so that new
> > > packagers don't immediately have to jump in into debugging a depgen they
> > > have no clue how really works.
> >
> > We'll see how it will go. we have depgen for pkgconfig, libraries,
> > etc. for many years and people don't go and debug it immediately, but
> > for many of packages it will help a lot. Anyhow, we'll see after
> > couple of releases.
> >
> > Neal suggested to have:
> > %__python_requires
> > %{_rpmconfigdir}/%{?pythondistdeps_enable:pythondistdeps.py}%{!?pythondistdeps_enable:pythondeps.sh}
> > --requires
> > in python.attr inside RPM.
> >
> > I tested it and it just works once I specify `%global
> > pythondistdeps_enable 1` in spec. Can you help me to get this
> > included? With RPM part it's clear how to get this, but updating
> > guidelines and other stuff...
>
> This will also drastically simplify the work of tools like pyp2rpm,
> since instead of having to do crazy processing of module names, it can
> just use the appropriate pypi provides/requires. If the macro template
> enables the requires generator, it won't even need to specify requires
> at all, as they'll be generated from the wheel data anyway.

+1, ideally that'll leave us with only one automatic dependency generator.

Problems with upstreams getting setup.py wrong should be treated as 
upstream bugs and treated accordingly: reported as pull requests, or, as 
the last resort, by a Fedora-specific patch to setup.py.

CCing Michal from the pyp2rpm project, so that everyone knows he's aware 
of the discussion here.

[Toshio Kuratomi]

On Dec 1, 2016 6:58 AM, "Petr Viktorin"  wrote:
> On 12/01/2016 02:42 PM, Neal Gompa wrote:
> > On Thu, Dec 1, 2016 at 8:36 AM, Igor Gnatenko

wrote:
> > > On Wed, Nov 30, 2016 at 2:53 PM, Tomas Orsava

wrote:
> > > > On 11/30/2016 02:44 PM, Neal Gompa wrote:
> > > > > On Wed, Nov 30, 2016 at 8:41 AM, Tomas Orsava

wrote:
> > > > > > I don't think the depgen should be enabled by default, at least not

in
> > > > > > the
> > > > > > foreseeable future. IIRC it's not that well implemented—e.g. I

believe it
> > > > > > doesn't read requirements.txt for example (but I might be wrong).
> > > > > > There will be a lot of cases where the generated requirements are
> > > > > > incomplete, or contain unnecessary entries, etc. I think it should

remain
> > > > > > an
> > > > > > opt-in.
> > > > >
> > > > > According to various Python people, we're not actually supposed to
> > > > > read requirements.txt. That file is explicitly designed for people to
> > > > > individualized deployments. The proper place to get this information
> > > > > is from the egg-info/dist-info data, which is what we read. The fact
> > > > > that some people abuse requirements.txt and have it read in by their
> > > > > setup.py is beside the point. Whatever the setup.py (thus
> > > > > pip/easy_install/etc.) says it needs, the generator will dutifully
> > > > > report.
> > > >
> > > > The fact remains in too many cases it will need to be manually

adjusted, it
> > > > won't be foolproof.
> > > > Therefore I argue it would be better for it to be an opt-in so that new
> > > > packagers don't immediately have to jump in into debugging a depgen

they
> > > > have no clue how really works.
> > >
> > > We'll see how it will go. we have depgen for pkgconfig, libraries,
> > > etc. for many years and people don't go and debug it immediately, but
> > > for many of packages it will help a lot. Anyhow, we'll see after
> > > couple of releases.
> > >
> > > Neal suggested to have:
> > > %__python_requires

%{_rpmconfigdir}/%{?pythondistdeps_enable:pythondistdeps.py}%{!?pythondistdeps_enable:pythondeps.sh}
> > > --requires
> > > in python.attr inside RPM.
> > >
> > > I tested it and it just works once I specify `%global
> > > pythondistdeps_enable 1` in spec. Can you help me to get this
> > > included? With RPM part it's clear how to get this, but updating
> > > guidelines and other stuff...
> >
> > This will also drastically simplify the work of tools like pyp2rpm,
> > since instead of having to do crazy processing of module names, it can
> > just use the appropriate pypi provides/requires. If the macro template
> > enables the requires generator, it won't even need to specify requires
> > at all, as they'll be generated from the wheel data anyway.
>
> +1, ideally that'll leave us with only one automatic dependency generator.
>
> Problems with upstreams getting setup.py wrong should be treated as

upstream bugs and treated accordingly: reported as pull requests, or, as
the last resort, by a Fedora-specific patch to setup.py.

Note that fedora will have to carry more patches to setup.py's (but
hopefully of limited durations and pretty trivial).  Upstream versioned
requirements often don't match with distro packages because we backport to
work around bugs in compatible manners while upstream simply requires newer
versions that have the bugs fixed.

Not a showstopper by any means; just something to be aware of.

-Toshio

[Nick Coghlan]

On 3 December 2016 at 05:40, Toshio Kuratomi  wrote:
> On Dec 1, 2016 6:58 AM, "Petr Viktorin"  wrote:
> > Problems with upstreams getting setup.py wrong should be treated as
> > upstream bugs and treated accordingly: reported as pull requests, or, as the
> > last resort, by a Fedora-specific patch to setup.py.
>
> Note that fedora will have to carry more patches to setup.py's (but
> hopefully of limited durations and pretty trivial).  Upstream versioned
> requirements often don't match with distro packages because we backport to
> work around bugs in compatible manners while upstream simply requires newer
> versions that have the bugs fixed.
>
> Not a showstopper by any means; just something to be aware of.

Depending on how deep people want to dive into that aspect, it may
even make sense to start using the "Local version identifier" feature
in PEP 440 to expose RPM release numbers to Python level tooling:
https://www.python.org/dev/peps/pep-0440/#local-version-identifiers

Cheers,
Nick.

[Tomas Orsava]

On 12/01/2016 02:36 PM, Igor Gnatenko wrote:
> We'll see how it will go. we have depgen for pkgconfig, libraries,
> etc. for many years and people don't go and debug it immediately, but
> for many of packages it will help a lot. Anyhow, we'll see after
> couple of releases.

Yeah, absolutely. When it's battle-tested and solid, it will make 
packaging that much easier.
> Neal suggested to have:
> %__python_requires
> %{_rpmconfigdir}/%{?pythondistdeps_enable:pythondistdeps.py}%{!?pythondistdeps_enable:pythondeps.sh}
> --requires
> in python.attr inside RPM.

Oh that's clever, I was wondering how things like that are done!
> I tested it and it just works once I specify `%global
> pythondistdeps_enable 1` in spec. Can you help me to get this
> included? With RPM part it's clear how to get this, but updating
> guidelines and other stuff...

AFAIK the best way to get the guidelines updated is to create an 
accompanying Fedora Change. [0]
I sadly don't have free cycles to take that on, as I'm currently 
involved in 2 upcoming Fedora Changes, nevertheless I will gladly 
provide any help and guidance you might need!

[0] https://fedoraproject.org/wiki/Changes/Policy

[Charalampos Stratakis]

Seems like a solid plan. Smaller details can be refined as well, but overall seems like a good direction.

Unfortunately at this time there aren't free cycles for the python-maint team members, so I doubt we would be able to push this for F26 as a system wide change on behalf of our team at least.

So if noone is up for chasing this for the current development cycle, I would suggest pushing it for F27.

Regards,

Charalampos Stratakis
Associate Software Engineer
Python Maintenance Team, Red Hat

----- Original Message -----
From: "Igor Gnatenko" 
To: "Fedora Python SIG" 
Sent: Thursday, December 1, 2016 2:36:48 PM
Subject: Re: [RFC] RPM's Python dependency generator

On Wed, Nov 30, 2016 at 2:53 PM, Tomas Orsava  wrote:
> On 11/30/2016 02:44 PM, Neal Gompa wrote:
> > On Wed, Nov 30, 2016 at 8:41 AM, Tomas Orsava  wrote:
> > > I don't think the depgen should be enabled by default, at least not in
> > > the
> > > foreseeable future. IIRC it's not that well implemented—e.g. I believe it
> > > doesn't read requirements.txt for example (but I might be wrong).
> > > There will be a lot of cases where the generated requirements are
> > > incomplete, or contain unnecessary entries, etc. I think it should remain
> > > an
> > > opt-in.
> >
> > According to various Python people, we're not actually supposed to
> > read requirements.txt. That file is explicitly designed for people to
> > individualized deployments. The proper place to get this information
> > is from the egg-info/dist-info data, which is what we read. The fact
> > that some people abuse requirements.txt and have it read in by their
> > setup.py is beside the point. Whatever the setup.py (thus
> > pip/easy_install/etc.) says it needs, the generator will dutifully
> > report.
>
> The fact remains in too many cases it will need to be manually adjusted, it
> won't be foolproof.
> Therefore I argue it would be better for it to be an opt-in so that new
> packagers don't immediately have to jump in into debugging a depgen they
> have no clue how really works.

We'll see how it will go. we have depgen for pkgconfig, libraries,
etc. for many years and people don't go and debug it immediately, but
for many of packages it will help a lot. Anyhow, we'll see after
couple of releases.

Neal suggested to have:
%__python_requires
%{_rpmconfigdir}/%{?pythondistdeps_enable:pythondistdeps.py}%{!?pythondistdeps_enable:pythondeps.sh}
--requires
in python.attr inside RPM.

I tested it and it just works once I specify `%global
pythondistdeps_enable 1` in spec. Can you help me to get this
included? With RPM part it's clear how to get this, but updating
guidelines and other stuff...
> python-devel mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]