Python RPM dependency generators and LegacyVersion, time to error?

[Miro Hrončok]

Hello Pythonistas.

After some recent improvements in the Python RPM dependency generators, a 
regression was discovered [0].

Turns out the error happened when the upstream metadata contained a requirement 
with a PEP 440 [1] incompatible version. A fix is pending and only one package 
in Fedora was affected.

However, this brings me to a question: What shall we do with such incompatible 
versions in the future? Implementation-wise, the generators use the 
pypa/packaging library to parse the requirements. When a PEP 440 incompatible 
version is found, it's parsed as a LegacyVersion [3]. A LegacyVersion has been 
deprecated for 1 year now and when found it might indicate a packaging problem.

I've experimented with a patched version of the generators that would error out 
right away when a LegacyVersion is parsed. I found the following categories of 
problems:

1) A requirement uses invalid/legacy version (2 packages):

python-celery: pytz>dev

This is a pip-compatible shortcut for 0.dev.0, which is only required as a hack 
for PEP 440 incompatible versions of pytz from before 2013 (2012a, 2012b, 
2012c...).

python-pvc: pyvmomi>=5.5.0-2014.1.1

This is an existing PEP 440 incompatible version of pyvmomi that has been since 
replaced with PEP 440 compatible version on PyPI.

2) The package itself has an invalid/legacy version (3 packages):

python-haversion: main
python-ipmi: unknown
python-lacrosse: unknown

This is most likely a packaging problem introduced by the Fedora packager by 
using a GitHub tarball whereas the version information is only stored in sdist. 
I believe the generators should error out immediately when this happens to 
prevent packaging errors, but there might be a need to allow it to proceed if 
we want to package e.g. pyvmomi 5.5.0-2014.1.1 or pytz 2012d.


tl;dr:

  - there might be cases where legacy versions are needed
  - but in most cases, they should be avoided

Hence, I propose we do the following in Rawhide:

We turn LegacyVersions to failures, but we provide a stop-gap measure, such as 
(%global python_dependency_allow_legacy_version_provides 1 /  %global 
python_dependency_allow_legacy_version_requires 1 ) for packages that need to 
override this. When pypa/packaging actually drops LegacyVersion, this stop-gap 
measure will no longer work.


[0] https://bugzilla.redhat.com/show_bug.cgi?id=2019954
[1] https://www.python.org/dev/peps/pep-0440/
[2] https://packaging.pypa.io/
[3] 
https://packaging.pypa.io/en/latest/version.html#packaging.version.Legacy...

-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Neal Gompa]

On Thu, Nov 4, 2021 at 7:50 AM Miro Hrončok <mhroncok(a)redhat.com&gt; wrote:
>  Hello Pythonistas.
>
>  After some recent improvements in the Python RPM dependency generators, a
>  regression was discovered [0].
>
>  Turns out the error happened when the upstream metadata contained a requirement
>  with a PEP 440 [1] incompatible version. A fix is pending and only one package
>  in Fedora was affected.
>
>  However, this brings me to a question: What shall we do with such incompatible
>  versions in the future? Implementation-wise, the generators use the
>  pypa/packaging library to parse the requirements. When a PEP 440 incompatible
>  version is found, it's parsed as a LegacyVersion [3]. A LegacyVersion has been
>  deprecated for 1 year now and when found it might indicate a packaging problem.
>
>  I've experimented with a patched version of the generators that would error out
>  right away when a LegacyVersion is parsed. I found the following categories of
>  problems:
>
>  1) A requirement uses invalid/legacy version (2 packages):
>
>  python-celery: pytz>dev
>
>  This is a pip-compatible shortcut for 0.dev.0, which is only required as a hack
>  for PEP 440 incompatible versions of pytz from before 2013 (2012a, 2012b,
>  2012c...).
>
>  python-pvc: pyvmomi>=5.5.0-2014.1.1
>
>  This is an existing PEP 440 incompatible version of pyvmomi that has been since
>  replaced with PEP 440 compatible version on PyPI.
>
>  2) The package itself has an invalid/legacy version (3 packages):
>
>  python-haversion: main
>  python-ipmi: unknown
>  python-lacrosse: unknown
>
>  This is most likely a packaging problem introduced by the Fedora packager by
>  using a GitHub tarball whereas the version information is only stored in sdist.
>  I believe the generators should error out immediately when this happens to
>  prevent packaging errors, but there might be a need to allow it to proceed if
>  we want to package e.g. pyvmomi 5.5.0-2014.1.1 or pytz 2012d.
>
>
>  tl;dr:
>
>    - there might be cases where legacy versions are needed
>    - but in most cases, they should be avoided
>
>  Hence, I propose we do the following in Rawhide:
>
>  We turn LegacyVersions to failures, but we provide a stop-gap measure, such as
>  (%global python_dependency_allow_legacy_version_provides 1 /  %global
>  python_dependency_allow_legacy_version_requires 1 ) for packages that need to
>  override this. When pypa/packaging actually drops LegacyVersion, this stop-gap
>  measure will no longer work.
>
>
>  [0] https://bugzilla.redhat.com/show_bug.cgi?id=2019954
>  [1] https://www.python.org/dev/peps/pep-0440/
>  [2] https://packaging.pypa.io/
>  [3]
>  https://packaging.pypa.io/en/latest/version.html#packaging.version.Legacy...
>  
This makes sense to me, but would it also make sense to have some
documentation about this case and how to resolve it to PEP 440
compatible versions, too?



-- 
真実はいつも一つ！/ Always, there's only one truth!

[Miro Hrončok]

On 04. 11. 21 13:42, Neal Gompa wrote:
>  On Thu, Nov 4, 2021 at 7:50 AM Miro Hrončok
> <mhroncok(a)redhat.com&gt; wrote:
> > ...
> > Hence, I propose we do the following in Rawhide:
> >
> > We turn LegacyVersions to failures, but we provide a stop-gap measure, such as
> > (%global python_dependency_allow_legacy_version_provides 1 /  %global
> > python_dependency_allow_legacy_version_requires 1 ) for packages that need to
> > override this. When pypa/packaging actually drops LegacyVersion, this stop-gap
> > measure will no longer work.
>  
>  This makes sense to me, but would it also make sense to have some
>  documentation about this case and how to resolve it to PEP 440
>  compatible versions, too? 
I would definitively create a description of this in the guidelines and we 
could link the error to it. If we want some distro-agnostic place, we could 
create a document in 
https://github.com/rpm-software-management/python-rpm-packaging

The description should definitively contain migration advice and some 
actionable recommendations. However, there is no deterministic way to resolve a 
PEP 440 incompatible version to a PEP 440 compatible version, otherwise we 
would just do that.

Consider that PEP 440 incompatible version can be practically anything within a 
reasonable character set.


-- 
Miro Hrončok
-- 
Phone: [phone]
IRC: mhroncok

[Petr Viktorin]

On 04. 11. 21 12:49, Miro Hrončok wrote:
>  Hello Pythonistas.
>  
>  After some recent improvements in the Python RPM dependency generators, 
>  a regression was discovered [0].
>  
>  Turns out the error happened when the upstream metadata contained a 
>  requirement with a PEP 440 [1] incompatible version. A fix is pending 
>  and only one package in Fedora was affected.
>   
[...]
>  
>  tl;dr:
>  
>    - there might be cases where legacy versions are needed
>    - but in most cases, they should be avoided
>  
>  Hence, I propose we do the following in Rawhide:
>  
>  We turn LegacyVersions to failures, but we provide a stop-gap measure, 
>  such as (%global python_dependency_allow_legacy_version_provides 1 /  
>  %global python_dependency_allow_legacy_version_requires 1 ) for packages 
>  that need to override this. When pypa/packaging actually drops 
>  LegacyVersion, this stop-gap measure will no longer work. 

I suggest removing LegacyVersion support without replacement. I think 
neither creating nor using 
"python_dependency_allow_legacy_version_requires" would be worth the 
extra work.
All the uses of it that we've seen were mistakes, and since upstream 
tools are removing support for it, the stop-gap is just delaying the 
inevitable.
If we find an actual use case for it, it can always be added.