To help automatic building RPMs from native Python packages, we need an automatic way to record the software's upstream name (dist name, name on PyPI) in Fedora packages.
For this, we are using RPM's automatic dependency generator written by Per Øyvind Karlsen and Neal Gompa, which automatically scans ".dist-info" and similar files and generates virtual Provides, currently in the form "pythonX.Ydist(name)". (The generator was originally developed for Mandriva and Mageia; finally it's coming to Fedora as well!) The Fedora Change page for this feature is here: https://fedoraproject.org/wiki/Changes/Automatic_Provides_for_Python_RPM_Pac...
Unfortunately, while implementing this, we failed to understand how the virtual provides work with source RPMs. This leads to the situation that, as currently planned, the automatic provides are only useful in "Requires" lines, not in "BuildRequires".
The problem is that if a requirement like "python3.5dist(name)" is present in a SRPM, the SRPM can't be rebuilt on systems with other Python versions, which would provide, say, "python3.6dist(name)". For these systems, the SRPM would need to be rebuilt, which not all tools do.
Igor Gnatenko (ignatenkobrain) explained this on IRC on #fedora-python, after which we had a long discussion about the problem and possible solutions. We're sorry for not getting this earlier -- despite several people mentioning it (including Neal who wrote the generator)!
The fix is to enable "--majorver-provides" in the dependency generator, so that each package will provide two forms of the virtual provide: python3.5dist(name) python3dist(name) The full version should then be used for runtime Requires, while the one with major version only should be used for BuildRequires.
We'll wrap this up in easy-to use (and hopefully future-proof) macros:
* One macro for getting the canonical (normalized) dist-name: %{python_dist_name NAME}
* Four macros for adding Requires and BuildRequires lines (which use the python_dist_name macro above):
%{requires_python3_dist NAME} => Requires: python3.Ydist(name) %{buildrequires_python3_dist NAME} => BuildRequires: python3dist(name) and similarly for Python 2.
An alternative would be macros that don't include the keyword "Requires:" or "BuildRequires:". This would result in specfiles with lines like: BuildRequires: %{python3_dist_br name} Requires: %{python3_dist name} This would be susceptible to people copying the Requires line, adding "Build" to the front, and forgetting to change the macro as well, leading to a hard-to-catch failure (working binary RPMs but SRPMs that fail to rebuild on other distro versions). Macros that include the keyword are more robust to copy-pasting.
Since the %buildrequires_python3_dist and %python_dist_name macros will be used in the SRPM, they'll need to go in macros.python-srpm to be present in the default buildroot. The %requires_python3_dist macro (and %pythonX_version) can live in macros.pythonX.
Following is the new road plan:
# Plan for Fedora 25: * The 5 macros will be created and deployed. * The --majorver-provides swich for the Provides generator in Fedora RPM will be enabled * Fedora Packaging Guidelines for Python will be amended: * The addition of the pythonX.Ydist(..) tags will be explained. * The %{python_dist_name} macro will be advertised. * The %{requires_pythonX_dist} macros will be advertised for use in generating Requires tags. * It will be explained that, at this time, it is impossible to generate BuildRequires without the providing package being rebuilt, so the %{buildrequires_pythonX_dist} macros will be discouraged for now. * See if we can get in another targeted mass rebuild. If yes, continue with the f26 plan early.
# Plan for Fedora 26: * All Provides will be regenerated in the regular Fedora 26 mass rebuild * Change Python guidelines so the %{buildrequires_pythonX_dist} macros are now encouraged.
"PV" == Petr Viktorin pviktori@redhat.com writes:
PV> * One macro for getting the canonical (normalized) dist-name: PV> %{python_dist_name NAME}
Do you mean "setting" here?
PV> * Four macros for adding Requires and BuildRequires lines (which use PV> the python_dist_name macro above):
PV> %{requires_python3_dist NAME} => Requires: python3.Ydist(name) PV> %{buildrequires_python3_dist NAME} => BuildRequires: %python3dist(name) PV> and similarly for Python 2.
Hmm, I'll have to think about this. I do prefer the python macros all actually start with "python" or "py" just to make them easily recognizable and memorable. And making them too long, well, just contributes to verbosity which we'd like to get away from.
I also need to think about how this fits in with other macro work that's been happening.
- J<
On 08/10/2016 06:49 PM, Jason L Tibbitts III wrote:
"PV" == Petr Viktorin pviktori@redhat.com writes:
PV> * One macro for getting the canonical (normalized) dist-name: PV> %{python_dist_name NAME}
Do you mean "setting" here?
No, getting. Example expansion: %{python_dist_name ndg_HTTPSclient} => ndg-httpsclient
This is normalization that PyPI/pip apply. The goal is that if you can do "pip install X", you can also use "X" in these macros (as long as it's packaged).
PV> * Four macros for adding Requires and BuildRequires lines (which use PV> the python_dist_name macro above):
PV> %{requires_python3_dist NAME} => Requires: python3.Ydist(name) PV> %{buildrequires_python3_dist NAME} => BuildRequires: %python3dist(name) PV> and similarly for Python 2.
Hmm, I'll have to think about this. I do prefer the python macros all actually start with "python" or "py" just to make them easily recognizable and memorable. And making them too long, well, just contributes to verbosity which we'd like to get away from.
Would "python3_requires" and "python3_buildrequires" be better?
I had them start with "requires" as they expand to "Requires:", but I'm not attached to the name.
I also need to think about how this fits in with other macro work that's been happening.
Please do and share your thoughts, I trust you have a good feel for consistency here.
"PV" == Petr Viktorin pviktori@redhat.com writes:
PV> No, getting. Example expansion: PV> %{python_dist_name ndg_HTTPSclient} => ndg-httpsclient
Ah, OK, that makes much more sense.
PV> Would "python3_requires" and "python3_buildrequires" be better?
I think so.
PV> Please do and share your thoughts, I trust you have a good feel for PV> consistency here.
Yeah, I really wish I had actually pushed through the macro work I had done last year. You can see that at https://pagure.io/python-macros
A spec would look like this: https://pagure.io/python-macros/blob/master/f/test1.spec
And most of that is actually implemented. Note the almost complete absence of version-specific bits. That package builds for an arbitrary number of system python versions automatically.
- J<
On 10.8.2016 23:36, Jason L Tibbitts III wrote:
A spec would look like this: https://pagure.io/python-macros/blob/master/f/test1.spec
And most of that is actually implemented. Note the almost complete absence of version-specific bits. That package builds for an arbitrary number of system python versions automatically.
Wow.
Yeah, I really wish I had actually pushed through the macro work I had done last year. You can see that at https://pagure.io/python-macros
A spec would look like this: https://pagure.io/python-macros/blob/master/f/test1.spec
And most of that is actually implemented. Note the almost complete absence of version-specific bits. That package builds for an arbitrary number of system python versions automatically.
That looks incredible! Why didn't it see the light of day? Time constraints or some technical issues?
Maybe it could be revived?
Tomas
"TO" == Tomas Orsava torsava@redhat.com writes:
TO> That looks incredible! Why didn't it see the light of day? Time TO> constraints or some technical issues?
Well, it sort of fell by the wayside as I got involved with other things. I've learned a lot about RPM internals since then and I know I really should get back to it. So many things higher up in the (incredibly huge) queue.
TO> Maybe it could be revived?
Technically it isn't dead; I just haven't worked on it in a (long) while.
Basically I tried to come up with the ideal spec file and worked backwards from that. It's still not going to work for every package, and it still isn't remotely as nice as simply not using an RPM spec at all (and just generating one from metadata) but I think it's at least a start.
Also, it does horrible, horrible things behind the scenes because RPM just doesn't give us a couple of needed bits. I filed a ticket with RPM upstream to try and have it do that but no luck. But the actual macros behind the scenes are very well commented so at least things should be moderately obvious.
Anyway, I'll try to have another look at it at some point. I need to extract the debug scaffolding and such and put that into Fedora separately so that I can avoid rewriting it all the time.
- J<
On 08/11/2016 07:37 PM, Jason L Tibbitts III wrote:
"TO" == Tomas Orsava torsava@redhat.com writes:
TO> That looks incredible! Why didn't it see the light of day? Time TO> constraints or some technical issues?
Well, it sort of fell by the wayside as I got involved with other things. I've learned a lot about RPM internals since then and I know I really should get back to it. So many things higher up in the (incredibly huge) queue.
TO> Maybe it could be revived?
Technically it isn't dead; I just haven't worked on it in a (long) while.
Basically I tried to come up with the ideal spec file and worked backwards from that. It's still not going to work for every package, and it still isn't remotely as nice as simply not using an RPM spec at all (and just generating one from metadata) but I think it's at least a start.
Also, it does horrible, horrible things behind the scenes because RPM just doesn't give us a couple of needed bits. I filed a ticket with RPM upstream to try and have it do that but no luck. But the actual macros behind the scenes are very well commented so at least things should be moderately obvious.
The magic worries me. It seems like if these macros were finished, you'd be about the only person capable of maintaining them. And if something goes wrong (magic tends to imply fragility), I'm not looking forward to the debugging sessions. So, while I am blown away by this project, I'm inclined to place my bets on pyp2rpm instead.
I don't think the end goals – not having to write a spec at all, or write an ideal spec – are as important as the debugging experience.
But, that's all just my view; I have no intentions of hindering the project, and I encourage anyone involved with RPM macros to study it and see what's possible.
"PV" == Petr Viktorin pviktori@redhat.com writes:
PV> The magic worries me. It seems like if these macros were finished, PV> you'd be about the only person capable of maintaining them.
I don't think so. There are other committers, and the core of it didn't even come from me so there's certainly at least two other people around who can handle this stuff. It's true that not a whole lot of people understand the deeper interactions between the RPM Lua interpreter and the rest of RPM, but it took me only about two days to figure out most of it _without_ having any documented examples. And I have a real job that isn't Fedora.
The macros are very well commented; the magic is described in detail. They also include a debugging framework. Which is more than, well, any other current macro magic. Ever tried to debug debuginfo package generation? Or even looked at the SCL macros? (Which are about on par with magic-ness, but which are completely unreadable and have no debugging.)
PV> And if something goes wrong (magic tends to imply fragility), I'm PV> not looking forward to the debugging sessions. So, while I am blown PV> away by this project, I'm inclined to place my bets on pyp2rpm PV> instead.
Well, there's no reason not to have more than one solution. However, pyp2rpm just gets you a spec. You still have to maintain said spec, and wiping it out with a fresh run of the generator is not really acceptable. I'd generally argue that pyp2rpm would actually generate a spec using this stuff, once it's actually proven that it works.
I would still like to have the spec file be simply an intermediary between some metadata and the build system. Or, if you can write pyp2rpm in lua, I could actually build it directly into rpm. Have the regular RPM header, then a %prep section that unpacks the source and calls a macro. The rest of the spec (except the %changelog section) is simply generated from the metadata.
- J<
On 08/12/2016 03:17 PM, Jason L Tibbitts III wrote:
"PV" == Petr Viktorin pviktori@redhat.com writes:
PV> The magic worries me. It seems like if these macros were finished, PV> you'd be about the only person capable of maintaining them.
I don't think so. There are other committers, and the core of it didn't even come from me so there's certainly at least two other people around who can handle this stuff. It's true that not a whole lot of people understand the deeper interactions between the RPM Lua interpreter and the rest of RPM, but it took me only about two days to figure out most of it _without_ having any documented examples. And I have a real job that isn't Fedora.
Thanks for clearing that up, I'm less worried now.
The macros are very well commented; the magic is described in detail. They also include a debugging framework. Which is more than, well, any other current macro magic. Ever tried to debug debuginfo package generation? Or even looked at the SCL macros? (Which are about on par with magic-ness, but which are completely unreadable and have no debugging.)
The macros are definitely of exceptional quality. But, still they're RPM macros. (Yes, I have looked SCL macros – that's something I'd not encourage people to study...)
PV> And if something goes wrong (magic tends to imply fragility), I'm PV> not looking forward to the debugging sessions. So, while I am blown PV> away by this project, I'm inclined to place my bets on pyp2rpm PV> instead.
Well, there's no reason not to have more than one solution. However, pyp2rpm just gets you a spec. You still have to maintain said spec, and wiping it out with a fresh run of the generator is not really acceptable. I'd generally argue that pyp2rpm would actually generate a spec using this stuff, once it's actually proven that it works.
The idea with pyp2rpm is to work with the Python Packaging Authority so that the upstream metadata *can* be converted automatically. Ideally Fedora packagers would fix packaging problems upstream, rather than maintaining spec files. Ideas for a tool for *updating* spec files are also floating around.
The idea of pyp2rpm generating spec files using the macros sounds good. Indeed the projects are more orthogonal than I realized.
I would still like to have the spec file be simply an intermediary between some metadata and the build system. Or, if you can write pyp2rpm in lua, I could actually build it directly into rpm. Have the regular RPM header, then a %prep section that unpacks the source and calls a macro. The rest of the spec (except the %changelog section) is simply generated from the metadata.
Agreed; it would be nice to get to that kind of situation in the future. Not sure about the lua part, at least while pyp2rpm needs heavy development :)
On 08/12/2016 05:40 PM, Petr Viktorin wrote:
The idea with pyp2rpm is to work with the Python Packaging Authority so that the upstream metadata *can* be converted automatically. Ideally Fedora packagers would fix packaging problems upstream, rather than maintaining spec files. Ideas for a tool for *updating* spec files are also floating around.
There's already rebase-helper (it's an interactive tool as well as an automatic one) that specializes in updating spec files.
It can't track/change BR/R's as RPM is Turing complete and impossible to parse. Imagine, we have pythonXdist(foo) extracted from PyPI metadata, but in Fedora we still need to add some more BR for that, so we add it. When new release comes (still without added BR in upstream) rebase-helper will not be helpful. It can change only version of spec.
On Mon, Aug 15, 2016 at 11:25 AM, Tomas Orsava torsava@redhat.com wrote:
On 08/12/2016 05:40 PM, Petr Viktorin wrote:
The idea with pyp2rpm is to work with the Python Packaging Authority so that the upstream metadata *can* be converted automatically. Ideally Fedora packagers would fix packaging problems upstream, rather than maintaining spec files. Ideas for a tool for *updating* spec files are also floating around.
There's already rebase-helper (it's an interactive tool as well as an automatic one) that specializes in updating spec files.
python-devel mailing list python-devel@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject...
On 15 August 2016 at 19:42, Igor Gnatenko ignatenko@redhat.com wrote:
It can't track/change BR/R's as RPM is Turing complete and impossible to parse. Imagine, we have pythonXdist(foo) extracted from PyPI metadata, but in Fedora we still need to add some more BR for that, so we add it. When new release comes (still without added BR in upstream) rebase-helper will not be helpful. It can change only version of spec.
This is a self-inflicted problem arising from our current tooling design, though, since "generate-and-edit" isn't the only way to supplement upstream metadata: we can also design spec file generators to accept a supplemental config file in addition to the upstream metadata.
If we're using that alternate model, then rebase-helper can have a much easier time of things, since it isn't trying to edit a previously generated spec file, it's just generating a new one based on the new upstream metadata and the old supplemental metadata, and seeing if the result still passes CI.
Cheers, Nick.
On 08/17/2016 06:52 PM, Nick Coghlan wrote:
On 15 August 2016 at 19:42, Igor Gnatenko ignatenko@redhat.com wrote:
It can't track/change BR/R's as RPM is Turing complete and impossible to parse. Imagine, we have pythonXdist(foo) extracted from PyPI metadata, but in Fedora we still need to add some more BR for that, so we add it. When new release comes (still without added BR in upstream) rebase-helper will not be helpful. It can change only version of spec.
This is a self-inflicted problem arising from our current tooling design, though, since "generate-and-edit" isn't the only way to supplement upstream metadata: we can also design spec file generators to accept a supplemental config file in addition to the upstream metadata.
If we're using that alternate model, then rebase-helper can have a much easier time of things, since it isn't trying to edit a previously generated spec file, it's just generating a new one based on the new upstream metadata and the old supplemental metadata, and seeing if the result still passes CI.
The more I think about it, the more it seems to me that our plan should be: - Make it possible for pyp2rpm to use extra data to augment/override what's in the upstream package. - Make it standard practice in Fedora to use this data and treat the spec file as an immutable generated artifact. - Treat metadata errors/omissions as upstream bugs, provided the desired state can be expressed in setup.py - For kinds of metadata that setup.py can't express, strive to add them to future versions of the upstream packaging format. - For the far future, perhaps start getting rid of spec files: teach Koji to generate them, and stop storing them in dist-git.
Helper macros stay orthogonal -- pyp2rpm would just need to learn to use them.
"PV" == Petr Viktorin pviktori@redhat.com writes:
PV> - Make it standard practice in Fedora to use this data and treat the PV> spec file as an immutable generated artifact.
If you're saying that any changes which are made to the spec file (say, by release engineering doing a rebuild or by someone tweaking the package for pretty much any reason) will be overwritten, this is not a particularly good idea.
- J<
On Tue, Aug 23, 2016 at 11:10:08AM -0500, Jason L Tibbitts III wrote:
"PV" == Petr Viktorin pviktori@redhat.com writes:
PV> - Make it standard practice in Fedora to use this data and treat the PV> spec file as an immutable generated artifact.
If you're saying that any changes which are made to the spec file (say, by release engineering doing a rebuild or by someone tweaking the package for pretty much any reason) will be overwritten, this is not a particularly good idea.
%changelog would have to be stored separately. This would apply even for the case when no "external" changes were made — just normal updates by the package managers — since there's no way for pyp2rpm to know what happened in the past.
Anyway, I think it's a good idea to try to make automatically generated spec files work, but making this mandatory sound premature. Even in the future, there's always be packages which need some manual fixup to whatever pyp2rpm produces.
Zbyszek
"ZJ" == Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl writes:
ZJ> %changelog would have to be stored separately. This would apply even ZJ> for qthe case when no "external" changes were made — just normal updates ZJ> by the package managers — since there's no way for pyp2rpm to know ZJ> what happened in the past.
It certainly could; it has the existing spec.
- J<
On Tue, Aug 23, 2016 at 12:48:40PM -0500, Jason L Tibbitts III wrote:
"ZJ" == Zbigniew Jędrzejewski-Szmek zbyszek@in.waw.pl writes:
ZJ> %changelog would have to be stored separately. This would apply even ZJ> for qthe case when no "external" changes were made — just normal updates ZJ> by the package managers — since there's no way for pyp2rpm to know ZJ> what happened in the past.
It certainly could; it has the existing spec.
Yeah, that'd be even better to pull the changelog from existing spec.
Zbyszek
On 24 August 2016 at 02:10, Jason L Tibbitts III tibbs@math.uh.edu wrote:
"PV" == Petr Viktorin pviktori@redhat.com writes:
PV> - Make it standard practice in Fedora to use this data and treat the PV> spec file as an immutable generated artifact.
If you're saying that any changes which are made to the spec file (say, by release engineering doing a rebuild or by someone tweaking the package for pretty much any reason) will be overwritten, this is not a particularly good idea.
I think it's the direction we should be aiming to go, though - creating snowflake Python packages in a pyp2rpm world should be as questionable a decision as creating a snowflake server is in an Ansible world.
If a stock standard "dump some files in site-packages, and maybe install a command line entry point") Python project cannot be automatically rebased and the spec file regenerated without losing any data, then that's either a failure to upstream a metadata fix, or else a problem to be addressed in the spec file generator.
Folks should still have a way to say "this project is a special snowflake, and I am going to carefully curate the spec file by hand", but they should also have to opt in to that, rather than having it as the default (just as all deployed servers should be under configuration management by default, and people have to turn that off if they want their local modifications to stick).
Cheers, Nick.
Is an unversioned python(name) also being created automagically? I manually create these for the command line tools which support multiple Python versions, where the python2-name and python3-name package both provide python(name) and then the name package requires python(name) with a recommends for one or the other.
On Wed, Aug 10, 2016 at 12:38 PM, Petr Viktorin pviktori@redhat.com wrote:
To help automatic building RPMs from native Python packages, we need an automatic way to record the software's upstream name (dist name, name on PyPI) in Fedora packages.
For this, we are using RPM's automatic dependency generator written by Per Øyvind Karlsen and Neal Gompa, which automatically scans ".dist-info" and similar files and generates virtual Provides, currently in the form "pythonX.Ydist(name)". (The generator was originally developed for Mandriva and Mageia; finally it's coming to Fedora as well!) The Fedora Change page for this feature is here: https://fedoraproject.org/wiki/Changes/Automatic_Provides_ for_Python_RPM_Packages
Unfortunately, while implementing this, we failed to understand how the virtual provides work with source RPMs. This leads to the situation that, as currently planned, the automatic provides are only useful in "Requires" lines, not in "BuildRequires".
The problem is that if a requirement like "python3.5dist(name)" is present in a SRPM, the SRPM can't be rebuilt on systems with other Python versions, which would provide, say, "python3.6dist(name)". For these systems, the SRPM would need to be rebuilt, which not all tools do.
Igor Gnatenko (ignatenkobrain) explained this on IRC on #fedora-python, after which we had a long discussion about the problem and possible solutions. We're sorry for not getting this earlier -- despite several people mentioning it (including Neal who wrote the generator)!
The fix is to enable "--majorver-provides" in the dependency generator, so that each package will provide two forms of the virtual provide: python3.5dist(name) python3dist(name) The full version should then be used for runtime Requires, while the one with major version only should be used for BuildRequires.
We'll wrap this up in easy-to use (and hopefully future-proof) macros:
One macro for getting the canonical (normalized) dist-name: %{python_dist_name NAME}
Four macros for adding Requires and BuildRequires lines (which use the python_dist_name macro above):
%{requires_python3_dist NAME} => Requires: python3.Ydist(name) %{buildrequires_python3_dist NAME} => BuildRequires: python3dist(name) and similarly for Python 2.
An alternative would be macros that don't include the keyword "Requires:" or "BuildRequires:". This would result in specfiles with lines like: BuildRequires: %{python3_dist_br name} Requires: %{python3_dist name} This would be susceptible to people copying the Requires line, adding "Build" to the front, and forgetting to change the macro as well, leading to a hard-to-catch failure (working binary RPMs but SRPMs that fail to rebuild on other distro versions). Macros that include the keyword are more robust to copy-pasting.
Since the %buildrequires_python3_dist and %python_dist_name macros will be used in the SRPM, they'll need to go in macros.python-srpm to be present in the default buildroot. The %requires_python3_dist macro (and %pythonX_version) can live in macros.pythonX.
Following is the new road plan:
# Plan for Fedora 25:
- The 5 macros will be created and deployed.
- The --majorver-provides swich for the Provides generator in Fedora RPM
will be enabled
- Fedora Packaging Guidelines for Python will be amended:
- The addition of the pythonX.Ydist(..) tags will be explained.
- The %{python_dist_name} macro will be advertised.
- The %{requires_pythonX_dist} macros will be advertised for use in generating Requires tags.
- It will be explained that, at this time, it is impossible to generate BuildRequires without the providing package being rebuilt, so the %{buildrequires_pythonX_dist} macros will be discouraged for now.
- See if we can get in another targeted mass rebuild. If yes, continue with the f26 plan early.
# Plan for Fedora 26:
- All Provides will be regenerated in the regular Fedora 26 mass rebuild
- Change Python guidelines so the %{buildrequires_pythonX_dist} macros are now encouraged.
-- Petr Viktorin _______________________________________________ python-devel mailing list python-devel@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/python-devel@lis ts.fedoraproject.org
Hi! No, this automatic Provides generator does not create "python(name)". Only "pythonX.Ydist()" and "pythonXdist()".
On 08/10/2016 07:09 PM, Avram Lubkin wrote:
Is an unversioned python(name) also being created automagically? I manually create these for the command line tools which support multiple Python versions, where the python2-name and python3-name package both provide python(name) and then the name package requires python(name) with a recommends for one or the other.
On Wed, Aug 10, 2016 at 12:38 PM, Petr Viktorin <pviktori@redhat.com mailto:pviktori@redhat.com> wrote:
To help automatic building RPMs from native Python packages, we need an automatic way to record the software's upstream name (dist name, name on PyPI) in Fedora packages. For this, we are using RPM's automatic dependency generator written by Per Øyvind Karlsen and Neal Gompa, which automatically scans ".dist-info" and similar files and generates virtual Provides, currently in the form "pythonX.Ydist(name)". (The generator was originally developed for Mandriva and Mageia; finally it's coming to Fedora as well!) The Fedora Change page for this feature is here: https://fedoraproject.org/wiki/Changes/Automatic_Provides_for_Python_RPM_Packages <https://fedoraproject.org/wiki/Changes/Automatic_Provides_for_Python_RPM_Packages> Unfortunately, while implementing this, we failed to understand how the virtual provides work with source RPMs. This leads to the situation that, as currently planned, the automatic provides are only useful in "Requires" lines, not in "BuildRequires". The problem is that if a requirement like "python3.5dist(name)" is present in a SRPM, the SRPM can't be rebuilt on systems with other Python versions, which would provide, say, "python3.6dist(name)". For these systems, the SRPM would need to be rebuilt, which not all tools do. Igor Gnatenko (ignatenkobrain) explained this on IRC on #fedora-python, after which we had a long discussion about the problem and possible solutions. We're sorry for not getting this earlier -- despite several people mentioning it (including Neal who wrote the generator)! The fix is to enable "--majorver-provides" in the dependency generator, so that each package will provide two forms of the virtual provide: python3.5dist(name) python3dist(name) The full version should then be used for runtime Requires, while the one with major version only should be used for BuildRequires. We'll wrap this up in easy-to use (and hopefully future-proof) macros: * One macro for getting the canonical (normalized) dist-name: %{python_dist_name NAME} * Four macros for adding Requires and BuildRequires lines (which use the python_dist_name macro above): %{requires_python3_dist NAME} => Requires: python3.Ydist(name) %{buildrequires_python3_dist NAME} => BuildRequires: python3dist(name) and similarly for Python 2. An alternative would be macros that don't include the keyword "Requires:" or "BuildRequires:". This would result in specfiles with lines like: BuildRequires: %{python3_dist_br name} Requires: %{python3_dist name} This would be susceptible to people copying the Requires line, adding "Build" to the front, and forgetting to change the macro as well, leading to a hard-to-catch failure (working binary RPMs but SRPMs that fail to rebuild on other distro versions). Macros that include the keyword are more robust to copy-pasting. Since the %buildrequires_python3_dist and %python_dist_name macros will be used in the SRPM, they'll need to go in macros.python-srpm to be present in the default buildroot. The %requires_python3_dist macro (and %pythonX_version) can live in macros.pythonX. Following is the new road plan: # Plan for Fedora 25: * The 5 macros will be created and deployed. * The --majorver-provides swich for the Provides generator in Fedora RPM will be enabled * Fedora Packaging Guidelines for Python will be amended: * The addition of the pythonX.Ydist(..) tags will be explained. * The %{python_dist_name} macro will be advertised. * The %{requires_pythonX_dist} macros will be advertised for use in generating Requires tags. * It will be explained that, at this time, it is impossible to generate BuildRequires without the providing package being rebuilt, so the %{buildrequires_pythonX_dist} macros will be discouraged for now. * See if we can get in another targeted mass rebuild. If yes, continue with the f26 plan early. # Plan for Fedora 26: * All Provides will be regenerated in the regular Fedora 26 mass rebuild * Change Python guidelines so the %{buildrequires_pythonX_dist} macros are now encouraged. -- Petr Viktorin _______________________________________________ python-devel mailing list python-devel@lists.fedoraproject.org <mailto:python-devel@lists.fedoraproject.org> https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org <https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject.org>
python-devel mailing list python-devel@lists.fedoraproject.org https://lists.fedoraproject.org/admin/lists/python-devel@lists.fedoraproject...
python-devel@lists.fedoraproject.org
-
Avram Lubkin -
Igor Gnatenko -
Jason L Tibbitts III -
Miro Hrončok -
Nick Coghlan -
Petr Viktorin -
Tomas Orsava -
Zbigniew Jędrzejewski-Szmek