8:42 a.m.
Hi all,
so I started to work on packaging Python 3.4 beta 1 and I wanted to create ensurepip patch
also acceptable for upstream - Toshio and Nick talked about that at [1]. I tried to
analyze the situation and here is what I have. First some facts:
- with ensurepip, Python 3.4 bundles setuptools and pip wheel archives
- when ensurepip is run, it installs (or upgrades) currently installed pip from the
bundled wheels
A way to solve this "properly" in a distro (as mentioned in [1]) would be:
- throw away bundled setuptools and pip wheels when building Python
- making python3 require python-pip
- patching ensurepip (this is the fun part):
-- if ensurepip is run in venv:
--- we will need to reconstruct the wheels and install them into venv; or just grab the
RECORD files and use them as a list of files we need to copy into venv
--- to reconstruct the wheels from installed RPMs, it'd be a good idea to package
python-{setuptools,pip} as wheels (with the ".dist-info" directory and all the
wheel goodies inside) so that we have the RECORD files
--- to package setuptools and pip as wheels, they'll have to BR: python3-pip (I saw
Toshio's notes from flock and I agree with the part about how Fedora should use wheels
[2])
-- if ensurepip is run systemwide:
--- if "--upgrade" is used, should this run "yum update python3-pip"?
Now, when new X.Y version Python is introduced, we will have to:
- bootstrap the new python3 interpreter to not require python3-pip, since python3-pip will
require python(abi)=X.(Y-1)
- then we will need to build setuptools and pip *not as wheels*, because we still
won't have pip that would work with our new Python X.Y - that means boostrapping them
too and building them using the current "setup.py install" approach
- when we will have the new pip, we will be able to reenable python3 requires for
python3-pip
- and we will also be able to rebuild setuptools and pip *as wheels*
There are also some more minor issues, that are worth mentioning:
- We typically won't update pip after Fedora is released - if there are security
issues, we will backport patches and bump the release. The problem with this is that
"ensurepip --upgrade" is not able to see Fedora releases and wouldn't work
in venv, leaving it vulnerable. I think we have three options here:
-- Either we use some Fedora-specific RPM magic, which won't be acceptable for
upstream
-- Or we will use a nice feature of wheel RECORD files - they keep file hashes, so we can
check venv pip against system pip and see if something has changed when "ensurepip
--upgrade" is run
-- Or we can just copy whole pip again into venv, regardless of what is currently there
(doesn't seem very nice..., but it's simple and in the end it works as the above
method)
- We can't copy the *.pyc and *.pyo files from system to venv, because they have file
locations hardcoded in them, that are used in tracebacks. We can solve this by just
copying *.py files and running py_compile in our ensurepip.
I hope I covered all the important points. Basically, we can make this work in a way
acceptable for upstream, if we package setupttols and pip as wheels. It'll require
some extra effort, but I think it's worth it.
Thoughts? Anyone has better/simpler ideas?
Thanks a lot,
Slavek.
[1] https://lists.fedoraproject.org/pipermail/python-devel/2013-November/0005...
[2]
https://fedoraproject.org/wiki/User:Toshio/Flock2013_Python_Guidelines#Wh...
7:19 p.m.
On 11/28/2013 12:42 AM, Bohuslav Kabrda wrote:
I hope I covered all the important points. Basically, we can make
this work in a way acceptable for upstream, if we package setupttols and pip as wheels.
It'll require some extra effort, but I think it's worth it.
Thoughts? Anyone has better/simpler ideas?
From an upstream point of view, so long as test.test_ensurepip and
test.test_venv still work, things should generally be OK.
I quite like the idea of checking for the consistency of the RECORD
files between the system pip and the one in the virtualenv (as well as
using RECORD as a guide to what to copy into a fresh venv). If you get
that working, I'd be interested in a Python 3.5 venv and/or ensurepip
patch to do that by default, and only bootstrap from the embedded wheel
if there was no system pip available.
Cheers,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
8:26 a.m.
----- Original Message -----
On 11/28/2013 12:42 AM, Bohuslav Kabrda wrote:
> I hope I covered all the important points. Basically, we can make this work
> in a way acceptable for upstream, if we package setupttols and pip as
> wheels. It'll require some extra effort, but I think it's worth it.
> Thoughts? Anyone has better/simpler ideas?
From an upstream point of view, so long as test.test_ensurepip and
test.test_venv still work, things should generally be OK.
I quite like the idea of checking for the consistency of the RECORD
files between the system pip and the one in the virtualenv (as well as
using RECORD as a guide to what to copy into a fresh venv). If you get
that working, I'd be interested in a Python 3.5 venv and/or ensurepip
patch to do that by default, and only bootstrap from the embedded wheel
if there was no system pip available.
Cheers,
Nick.
Actually, there seems to be a much simpler way of doing this in Fedora (and any distro
more generally):
- setuptools and pip RPMs will carry the wheel inside them and drop it into
ensurepip/_bundled
- the wheels will be rebuilt during every RPM build everytime *after patching*, so they
will carry security patches etc.
- we will use the RPM release as the "build tag" mentioned in PEP 427 [1], so
that when we e.g. fix a security bug but don't bump the version, "ensurepip
--upgrade" will still see that the wheel has to be reinstalled (otherwise it'd
say think the version is already there and wouldn't reinstall)
So the only thing we will need to implement will be autodiscovery of the wheels, since
they will change names independently on python3 package, but I think we can do that :)
From upstream point of view this shouldn't break anything, but it'd also probably
not have any benefit. Would you still accept such patch?
Regards,
Slavek.
[1] http://www.python.org/dev/peps/pep-0427/#id11
9:03 a.m.
On Dec 10, 2013 6:26 AM, "Bohuslav Kabrda" <bkabrda(a)redhat.com> wrote:
Note that this misses one thing mentioned in my mail: local changes.
Because scripting languages are readable and modifiable, sysadmins will
sometimes fix bugs that affect them by deploying just the changed .py files
(rather than rebuilding an rpm package). Configuration management (puppet,
Ansible, chef, etc) makes it easy to deploy and track hotfixes this way.
If wheels are cached at rpm buildtime then this workflow will cease to
function correctly for virtualenv and these bundled libraries. Since the
bundling is an implementation detail that sysadmins will have no reason to
know about until they want to hotfix whichever libraries are bundled, this
is likely to take them a while to figure out.
Creating the wheels from the files on the system keeps the number of copies
of files to a minimum. That, in turn, makes it easier for everyone
involved to discover which files are the ones that do something.
-Toshio
> Regards,
> Slavek.
> [1] http://www.python.org/dev/peps/pep-0427/#id11
> _______________________________________________
> python-devel mailing list
> python-devel(a)lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/python-devel
----- Original Message -----
> On 11/28/2013 12:42 AM, Bohuslav Kabrda wrote:
> > I hope I covered all the important points. Basically, we can make
this
work
> > in a way acceptable for upstream, if we package setupttols
and pip as
> > wheels. It'll require some extra effort, but I think it's worth it.
> > Thoughts? Anyone has better/simpler ideas?
> From an upstream point of view, so long as
test.test_ensurepip and
> test.test_venv still work, things should generally be OK.
> I quite like the idea of checking for the consistency of
the RECORD
> files between the system pip and the one in the virtualenv (as well as
> using RECORD as a guide to what to copy into a fresh venv). If you get
> that working, I'd be interested in a Python 3.5 venv and/or ensurepip
> patch to do that by default, and only bootstrap from the embedded wheel
> if there was no system pip available.
> Cheers,
> Nick.
Actually, there seems to be a much simpler way of doing this in Fedora
(and any
distro more generally):
- setuptools and pip RPMs will carry the wheel inside them and drop
it
into ensurepip/_bundled
- the wheels will be rebuilt during every RPM build everytime *after
patching*, so they will carry security patches etc.
- we will use the RPM release as the "build tag" mentioned
in PEP 427
[1], so that when we e.g. fix a security bug but don't bump the
version,
"ensurepip --upgrade" will still see that the wheel has to be reinstalled
(otherwise it'd say think the version is already there and wouldn't
reinstall)
So the only thing we will need to implement will be autodiscovery of
the
wheels, since they will change names independently on python3 package, but
I think we can do that :) From upstream point of view this shouldn't break
anything, but it'd also probably not have any benefit. Would you still
accept such patch?
9:28 a.m.
> Actually, there seems to be a much simpler way of doing this in
Fedora (and
> any distro more generally):
> - setuptools and pip RPMs will carry the wheel inside them and drop it into
> ensurepip/_bundled
> - the wheels will be rebuilt during every RPM build everytime *after
> patching*, so they will carry security patches etc.
> - we will use the RPM release as the "build tag" mentioned in PEP 427
[1],
> so that when we e.g. fix a security bug but don't bump the version,
> "ensurepip --upgrade" will still see that the wheel has to be reinstalled
> (otherwise it'd say think the version is already there and wouldn't
> reinstall)
> So the only thing we will need to implement will be autodiscovery of the
> wheels, since they will change names independently on python3 package, but
> I think we can do that :) From upstream point of view this shouldn't break
> anything, but it'd also probably not have any benefit. Would you still
> accept such patch?
>
Note that this misses one thing mentioned in my mail: local changes.
Because scripting languages are readable and modifiable, sysadmins
will
sometimes fix bugs that affect them by deploying just the changed .py files
(rather than rebuilding an rpm package). Configuration management (puppet,
Ansible, chef, etc) makes it easy to deploy and track hotfixes this way. If
wheels are cached at rpm buildtime then this workflow will cease to function
correctly for virtualenv and these bundled libraries. Since the bundling is
an implementation detail that sysadmins will have no reason to know about
until they want to hotfix whichever libraries are bundled, this is likely to
take them a while to figure out.
The upstream solution has this problem, too. In
fact, this approach will be much closer to the upstream solution in the sense that it uses
prebuilt wheels to install/update virtualenv. The wheels will just come from different
packages than python3 itself. IMO it'd be sufficient to place a README file into the
ensurepip/_bundled dir that would explain this. Actually, diverging from upstream in this
way is an argument for not doing the "files copying" solution now.
While trying to create the patch that would rebuild the wheels from system and install
them in venv, I bumped into few ugly problems, which is why I came up with this
"wheels in RPMs" solution in the first place. I'd like to do this for 3.4 in
Fedora and then start working on the "file copying" solution that would be
acceptable for upstream Python 3.5. When this is done upstream, we will move to there with
3.5.
Creating the wheels from the files on the system keeps the number of
copies
of files to a minimum. That, in turn, makes it easier for everyone involved
to discover which files are the ones that do something.
-Toshio
> Regards,
> Slavek.
>
> [1] http://www.python.org/dev/peps/pep-0427/#id11
> _______________________________________________
> python-devel mailing list
> python-devel(a)lists.fedoraproject.org
> https://admin.fedoraproject.org/mailman/listinfo/python-devel
--
Regards,
Bohuslav "Slavek" Kabrda.
7:36 p.m.
On 12/11/2013 01:28 AM, Bohuslav Kabrda wrote:
> Actually, there seems to be a much simpler way of doing this
in Fedora (and any distro more generally):
> - setuptools and pip RPMs will carry the wheel inside them and drop it into
ensurepip/_bundled
> - the wheels will be rebuilt during every RPM build everytime *after patching*,
so they will carry security patches etc.
> - we will use the RPM release as the "build tag" mentioned in PEP 427
[1], so that when we e.g. fix a security bug but don't bump the version,
"ensurepip --upgrade" will still see that the wheel has to be reinstalled
(otherwise it'd say think the version is already there and wouldn't reinstall)
> So the only thing we will need to implement will be autodiscovery of the wheels,
since they will change names independently on python3 package, but I think we can do that
:) From upstream point of view this shouldn't break anything, but it'd also
probably not have any benefit. Would you still accept such patch?
>
Note that this misses one thing mentioned in my mail: local changes.
Because scripting languages are readable and modifiable, sysadmins
will sometimes fix bugs that affect them by deploying just the
changed .py files (rather than rebuilding an rpm package).
Configuration management (puppet, Ansible, chef, etc) makes it easy
to deploy and track hotfixes this way. If wheels are cached at rpm
buildtime then this workflow will cease to function correctly for
virtualenv and these bundled libraries. Since the bundling is an
implementation detail that sysadmins will have no reason to know
about until they want to hotfix whichever libraries are bundled,
this is likely to take them a while to figure out.
The upstream solution has this problem, too. In fact, this approach will
be much closer to the upstream solution in the sense that it uses
prebuilt wheels to install/update virtualenv. The wheels will just come
from different packages than python3 itself. IMO it'd be sufficient to
place a README file into the ensurepip/_bundled dir that would explain
this. Actually, diverging from upstream in this way is an argument for
not doing the "files copying" solution now.
Well, we only went for the full bundling solution upstream as an
expedient hack that we *knew* we could get working in time for Python
3.4. I'd actually prefer it if, given a system install of pip (and
setuptools), ensurepip bootstrapped those into new environments rather
than the bundled ones.
At the moment, if you do "pip install --upgrade pip" in your system
Python (regardless of platform), future virtual environments will still
get the old pip until the next CPython maintenance release. Now, in
theory, there will be new CPython releases shortly after each new pip
release, but going to a new CPython maintenance release is a higher
impact change than just updating pip, so we have to expect substantial
lags between pip updates and CPython updates in many environments.
While trying to create the patch that would rebuild the wheels from
system and install them in venv, I bumped into few ugly problems, which
is why I came up with this "wheels in RPMs" solution in the first place.
I'd like to do this for 3.4 in Fedora and then start working on the
"file copying" solution that would be acceptable for upstream Python
3.5. When this is done upstream, we will move to there with 3.5.
Alas, it sounds like the pip folks were right when they said "we're not
sure that can actually work at this point" (something I hadn't thought
about previously: at the very least, the install time shebang line
rewriting and wrapper script generation will break. There's also the
problem of *which* scripts get installed, since a system pip for Python
3 likely won't have the bare "pip" command, but we *do* want that in a
virtual environment).
Creating the wheels from the files on the system keeps the number
of
copies of files to a minimum. That, in turn, makes it easier for
everyone involved to discover which files are the ones that do
something.
So, here's a crazy thought: what if, rather than copying the installed
files directly into the virtual environment, we reverse engineered a
wheel archive *dynamically* from the system install and then installed
from that? That would avoid the problems with trying to bypass pip's
script generation, while still bootstrapping new virtual environments
based on the installed versions of pip and setuptools.
Cheers,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
12:52 a.m.
----- Original Message -----
> While trying to create the patch that would rebuild the wheels
from
> system and install them in venv, I bumped into few ugly problems, which
> is why I came up with this "wheels in RPMs" solution in the first place.
> I'd like to do this for 3.4 in Fedora and then start working on the
> "file copying" solution that would be acceptable for upstream Python
> 3.5. When this is done upstream, we will move to there with 3.5.
Alas, it sounds like the pip folks were right when they said "we're not
sure that can actually work at this point" (something I hadn't thought
about previously: at the very least, the install time shebang line
rewriting and wrapper script generation will break. There's also the
problem of *which* scripts get installed, since a system pip for Python
3 likely won't have the bare "pip" command, but we *do* want that in a
virtual environment).
> Creating the wheels from the files on the system keeps the number of
> copies of files to a minimum. That, in turn, makes it easier for
> everyone involved to discover which files are the ones that do
> something.
So, here's a crazy thought: what if, rather than copying the installed
files directly into the virtual environment, we reverse engineered a
wheel archive *dynamically* from the system install and then installed
from that? That would avoid the problems with trying to bypass pip's
script generation, while still bootstrapping new virtual environments
based on the installed versions of pip and setuptools.
There is a slight problem here:
In a stable Fedora release, we typically won't change pip's version, but rather
backport security patches and the version will stay the same. So if you reconstruct the
wheel from that, running "ensurepip --upgrade" in virtualenvs won't actually
do anything, since pip will see the same version in virtualenv as is in the system and do
nothing. There are few possible solutions to this:
- if version in system is the same as in virtualenv, check file checksums and if some
differ, reinstall
- if version in system is the same as in virtualenv, uninstall pip in virtualenv and then
reinstall the one from the system
- (ugly, not upstream acceptable) do "rpm -q python3-pip" and use the Fedora
release as a "build tag" for reconstructed wheel (similar to what I proposed in
the "bundled wheels" solution)
If upstream (you :)) accepts any of these types of behaviour (the second one seems the
easiest and should work fine) in the PEP, I'll work on that.
Quite literally this time, we would reinvent the wheel :)
Cheers,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
--
Regards,
Bohuslav "Slavek" Kabrda.
1:31 a.m.
On 12/11/2013 04:52 PM, Bohuslav Kabrda wrote:
----- Original Message -----
>> While trying to create the patch that would rebuild the wheels from
>> system and install them in venv, I bumped into few ugly problems, which
>> is why I came up with this "wheels in RPMs" solution in the first
place.
>> I'd like to do this for 3.4 in Fedora and then start working on the
>> "file copying" solution that would be acceptable for upstream Python
>> 3.5. When this is done upstream, we will move to there with 3.5.
>
> Alas, it sounds like the pip folks were right when they said "we're not
> sure that can actually work at this point" (something I hadn't thought
> about previously: at the very least, the install time shebang line
> rewriting and wrapper script generation will break. There's also the
> problem of *which* scripts get installed, since a system pip for Python
> 3 likely won't have the bare "pip" command, but we *do* want that in a
> virtual environment).
>
>> Creating the wheels from the files on the system keeps the number of
>> copies of files to a minimum. That, in turn, makes it easier for
>> everyone involved to discover which files are the ones that do
>> something.
>
> So, here's a crazy thought: what if, rather than copying the installed
> files directly into the virtual environment, we reverse engineered a
> wheel archive *dynamically* from the system install and then installed
> from that? That would avoid the problems with trying to bypass pip's
> script generation, while still bootstrapping new virtual environments
> based on the installed versions of pip and setuptools.
There is a slight problem here:
In a stable Fedora release, we typically won't change pip's version, but rather
backport security patches and the version will stay the same. So if you reconstruct the
wheel from that, running "ensurepip --upgrade" in virtualenvs won't actually
do anything, since pip will see the same version in virtualenv as is in the system and do
nothing. There are few possible solutions to this:
- if version in system is the same as in virtualenv, check file checksums and if some
differ, reinstall
- if version in system is the same as in virtualenv, uninstall pip in virtualenv and then
reinstall the one from the system
- (ugly, not upstream acceptable) do "rpm -q python3-pip" and use the Fedora
release as a "build tag" for reconstructed wheel (similar to what I proposed in
the "bundled wheels" solution)
If upstream (you :)) accepts any of these types of behaviour (the second one seems the
easiest and should work fine) in the PEP, I'll work on that.
Quite literally this time, we would reinvent the wheel :)
Ah, true, this is the thing we're planning to fix by formally adding
local version numbering as part of the upstream data model in metadata
2.0 (that is, pip, setuptools, etc would actually be *required* to cope
properly if the version metadata contained a number like "1.5-2" rather
than just "1.5", so that patched downstream rebuilds can have different
version numbers in the metadata than vanilla upstream versions). (This
is already in the draft metadata spec, but I haven't quite worked
through all the implications for version pinning yet)
You may even want to try that out (patching the pip and setuptools
version numbers to include a release suffix) and see if it already
works, since the scheme we're making official in metadata 2.0 is based
on the way pkg_resources.parse_version() already behaves:
>> pkg_resources.parse_version("1.5") <
pkg_resources.parse_version("1.5-1") <
pkg_resources.parse_version("1.5-2")
True
Failing that, I think the uninstall/reinstall trick (preferably combined
with some kind of marker so it only happens when there *is* an upgrade
to be installed) sounds like a reasonable workaround until we get
metadata 2.0 formalised upstream (which will hopefully happen before pip
1.6 is released mid next year).
Cheers,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
9:18 a.m.
----- Original Message -----
On 12/11/2013 04:52 PM, Bohuslav Kabrda wrote:
>> So, here's a crazy thought: what if, rather than copying the installed
>> files directly into the virtual environment, we reverse engineered a
>> wheel archive *dynamically* from the system install and then installed
>> from that? That would avoid the problems with trying to bypass pip's
>> script generation, while still bootstrapping new virtual environments
>> based on the installed versions of pip and setuptools.
My attempt on recreating wheels: https://github.com/bkabrda/rewheel
This implementation doesn't handle included scripts, although it should work fine with
autogenerated entrypoints.
> There is a slight problem here:
> In a stable Fedora release, we typically won't change pip's version, but
> rather backport security patches and the version will stay the same. So if
> you reconstruct the wheel from that, running "ensurepip --upgrade" in
> virtualenvs won't actually do anything, since pip will see the same
> version in virtualenv as is in the system and do nothing. There are few
> possible solutions to this:
> - if version in system is the same as in virtualenv, check file checksums
> and if some differ, reinstall
> - if version in system is the same as in virtualenv, uninstall pip in
> virtualenv and then reinstall the one from the system
> - (ugly, not upstream acceptable) do "rpm -q python3-pip" and use the
> Fedora release as a "build tag" for reconstructed wheel (similar to what
I
> proposed in the "bundled wheels" solution)
> If upstream (you :)) accepts any of these types of behaviour (the second
> one seems the easiest and should work fine) in the PEP, I'll work on that.
> Quite literally this time, we would reinvent the wheel :)
Ah, true, this is the thing we're planning to fix by formally adding
local version numbering as part of the upstream data model in metadata
2.0 (that is, pip, setuptools, etc would actually be *required* to cope
properly if the version metadata contained a number like "1.5-2" rather
than just "1.5", so that patched downstream rebuilds can have different
version numbers in the metadata than vanilla upstream versions). (This
is already in the draft metadata spec, but I haven't quite worked
through all the implications for version pinning yet)
You may even want to try that out (patching the pip and setuptools
version numbers to include a release suffix) and see if it already
works, since the scheme we're making official in metadata 2.0 is based
on the way pkg_resources.parse_version() already behaves:
>>> pkg_resources.parse_version("1.5") <
pkg_resources.parse_version("1.5-1")
>>> < pkg_resources.parse_version("1.5-2")
True
Failing that, I think the uninstall/reinstall trick (preferably combined
with some kind of marker so it only happens when there *is* an upgrade
to be installed) sounds like a reasonable workaround until we get
metadata 2.0 formalised upstream (which will hopefully happen before pip
1.6 is released mid next year).
Well yeah, my point is that there is no upstream-acceptable way other than checking the
file hashes by ensurepip, is there? If I wouldn't want to check file hashes, I'd
have to query RPM for release - or is there some other way you're thinking of?
So I'm basically back at the original idea, you got me :) Although the wheel
reconstruction feels a lot cleaner than just copying files and has some pros like the
entrypoint generation.
Cheers,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
--
Regards,
Bohuslav "Slavek" Kabrda.
10:53 p.m.
On 12/12/2013 01:18 AM, Bohuslav Kabrda wrote:
Well yeah, my point is that there is no upstream-acceptable way other
than checking the file hashes by ensurepip, is there? If I wouldn't want to check file
hashes, I'd have to query RPM for release - or is there some other way you're
thinking of?
I think doing it initially as a downstream only change where you query
RPM will work for now (perhaps by patching the way pip handles the case
where ENSUREPIP_OPTIONS is set?).
By the time this approach is posted upstream, then PEP 426/440 will
hopefully by Final and we can just use the metadata version field
directly rather than needing to grab the release increment from the RPM
repo. (I think this situation provides a good practical use case for why
it's important to standardise this feature upstream, too).
Cheers,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
7:09 a.m.
----- Original Message -----
On 12/12/2013 01:18 AM, Bohuslav Kabrda wrote:
> Well yeah, my point is that there is no upstream-acceptable way other than
> checking the file hashes by ensurepip, is there? If I wouldn't want to
> check file hashes, I'd have to query RPM for release - or is there some
> other way you're thinking of?
I think doing it initially as a downstream only change where you query
RPM will work for now (perhaps by patching the way pip handles the case
where ENSUREPIP_OPTIONS is set?).
By the time this approach is posted upstream, then PEP 426/440 will
hopefully by Final and we can just use the metadata version field
directly rather than needing to grab the release increment from the RPM
repo. (I think this situation provides a good practical use case for why
it's important to standardise this feature upstream, too).
So, getting back to Toshio's concern about sysadmins who just update'n'upload
files, the workflow for them would be "update files, bump build tag and then
upload", right?
What I mean is, this still has two solutions:
- checking the build tag (seems to be very simple to do)
- check the file hashes
Both solutions are IMO acceptable upstream (when we can actually do build tags), but my
question is: Which one is more likely to be accepted? I'm for "checking the build
tag", even if it means the extra step for sysadmins who will need to bump the build
tag when doing changes. (We may need to tell them to not bump the build tag "major
number", but add something to it like "1" => "1.1", since we
want distro package with "2" win over sysadmin's change.)
Does that make sense? What I'd love to hear is which of the two approaches is more
likely to get accepted upstream, so that I can concentrate on that one approach.
Thanks,
Slavek.
Cheers,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
--
Regards,
Bohuslav "Slavek" Kabrda.
4:45 p.m.
On 12/12/2013 11:09 PM, Bohuslav Kabrda wrote:
----- Original Message -----
> On 12/12/2013 01:18 AM, Bohuslav Kabrda wrote:
>> Well yeah, my point is that there is no upstream-acceptable way other than
>> checking the file hashes by ensurepip, is there? If I wouldn't want to
>> check file hashes, I'd have to query RPM for release - or is there some
>> other way you're thinking of?
>
> I think doing it initially as a downstream only change where you query
> RPM will work for now (perhaps by patching the way pip handles the case
> where ENSUREPIP_OPTIONS is set?).
>
> By the time this approach is posted upstream, then PEP 426/440 will
> hopefully by Final and we can just use the metadata version field
> directly rather than needing to grab the release increment from the RPM
> repo. (I think this situation provides a good practical use case for why
> it's important to standardise this feature upstream, too).
So, getting back to Toshio's concern about sysadmins who just update'n'upload
files, the workflow for them would be "update files, bump build tag and then
upload", right?
What I mean is, this still has two solutions:
- checking the build tag (seems to be very simple to do)
- check the file hashes
Both solutions are IMO acceptable upstream (when we can actually do build tags), but my
question is: Which one is more likely to be accepted? I'm for "checking the build
tag", even if it means the extra step for sysadmins who will need to bump the build
tag when doing changes. (We may need to tell them to not bump the build tag "major
number", but add something to it like "1" => "1.1", since we
want distro package with "2" win over sysadmin's change.)
Does that make sense? What I'd love to hear is which of the two approaches is more
likely to get accepted upstream, so that I can concentrate on that one approach.
I can't speak for the pip devs, but I think either would be acceptable
from an ensurepip point of view. That means the decision comes down to
how we want to deal with the "patch source in place" admins creating
their snowflake servers that can't easily be rebuilt from scratch:
- checking file hashes would mean that directly patching the system pip
not only affects future virtual environments, but also "python -m
ensurepip --upgrade" in existing virtual environments
- relying on the build tag would mean that directly patching the system
pip would affect future virtual environments, but *not* "python -m
ensurepip --upgrade" in existing virtual environments
I think the latter limitation is fine, and more in line with the
direction we're heading upstream (with the "local" suffix on build
numbers), so unless Toshio thinks we absolutely *have* to provide the
former behaviour, +1 for the build tag approach from me.
Cheers,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
3793
days inactive
3808
days old
python-devel@lists.fedoraproject.org
11 comments
3 participants
participants (3)
-
Bohuslav Kabrda -
Nick Coghlan -
Toshio Kuratomi