Python 3 discussion

[Orion Poplawski]

This is a followup to the EPEL IRC meeting discussion about python 3.  I had a
couple questions which led me to take Slavek's excellent work and try some
changes here: https://fedoraproject.org/wiki/User:Orion/EPEL7_Python3

Main ideas:
- (bikeshedding) - I didn't like the wording "other", so I went with "next".
- I didn't like having to toggle a macro in the spec files, I'm lazy
- What happens on the user's end?

So:

Lifecycle of python3X stacks, rebuilding:

when a new python3X+1 is built in EPEL - let's say that there is python34
and python35 has just been introduced:
        A new python3-pkgversion-macros is build defining the %python3_next*
macros.
        (scripted) mass rebuild is run to build as much of the new stack
possible automatically.
        At some point /usr/bin/python3 is switched from python34 to python35.
        at a certain point at time an announcement is made that python34 is to
be retired and python35 is to be *the* one. At this point:
            python3-pkgversion-macros is rebuilt removing the %python3_next*
macros.
            python35 is rebuilt defining the normal %python3_* macros
            all python34 packages are retired

At this point all packages build just python35-X subpackages

What I still don't understand is what this looks like on the user end, how do
they go from 34 to 35?  For app users (#!/usr/bin/python3), it seems like this
should be as automatic as possible.  So shouldn't they still use
/usr/bin/python3 rather than /usr/bin/python3X so they get updated automatically?

What about all of the old python34 packages left on their systems after
retirement?  Is there some way they can get cleaned up automatically?

[Dan Callaghan]

Excerpts from Orion Poplawski's message of 2015-02-28 04:36 +10:00:
>         all python34 packages are retired

Except there is no way to retire an individual subpackage, is there? 
Instead we are saying, the python34-* subpackage will just go away.
> What I still don't understand is what this looks like on the user end, 
> how do they go from 34 to 35?  For app users (#!/usr/bin/python3), it 
> seems like this should be as automatic as possible.  So shouldn't they 
> still use /usr/bin/python3 rather than /usr/bin/python3X so they get 
> updated automatically?
>
> What about all of the old python34 packages left on their systems 
> after retirement?  Is there some way they can get cleaned up 
> automatically?

Well I guess normally when a subpackage goes away (because it is being 
renamed or subsumed into some other package) the replacement would add 
Provides and Obsoletes to handle the upgrade path.

But I don't think we want the python35-* packages to Provide/Obsolete 
the python34-* packages because then everyone will be automatically 
upgraded to the 3.5 stack as soon as it appears, in which case why are 
we even bothering with python3x-* at all?

Under the current proposal every package with Python 3 dependencies 
would have to depend on a specific python3x-* package, so then it would 
be up to the maintainers of all those packages to manually bump their 
Requires from python34-* to python35-* at some point. Which, now that 
I think about it, is not that great. Even worse, if any packages form 
a transitive dependency chain then *all* packages in the chain have to 
update their Requires at the same time to avoid having a mix of 
python34-* and python35-* requirements.

So we probably need to make the python3x-* subpackages provide python3-* 
= %{version}-%{release}. Maybe Bohuslav already had this in mind, but 
it's not mentioned on the proposal page. So then packages would just 
require python3-* as they do in Fedora, and when the python35-* stack 
appears yum/dnf would just automatically upgrade. The /usr/bin/python3 
symlink means that everything will just keep working for applications.

On the other hand, if someone has explicitly installed a library (yum 
install python34-requests), for example because they are developing 
against it, then I guess they will left with the 3.4 build forever. It 
is up to them to install python35-requests if/when they are ready.

[Bohuslav Kabrda]

----- Original Message -----
> Excerpts from Orion Poplawski's message of 2015-02-28 04:36 +10:00:
> >         all python34 packages are retired
>
> Except there is no way to retire an individual subpackage, is there?
> Instead we are saying, the python34-* subpackage will just go away.

Yeah, I'm still not sure how this should be handled. Does anyone know whether the python34- subpackage will disappear from the repo if only python35- is built in a newer build? I'd tend to think so, since I think Koji builds repos from the newest builds, so if the package is rebuilt without python34- subpackage, it won't be there when the repo regenerates... But maybe I'm completely wrong here.
> > What I still don't understand is what this looks like on the user end,
> > how do they go from 34 to 35?  For app users (#!/usr/bin/python3), it
> > seems like this should be as automatic as possible.  So shouldn't they
> > still use /usr/bin/python3 rather than /usr/bin/python3X so they get
> > updated automatically?
> >
> > What about all of the old python34 packages left on their systems
> > after retirement?  Is there some way they can get cleaned up
> > automatically?
>
> Well I guess normally when a subpackage goes away (because it is being
> renamed or subsumed into some other package) the replacement would add
> Provides and Obsoletes to handle the upgrade path.
>
> But I don't think we want the python35-* packages to Provide/Obsolete
> the python34-* packages because then everyone will be automatically
> upgraded to the 3.5 stack as soon as it appears, in which case why are
> we even bothering with python3x-* at all?

Yeah, my intention was to build these two stacks without any conflicts/obsoletes/replacements.
> Under the current proposal every package with Python 3 dependencies
> would have to depend on a specific python3x-* package, so then it would
> be up to the maintainers of all those packages to manually bump their
> Requires from python34-* to python35-* at some point. Which, now that
> I think about it, is not that great. Even worse, if any packages form
> a transitive dependency chain then *all* packages in the chain have to
> update their Requires at the same time to avoid having a mix of
> python34-* and python35-* requirements.

Not really. The requires/buildrequires should be in form of
Requires: python%{python3_pkgversion}-six
so when we change %python3_pkgversion in the minimal buildroot, maintainer just rebuilds and gets updated requires.
> So we probably need to make the python3x-* subpackages provide python3-*
> = %{version}-%{release}. Maybe Bohuslav already had this in mind, but
> it's not mentioned on the proposal page. So then packages would just
> require python3-* as they do in Fedora, and when the python35-* stack
> appears yum/dnf would just automatically upgrade. The /usr/bin/python3
> symlink means that everything will just keep working for applications.
>
> On the other hand, if someone has explicitly installed a library (yum
> install python34-requests), for example because they are developing
> against it, then I guess they will left with the 3.4 build forever. It
> is up to them to install python35-requests if/when they are ready.

Yeah, I agree. We *will* make it possible to update with a simple rebuild as mentioned above, since we'll be able to control the value of %python3_pkgversion in the minimal buildroot.
> --
> Dan Callaghan 
> Software Engineer, Hosted & Shared Services
> Red Hat, Inc.

Slavek

[Dan Callaghan]

Excerpts from Bohuslav Kabrda's message of 2015-03-02 21:59 +10:00:
> ----- Original Message -----
> > Under the current proposal every package with Python 3 dependencies
> > would have to depend on a specific python3x-* package, so then it would
> > be up to the maintainers of all those packages to manually bump their
> > Requires from python34-* to python35-* at some point. Which, now that
> > I think about it, is not that great. Even worse, if any packages form
> > a transitive dependency chain then *all* packages in the chain have to
> > update their Requires at the same time to avoid having a mix of
> > python34-* and python35-* requirements.
>
> Not really. The requires/buildrequires should be in form of
> Requires: python%{python3_pkgversion}-six
> so when we change %python3_pkgversion in the minimal buildroot, 
> maintainer just rebuilds and gets updated requires.

Hmm okay. I didn't realise this.

So that means that:
* Fedora specfiles can't be used unchanged (they will require python3-*, 
  needs to have %{python3_pkgversion} macro inserted)
* applications will need to be rebuilt to pick up a change from 
  python34-* to python35-*
which is a bit unfortunate.

Is there any reason why we shouldn't just upgrade applications to the 
python35-* stack straight away, by providing python3-*?

[Bohuslav Kabrda]

----- Original Message -----
> Excerpts from Bohuslav Kabrda's message of 2015-03-02 21:59 +10:00:
> > ----- Original Message -----
> > > Under the current proposal every package with Python 3 dependencies
> > > would have to depend on a specific python3x-* package, so then it would
> > > be up to the maintainers of all those packages to manually bump their
> > > Requires from python34-* to python35-* at some point. Which, now that
> > > I think about it, is not that great. Even worse, if any packages form
> > > a transitive dependency chain then *all* packages in the chain have to
> > > update their Requires at the same time to avoid having a mix of
> > > python34-* and python35-* requirements.
> >
> > Not really. The requires/buildrequires should be in form of
> > Requires: python%{python3_pkgversion}-six
> > so when we change %python3_pkgversion in the minimal buildroot,
> > maintainer just rebuilds and gets updated requires.
>
> Hmm okay. I didn't realise this.
>
> So that means that:
> Fedora specfiles can't be used unchanged (they will require python3-*,
> needs to have %{python3_pkgversion} macro inserted)

True, but note that we'll make %python3_pkgversion macro available also in Fedora (always defined to "3"), so once this change is done, it'll be possible to have the same specfile both in Fedora and EPEL.
> applications will need to be rebuilt to pick up a change from
> python34-* to python35-*
>
> which is a bit unfortunate.
>
> Is there any reason why we shouldn't just upgrade applications to the
> python35-* stack straight away, by providing python3-*?

Yeah. First of all, it may happen that there are some minor backwards incompatibilities. Lots of packages run tests during buildtime, so these will be caught.
Another reason is that there are applications, which store files in /usr/lib/python3.X/site-packages - these need to be rebuilt anyway, since they have the Python minor version incorporated in path of files.
I really think that we should rebuild applications with new Python 3.X.

Slavek
> --
> Dan Callaghan 
> Software Engineer, Hosted & Shared Services
> Red Hat, Inc.

[Dan Callaghan]

Excerpts from Bohuslav Kabrda's message of 2015-03-02 22:17 +10:00:
> ----- Original Message -----
> > Excerpts from Bohuslav Kabrda's message of 2015-03-02 21:59 +10:00:
> > > ----- Original Message -----
> > > > Under the current proposal every package with Python 3 dependencies
> > > > would have to depend on a specific python3x-* package, so then it would
> > > > be up to the maintainers of all those packages to manually bump their
> > > > Requires from python34-* to python35-* at some point. Which, now that
> > > > I think about it, is not that great. Even worse, if any packages form
> > > > a transitive dependency chain then *all* packages in the chain have to
> > > > update their Requires at the same time to avoid having a mix of
> > > > python34-* and python35-* requirements.
> > >
> > > Not really. The requires/buildrequires should be in form of
> > > Requires: python%{python3_pkgversion}-six
> > > so when we change %python3_pkgversion in the minimal buildroot,
> > > maintainer just rebuilds and gets updated requires.
> >
> > Hmm okay. I didn't realise this.
> >
> > So that means that:
> > Fedora specfiles can't be used unchanged (they will require python3-*,
> > needs to have %{python3_pkgversion} macro inserted)
>
> True, but note that we'll make %python3_pkgversion macro available 
> also in Fedora (always defined to "3"), so once this change is done, 
> it'll be possible to have the same specfile both in Fedora and EPEL.

Okay that's good.
> > applications will need to be rebuilt to pick up a change from
> > python34-* to python35-*
> >
> > which is a bit unfortunate.
> >
> > Is there any reason why we shouldn't just upgrade applications to the
> > python35-* stack straight away, by providing python3-*?
>
> Yeah. First of all, it may happen that there are some minor backwards 
> incompatibilities. Lots of packages run tests during buildtime, so 
> these will be caught.
> Another reason is that there are applications, which store files in 
> /usr/lib/python3.X/site-packages - these need to be rebuilt anyway, 
> since they have the Python minor version incorporated in path of 
> files.
> I really think that we should rebuild applications with new Python 
> 3.X.

Well, they should really be using pkg_resources instead of hardcoding 
the path... but yes I take your point. Rebuilding for the newer Python 
stack makes sense.

We will need to make sure that setuptools-generated entry points have 
a shebang of /usr/bin/python3.4 rather than just /usr/bin/python3 
though, so that the scripts are always invoked with the same Python 
stack they are built for. Currently on Fedora they have 
/usr/bin/python3. This might need a patch to 
setuptools/distutils/whatever it is?

[Bohuslav Kabrda]

----- Original Message -----
<snip>
> > > applications will need to be rebuilt to pick up a change from
> > > python34-* to python35-*
> > >
> > > which is a bit unfortunate.
> > >
> > > Is there any reason why we shouldn't just upgrade applications to the
> > > python35-* stack straight away, by providing python3-*?
> >
> > Yeah. First of all, it may happen that there are some minor backwards
> > incompatibilities. Lots of packages run tests during buildtime, so
> > these will be caught.
> > Another reason is that there are applications, which store files in
> > /usr/lib/python3.X/site-packages - these need to be rebuilt anyway,
> > since they have the Python minor version incorporated in path of
> > files.
> > I really think that we should rebuild applications with new Python
> > 3.X.
>
> Well, they should really be using pkg_resources instead of hardcoding
> the path... but yes I take your point. Rebuilding for the newer Python
> stack makes sense.

There is no hardcoded path. The problem is that /usr/bin/python3.4 import modules from /usr/lib[64]/python3.4/site-packages. So when /usr/bin/python3 switches all of a sudden from python3.4 to python3.5, the app will try to import the module from /usr/lib[64]/python3.5, but it's not there. That's why I think we should use /usr/bin/python3.X rather than /usr/bin/python3.
> We will need to make sure that setuptools-generated entry points have
> a shebang of /usr/bin/python3.4 rather than just /usr/bin/python3
> though, so that the scripts are always invoked with the same Python
> stack they are built for. Currently on Fedora they have
> /usr/bin/python3. This might need a patch to
> setuptools/distutils/whatever it is?

Good point. I *think* setuptools uses the executable with which it was executed for hashbangs, so if you use "/usr/bin/python3.4 setup.py build", hashbangs will have "#!/usr/bin/python3.4". I have already modified the __python3 macro in my copr to point to /usr/bin/python3.4 instead of /usr/bin/python3, but I haven't rebuilt the other packages with it yet. I'm putting it on my TODO list to check this.
> --
> Dan Callaghan 
> Software Engineer, Hosted & Shared Services
> Red Hat, Inc.

Thanks a lot,
Slavek

[Orion Poplawski]

On 03/02/2015 05:11 AM, Dan Callaghan wrote:
> Is there any reason why we shouldn't just upgrade applications to the
> python35-* stack straight away, by providing python3-*?

The main issue here is that EPEL doesn't have releases, so there is no 
way to take time to build everything and then push it out as a group.

[Kevin Fenzi]

On Mon, 2 Mar 2015 06:59:29 -0500 (EST)
Bohuslav Kabrda  wrote:
> ----- Original Message -----
> > Excerpts from Orion Poplawski's message of 2015-02-28 04:36 +10:00:
> > >         all python34 packages are retired
> >
> > Except there is no way to retire an individual subpackage, is there?
> > Instead we are saying, the python34-* subpackage will just go away.
>
> Yeah, I'm still not sure how this should be handled. Does anyone know
> whether the python34- subpackage will disappear from the repo if only
> python35- is built in a newer build? I'd tend to think so, since I
> think Koji builds repos from the newest builds, so if the package is
> rebuilt without python34- subpackage, it won't be there when the repo
> regenerates... But maybe I'm completely wrong here.

It would. mash pulls the 'latest tagged' build from the tag. So, if the
newer one has different subpackages that will get used and the old one
is completely gone.

kevin

[Bohuslav Kabrda]

----- Original Message -----
> This is a followup to the EPEL IRC meeting discussion about python 3.  I had
> a
> couple questions which led me to take Slavek's excellent work and try some
> changes here: https://fedoraproject.org/wiki/User:Orion/EPEL7_Python3
>
> Main ideas:
> (bikeshedding) - I didn't like the wording "other", so I went with "next".

Right, this naming makes more sense due to the way your proposal works - in other words, "next" is always the higher version, assuming there are two parallel stacks. This isn't always the case in my proposal, which is why I used "other".
> I didn't like having to toggle a macro in the spec files, I'm lazy

+1, this can be done globally in python3-pkgversion-macros regardless which proposal we use.
> What happens on the user's end?
>
> So:
>
> Lifecycle of python3X stacks, rebuilding:
> when a new python3X+1 is built in EPEL - let's say that there is python34
>
> and python35 has just been introduced:
>         A new python3-pkgversion-macros is build defining the %python3_next*
> macros.

I think that macro file *in python35-devel* should define this - the "main" python3 defines %python3* macros, the "next"/"other" python3 defines %python3_next*.
>     (scripted) mass rebuild is run to build as much of the new stack
>
> possible automatically.
>         At some point /usr/bin/python3 is switched from python34 to python35.
>         at a certain point at time an announcement is made that python34 is
>         to
> be retired and python35 is to be *the* one. At this point:
>             python3-pkgversion-macros is rebuilt removing the %python3_next*
> macros.

As per my comment above, we wouldn't need to remove the "next" macros, we would rebuild python35 as "main" python3, thus giving it the normal %python3 macros.
>         python35 is rebuilt defining the normal %python3_* macros
>         all python34 packages are retired
>
>     At this point all packages build just python35-X subpackages
>
>
>
> What I still don't understand is what this looks like on the user end, how do
> they go from 34 to 35?  For app users (#!/usr/bin/python3), it seems like
> this
> should be as automatic as possible.  So shouldn't they still use
> /usr/bin/python3 rather than /usr/bin/python3X so they get updated
> automatically?

I think applications should use /usr/bin/python3.4 (at least packaged applications). Otherwise we could theoretically end up in a situation where /usr/bin/python3 is owned by python35, but the application RPM still has python34- dependencies and thus the app doesn't work.

I think this is actually one of the reasons why I thought it makes sense to keep both python34 and python35 living together in the state where python35 is the main python3 (having the default macros and owning /usr/bin/python3) and python34 is the "other". Let's take this example:
- there's an application "foo" written in python, which requires "six"
- it doesn't make sense to build the application for both python34 and python35, since it's an application, not a library
- this means it doesn't make sense for package maintainer to introduce the %python3_{other or next}* macros to the specfile, maintainer just wants to build with "the python3"
- so this means that we should do this:
-- python 3.5 is released, we build it in EPEL and turn with_python3_other to 1 in python3-pkgversion-macros
-- then there's a period when python34 and python35 coexist and python34 is "the main python" - in this period, *libraries* are rebuilt to provide both python34- and python35- subpackages
-- python34 and python35 are switched (the default macros now point to 35 and it also owns /usr/bin/python3 now)
-- then there's a period when python34 and python35 coexist and python35 is "the main python" - in this period, *applications* are rebuilt for python35 (may take some time, there will likely be a period when there are some apps on python34 and some already on python35)
-- when all applications are rebuilt for 35, with_python3_other is set to 0 and we now have just python35 and it's the "main" python3

Does this make sense or am I missing something? I'd need to do some minor changes+explanations to my proposal to accomodate this, but I still think it makes sense.
> What about all of the old python34 packages left on their systems after
> retirement?  Is there some way they can get cleaned up automatically?

I'm not sure about that... and I'm also not sure we want to do that, people may still want to keep these around for their own non-system applications to migrate.

Thanks a lot for this discussion,
Slavek
> --
> Orion Poplawski
> Technical Manager                     303-415-9701 x222
> NWRA, Boulder/CoRA Office             FAX: [phone]
> 3380 Mitchell Lane                       [e-mail]
> Boulder, CO 80301                   http://www.nwra.com

[Orion Poplawski]

On 03/02/2015 04:53 AM, Bohuslav Kabrda wrote:
> ----- Original Message -----
> > This is a followup to the EPEL IRC meeting discussion about python 3.  I had
> > a
> > couple questions which led me to take Slavek's excellent work and try some
> > changes here: https://fedoraproject.org/wiki/User:Orion/EPEL7_Python3
> >
> > Main ideas:
> > (bikeshedding) - I didn't like the wording "other", so I went with "next".
>
> Right, this naming makes more sense due to the way your proposal works - in other words, "next" is always the higher version, assuming there are two parallel stacks. This isn't always the case in my proposal, which is why I used "other".
> > I didn't like having to toggle a macro in the spec files, I'm lazy
>
> +1, this can be done globally in python3-pkgversion-macros regardless which proposal we use.

Agreed.
> > What happens on the user's end?
> >
> > So:
> >
> > Lifecycle of python3X stacks, rebuilding:
> >  when a new python3X+1 is built in EPEL - let's say that there is python34
> >
> > and python35 has just been introduced:
> >          A new python3-pkgversion-macros is build defining the %python3_next*
> > macros.
>
> I think that macro file *in python35-devel* should define this - the "main" python3 defines %python3* macros, the "next"/"other" python3 defines %python3_next*.

Sure - I guess I was using the presence of python3_next_pkgversion in 
python3-pkgversion-macros as the toggle.  But we could define 
with_python3_other directly there depending on what naming scheme is used.
> >      (scripted) mass rebuild is run to build as much of the new stack
> >
> > possible automatically.
> >          At some point /usr/bin/python3 is switched from python34 to python35.
> >          at a certain point at time an announcement is made that python34 is
> >          to
> > be retired and python35 is to be *the* one. At this point:
> >              python3-pkgversion-macros is rebuilt removing the %python3_next*
> > macros.
>
> As per my comment above, we wouldn't need to remove the "next" macros, we would rebuild python35 as "main" python3, thus giving it the normal %python3 macros.

I see:
- swapping python3_pkgversion and python3_other_pkgversion by rebuilding 
both 3X and 3X+1
- dropping 3X and renaming python3_next_* to python3_ in 3X+1

as functionally equivalent.  As I say, bikeshedding :)

-
> >          python35 is rebuilt defining the normal %python3_* macros
> >          all python34 packages are retired
> >
> >      At this point all packages build just python35-X subpackages
> >
> >
> >
> > What I still don't understand is what this looks like on the user end, how do
> > they go from 34 to 35?  For app users (#!/usr/bin/python3), it seems like
> > this
> > should be as automatic as possible.  So shouldn't they still use
> > /usr/bin/python3 rather than /usr/bin/python3X so they get updated
> > automatically?
>
> I think applications should use /usr/bin/python3.4 (at least packaged applications). Otherwise we could theoretically end up in a situation where /usr/bin/python3 is owned by python35, but the application RPM still has python34- dependencies and thus the app doesn't work.
>
> I think this is actually one of the reasons why I thought it makes sense to keep both python34 and python35 living together in the state where python35 is the main python3 (having the default macros and owning /usr/bin/python3) and python34 is the "other". Let's take this example:
> there's an application "foo" written in python, which requires "six"
> it doesn't make sense to build the application for both python34 and python35, since it's an application, not a library
> this means it doesn't make sense for package maintainer to introduce the %python3_{other or next}* macros to the specfile, maintainer just wants to build with "the python3"
> so this means that we should do this:
>
> -- python 3.5 is released, we build it in EPEL and turn with_python3_other to 1 in python3-pkgversion-macros
> -- then there's a period when python34 and python35 coexist and python34 is "the main python" - in this period, *libraries* are rebuilt to provide both python34- and python35- subpackages
> -- python34 and python35 are switched (the default macros now point to 35 and it also owns /usr/bin/python3 now)
> -- then there's a period when python34 and python35 coexist and python35 is "the main python" - in this period, *applications* are rebuilt for python35 (may take some time, there will likely be a period when there are some apps on python34 and some already on python35)
> -- when all applications are rebuilt for 35, with_python3_other is set to 0 and we now have just python35 and it's the "main" python3
>
> Does this make sense or am I missing something? I'd need to do some minor changes+explanations to my proposal to accomodate this, but I still think it makes sense.

Okay, a pure python app (named "app") that used /usr/bin/python3.4, 
would have to get rebuilt, version/release would get bumped, and pull in 
3.5.

But if it used /usr/bin/python3, there wouldn't be a need for a rebuild. 
  It would require /usr/bin/python3, and the installed python34 would 
always provide that, so it wouldn't migrate.  Now I get it.

Had to work that one out.
> > What about all of the old python34 packages left on their systems after
> > retirement?  Is there some way they can get cleaned up automatically?
>
> I'm not sure about that... and I'm also not sure we want to do that, people may still want to keep these around for their own non-system applications to migrate.
>
> Thanks a lot for this discussion,
> Slavek
> > --
> > Orion Poplawski
> > Technical Manager                     303-415-9701 x222
> > NWRA, Boulder/CoRA Office             FAX: [phone]
> > 3380 Mitchell Lane                       [e-mail]
> > Boulder, CO 80301                   http://www.nwra.com
>
> epel-devel mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/epel-devel

[Bohuslav Kabrda]

----- Original Message -----
<snip>
> > > What I still don't understand is what this looks like on the user end, how
> > > do
> > > they go from 34 to 35?  For app users (#!/usr/bin/python3), it seems like
> > > this
> > > should be as automatic as possible.  So shouldn't they still use
> > > /usr/bin/python3 rather than /usr/bin/python3X so they get updated
> > > automatically?
> >
> > I think applications should use /usr/bin/python3.4 (at least packaged
> > applications). Otherwise we could theoretically end up in a situation
> > where /usr/bin/python3 is owned by python35, but the application RPM still
> > has python34- dependencies and thus the app doesn't work.
> >
> > I think this is actually one of the reasons why I thought it makes sense to
> > keep both python34 and python35 living together in the state where
> > python35 is the main python3 (having the default macros and owning
> > /usr/bin/python3) and python34 is the "other". Let's take this example:
> > there's an application "foo" written in python, which requires "six"
> > it doesn't make sense to build the application for both python34 and
> >
> > python35, since it's an application, not a library
> > this means it doesn't make sense for package maintainer to introduce the
> >
> > %python3_{other or next}* macros to the specfile, maintainer just wants to
> > build with "the python3"
> > so this means that we should do this:
> >
> > -- python 3.5 is released, we build it in EPEL and turn with_python3_other
> > to 1 in python3-pkgversion-macros
> > -- then there's a period when python34 and python35 coexist and python34 is
> > "the main python" - in this period, *libraries* are rebuilt to provide
> > both python34- and python35- subpackages
> > -- python34 and python35 are switched (the default macros now point to 35
> > and it also owns /usr/bin/python3 now)
> > -- then there's a period when python34 and python35 coexist and python35 is
> > "the main python" - in this period, *applications* are rebuilt for
> > python35 (may take some time, there will likely be a period when there are
> > some apps on python34 and some already on python35)
> > -- when all applications are rebuilt for 35, with_python3_other is set to 0
> > and we now have just python35 and it's the "main" python3
> >
> > Does this make sense or am I missing something? I'd need to do some minor
> > changes+explanations to my proposal to accomodate this, but I still think
> > it makes sense.
>
> Okay, a pure python app (named "app") that used /usr/bin/python3.4,
> would have to get rebuilt, version/release would get bumped, and pull in
> 3.5.
>
> But if it used /usr/bin/python3, there wouldn't be a need for a rebuild.
>   It would require /usr/bin/python3, and the installed python34 would
> always provide that, so it wouldn't migrate.  Now I get it.
>
> Had to work that one out.

Good. I'll try to update my proposal to explain it better (if someone could suggest how to explain this nicely, I'd be happy to accept their wording ;))
> > > What about all of the old python34 packages left on their systems after
> > > retirement?  Is there some way they can get cleaned up automatically?
> >
> > I'm not sure about that... and I'm also not sure we want to do that, people
> > may still want to keep these around for their own non-system applications
> > to migrate.
> >
> > Thanks a lot for this discussion,
> > Slavek
> > > --
> > > Orion Poplawski
> > > Technical Manager                     303-415-9701 x222
> > > NWRA, Boulder/CoRA Office             FAX: [phone]
> > > 3380 Mitchell Lane                       [e-mail]
> > > Boulder, CO 80301                   http://www.nwra.com
>
> --
> Orion Poplawski
> Technical Manager                     303-415-9701 x222
> NWRA/CoRA Division                    FAX: [phone]
> 3380 Mitchell Lane                  [e-mail]
> Boulder, CO 80301              http://www.cora.nwra.com

Slavek

[Brian Stinson]

On Mar 02 06:53, Bohuslav Kabrda wrote:
> ----- Original Message -----

...SNIP...
> I think applications should use /usr/bin/python3.4 (at least packaged applications). Otherwise we could theoretically end up in a situation where /usr/bin/python3 is owned by python35, but the application RPM still has python34- dependencies and thus the app doesn't work.
>
> I think this is actually one of the reasons why I thought it makes sense to keep both python34 and python35 living together in the state where python35 is the main python3 (having the default macros and owning /usr/bin/python3) and python34 is the "other". Let's take this example:
> there's an application "foo" written in python, which requires "six"
> it doesn't make sense to build the application for both python34 and python35, since it's an application, not a library
> this means it doesn't make sense for package maintainer to introduce the %python3_{other or next}* macros to the specfile, maintainer just wants to build with "the python3"
> so this means that we should do this:
>
> -- python 3.5 is released, we build it in EPEL and turn with_python3_other to 1 in python3-pkgversion-macros
> -- then there's a period when python34 and python35 coexist and python34 is "the main python" - in this period, *libraries* are rebuilt to provide both python34- and python35- subpackages
> -- python34 and python35 are switched (the default macros now point to 35 and it also owns /usr/bin/python3 now)
> -- then there's a period when python34 and python35 coexist and python35 is "the main python" - in this period, *applications* are rebuilt for python35 (may take some time, there will likely be a period when there are some apps on python34 and some already on python35)

^ Is this step part of a coordinated mass-rebuild, or is this just a
period of time after we make the announcement: "Hey Packagers: be sure
to rebuild for python35"?

One alternative might be to do python35 in a side tag, after which we
could release the libraries and applications, and deprecate python34
together. This might take quite a bit more work though.
> -- when all applications are rebuilt for 35, with_python3_other is set to 0 and we now have just python35 and it's the "main" python3
>
> Does this make sense or am I missing something? I'd need to do some minor changes+explanations to my proposal to accomodate this, but I still think it makes sense.
> > What about all of the old python34 packages left on their systems after
> > retirement?  Is there some way they can get cleaned up automatically?
>
> I'm not sure about that... and I'm also not sure we want to do that, people may still want to keep these around for their own non-system applications to migrate.

Great discussion so far!

Brian

--
Brian Stinson
[e-mail] | IRC: bstinson | Bitbucket/Twitter: bstinsonmhk

[Bohuslav Kabrda]

----- Original Message -----
> On Mar 02 06:53, Bohuslav Kabrda wrote:
> > ----- Original Message -----
>
> ...SNIP...
> > I think applications should use /usr/bin/python3.4 (at least packaged
> > applications). Otherwise we could theoretically end up in a situation
> > where /usr/bin/python3 is owned by python35, but the application RPM still
> > has python34- dependencies and thus the app doesn't work.
> >
> > I think this is actually one of the reasons why I thought it makes sense to
> > keep both python34 and python35 living together in the state where
> > python35 is the main python3 (having the default macros and owning
> > /usr/bin/python3) and python34 is the "other". Let's take this example:
> > there's an application "foo" written in python, which requires "six"
> > it doesn't make sense to build the application for both python34 and
> >
> > python35, since it's an application, not a library
> > this means it doesn't make sense for package maintainer to introduce the
> >
> > %python3_{other or next}* macros to the specfile, maintainer just wants to
> > build with "the python3"
> > so this means that we should do this:
> >
> > -- python 3.5 is released, we build it in EPEL and turn with_python3_other
> > to 1 in python3-pkgversion-macros
> > -- then there's a period when python34 and python35 coexist and python34 is
> > "the main python" - in this period, *libraries* are rebuilt to provide
> > both python34- and python35- subpackages
> > -- python34 and python35 are switched (the default macros now point to 35
> > and it also owns /usr/bin/python3 now)
> > -- then there's a period when python34 and python35 coexist and python35 is
> > "the main python" - in this period, *applications* are rebuilt for
> > python35 (may take some time, there will likely be a period when there are
> > some apps on python34 and some already on python35)
>
> ^ Is this step part of a coordinated mass-rebuild, or is this just a
> period of time after we make the announcement: "Hey Packagers: be sure
> to rebuild for python35"?

That's a good question. I guess we should standardize how this will be done. Something like:
- there's an announcement on epel-devel that python35 is the "main python3" and packagers should rebuild their packagers (and users their apps/scripts/...)
- wait for a week (two?)
- open bugs for packages that haven't been rebuilt during the previous week and get them fixed ASAP
> One alternative might be to do python35 in a side tag, after which we
> could release the libraries and applications, and deprecate python34
> together. This might take quite a bit more work though.

+1, I don't want to introduce side tags to the process.
> > -- when all applications are rebuilt for 35, with_python3_other is set to 0
> > and we now have just python35 and it's the "main" python3
> >
> > Does this make sense or am I missing something? I'd need to do some minor
> > changes+explanations to my proposal to accomodate this, but I still think
> > it makes sense.
> > > What about all of the old python34 packages left on their systems after
> > > retirement?  Is there some way they can get cleaned up automatically?
> >
> > I'm not sure about that... and I'm also not sure we want to do that, people
> > may still want to keep these around for their own non-system applications
> > to migrate.
>
> Great discussion so far!
>
> Brian

Thanks,
Slavek
> --
> Brian Stinson
> [e-mail] | IRC: bstinson | Bitbucket/Twitter: bstinsonmhk

[Kevin Fenzi]

On Tue, 3 Mar 2015 07:46:00 -0500 (EST)
Bohuslav Kabrda  wrote:
> > ^ Is this step part of a coordinated mass-rebuild, or is this just a
> > period of time after we make the announcement: "Hey Packagers: be
> > sure to rebuild for python35"?
>
> That's a good question. I guess we should standardize how this will
> be done. Something like:
> there's an announcement on epel-devel that python35 is the "main
>
> python3" and packagers should rebuild their packagers (and users
> their apps/scripts/...)
> wait for a week (two?)
> open bugs for packages that haven't been rebuilt during the
>
> previous week and get them fixed ASAP

I'd say just doing the mass rebuild by a provenpackager or the like
would be easier than filing bugs and waiting.

Also, we need to coordinate the update with them in it... ideally they
would all be in the same update?

kevin

[Orion Poplawski]

On 03/03/2015 08:35 AM, Kevin Fenzi wrote:
> On Tue, 3 Mar 2015 07:46:00 -0500 (EST)
> Bohuslav Kabrda  wrote:
> > > ^ Is this step part of a coordinated mass-rebuild, or is this just a
> > > period of time after we make the announcement: "Hey Packagers: be
> > > sure to rebuild for python35"?
> >
> > That's a good question. I guess we should standardize how this will
> > be done. Something like:
> > there's an announcement on epel-devel that python35 is the "main
> >
> > python3" and packagers should rebuild their packagers (and users
> > their apps/scripts/...)
> > wait for a week (two?)
> > open bugs for packages that haven't been rebuilt during the
> >
> > previous week and get them fixed ASAP
>
> I'd say just doing the mass rebuild by a provenpackager or the like
> would be easier than filing bugs and waiting.

Yeah, just do the builds and be done with it.  Will need to get built in dep
order.
> Also, we need to coordinate the update with them in it... ideally they
> would all be in the same update?

That's seem sure to crash bodhi :)

As long as deps are pushed out in the right order things would be okay.

[Bohuslav Kabrda]

----- Original Message -----
> On 03/03/2015 08:35 AM, Kevin Fenzi wrote:
> > On Tue, 3 Mar 2015 07:46:00 -0500 (EST)
> > Bohuslav Kabrda  wrote:
> > > > ^ Is this step part of a coordinated mass-rebuild, or is this just a
> > > > period of time after we make the announcement: "Hey Packagers: be
> > > > sure to rebuild for python35"?
> > >
> > > That's a good question. I guess we should standardize how this will
> > > be done. Something like:
> > > there's an announcement on epel-devel that python35 is the "main
> > >
> > > python3" and packagers should rebuild their packagers (and users
> > > their apps/scripts/...)
> > > wait for a week (two?)
> > > open bugs for packages that haven't been rebuilt during the
> > >
> > > previous week and get them fixed ASAP
> >
> > I'd say just doing the mass rebuild by a provenpackager or the like
> > would be easier than filing bugs and waiting.
>
> Yeah, just do the builds and be done with it.  Will need to get built in dep
> order.

My concern is that in some cases we may actually need for upstream to release supported version for the new Python release. However much Python upstream is cautious about not introducing regressions, from time to time a small backwards incompatibility change can crawl in and cause problems to certain packages that rely on the changed code.
> > Also, we need to coordinate the update with them in it... ideally they
> > would all be in the same update?
>
> That's seem sure to crash bodhi :)
>
> As long as deps are pushed out in the right order things would be okay.

Yeah, ideally it should work like that, but in some cases we may want to just push everything that has been rebuilt to bodhi and not wait for one or two laggards.

Slavek
> --
> Orion Poplawski
> Technical Manager                     303-415-9701 x222
> NWRA, Boulder/CoRA Office             FAX: [phone]
> 3380 Mitchell Lane                       [e-mail]
> Boulder, CO 80301                   http://www.nwra.com

[Bohuslav Kabrda]

Today, I've made some changes to the proposal to accomodate comments from the previous meeting and from discussion on this list. The diff is here:

https://fedoraproject.org/w/index.php?title=User%3ABkabrda%2FEPEL7_Python3&a...

I'd appreciate comments, I hope I made it clearer and more explanatory.

Thanks for your comments,
Slavek

[Kevin Fenzi]

On Tue, 3 Mar 2015 08:31:20 -0500 (EST)
Bohuslav Kabrda  wrote:
> Today, I've made some changes to the proposal to accomodate comments
> from the previous meeting and from discussion on this list. The diff
> is here:
>
> https://fedoraproject.org/w/index.php?title=User%3ABkabrda%2FEPEL7_Python3&a...
>
> I'd appreciate comments, I hope I made it clearer and more
> explanatory.

Overall I like it. I suspect we will run into things and need to fine
tune it, but thats normal and good. ;)

kevin

[Orion Poplawski]

On 03/03/2015 06:31 AM, Bohuslav Kabrda wrote:
> Today, I've made some changes to the proposal to accomodate comments from the previous meeting and from discussion on this list. The diff is here:
>
> https://fedoraproject.org/w/index.php?title=User%3ABkabrda%2FEPEL7_Python3&a...
>
> I'd appreciate comments, I hope I made it clearer and more explanatory.
>
> Thanks for your comments,
> Slavek
> _______________________________________________

Can we add the proper _sitelib / _sitearch macros to the example spec file?
Thanks.

[Bohuslav Kabrda]

----- Original Message -----
> On 03/03/2015 06:31 AM, Bohuslav Kabrda wrote:
> > Today, I've made some changes to the proposal to accomodate comments from
> > the previous meeting and from discussion on this list. The diff is here:
> >
> > https://fedoraproject.org/w/index.php?title=User%3ABkabrda%2FEPEL7_Python3&a...
> >
> > I'd appreciate comments, I hope I made it clearer and more explanatory.
> >
> > Thanks for your comments,
> > Slavek
> > _______________________________________________
>
> Can we add the proper _sitelib / _sitearch macros to the example spec file?
> Thanks.

Done. I added one more change that's not mentioned in the yesterday's diff, so here's are all the changes since the last diff I sent:
https://fedoraproject.org/w/index.php?title=User%3ABkabrda%2FEPEL7_Python3&a...
(I made the example specfile look like a full example, I added rules for files in /usr/bin and I also included a definition of "libraries" and "applications" - the two package types I think we need to distinguish)

Slavek
> --
> Orion Poplawski
> Technical Manager                     303-415-9701 x222
> NWRA, Boulder/CoRA Office             FAX: [phone]
> 3380 Mitchell Lane                       [e-mail]
> Boulder, CO 80301                   http://www.nwra.com