Patch for Python 3 Support

[Ralph Bean]

Hello all,
  I created a patch to update python-bugzilla to work on both python 2
  and python 3.  The unit tests all pass under both py2.7 and py3.2.
  I have tested the login and query commands by hand on py2.6, py2.7,
  and py3.2.

  http://threebean.org/patches/python-bugzilla-Python3-support.patch

  The patch is against the master branch (7ca8a60).  If I should
  recreate it against another branch, please let me know.

  I'm also willing to update and maintain a spec file supporting both
  versions if and when that time comes.

Cheers-
  -Ralph

[Toshio Kuratomi]

On Tue, Jun 26, 2012 at 12:32:38AM -0400, Ralph Bean wrote:
>  Hello all,
>    I created a patch to update python-bugzilla to work on both python 2
>    and python 3.  The unit tests all pass under both py2.7 and py3.2.
>    I have tested the login and query commands by hand on py2.6, py2.7,
>    and py3.2.
>  
>    http://threebean.org/patches/python-bugzilla-Python3-support.patch
>  
>    The patch is against the master branch (7ca8a60).  If I should
>    recreate it against another branch, please let me know.
>  
>    I'm also willing to update and maintain a spec file supporting both
>    versions if and when that time comes.
>   
Dependency notes (and note, these are just notes -- I don't know if the
project has defintions of what is acceptable)

* Needs python-2.6 or greater (as python-2.5 doesn't have from __future__
  import print_function, the new format of "except Foo as bar:", and
  relative imports)
* New dependency on python-six
* There are places that str(foo) is used that haven't been converted to
  a six.XXX() method.  These might fail in some circumstances.

* Note: the to_encoding() function looks to handle edge cases wrong.  This
  would become more apparent in python3 than in python2.  It looks like it's
  intended to take a unicode string or a byte string and return a byte
  string.  However, if the parameter passed in is neither a unicode nor
  a byte string, it will return an empty unicode string.  This will likely
  trigger tracebacks in python3 where bytes and unicode cannot be combined
  implictly.
* Note2: Some of the lines that use to_encoding() look like they can still
  throw exceptions.  they look like they can mix unicode and bytes in some
  circumstances.  Running with python-unicodenazi installed and enabled (see
  that package's documentation) might find some of these cases.


* This might better be rewritten:
         to_remove = [k for k in query.keys() if query[k] is None]
         for key in to_remove:
             del query[key]
  as this:
         query = dict(((k, v) for k, v  in six.iteritems(query) if  v is not None))

  which condenses two loops into one (Note that the generator expression there
  can be made into a list comprehension if compatibility with older python's
  is desired).

-Toshio

[Ralph Bean]

On Mon, Jun 25, 2012 at 10:39:48PM -0700, Toshio Kuratomi wrote:
>  * Needs python-2.6 or greater (as python-2.5 doesn't have from
> __future__
>    import print_function, the new format of "except Foo as bar:", and
>    relative imports) 
This is correct.  Can anyone comment on what python versions
python-bugzilla is required to support?  EPEL6 is on python-2.6 now.

>  * New dependency on python-six 
Correct.

>  * There are places that str(foo) is used that haven't been
> converted to
>    a six.XXX() method.  These might fail in some circumstances. 
I took on the remainder of these in one of the new patches listed below.

>  * Note: the to_encoding() function looks to handle edge cases wrong. 
> This
>    would become more apparent in python3 than in python2.  It looks like it's
>    intended to take a unicode string or a byte string and return a byte
>    string.  However, if the parameter passed in is neither a unicode nor
>    a byte string, it will return an empty unicode string.  This will likely
>    trigger tracebacks in python3 where bytes and unicode cannot be combined
>    implictly. 
Nice catch!  I fixed to_encoding so it always returns byte strings now.

>  * Note2: Some of the lines that use to_encoding() look like they can
> still
>    throw exceptions.  they look like they can mix unicode and bytes in some
>    circumstances.  Running with python-unicodenazi installed and enabled (see
>    that package's documentation) might find some of these cases. 
I went through all the uses of it and couldn't find anywhere where byte or
unicode strings are implicitly coerced.  I checked the fields in query
results and in every case I tested, they were unicode strings (thus
making it all consistent).  unicode-nazi only complained about some stuff in
stdlib.

>  * This might better be rewritten:
>           to_remove = [k for k in query.keys() if query[k] is None]
>           for key in to_remove:
>               del query[key]
>    as this:
>           query = dict(((k, v) for k, v  in six.iteritems(query) if  v is not None))
Done!  (The *original* loop modified the dict during iteration;
illegal in Python 3).

Here's my list of updated patches (including a rename of my original).
I kept them separated out instead of rebasing just to preserve
history and make them easier to cherry-pick.

  http://threebean.org/patches/python-bugzilla/0001-Python3-support.patch
 
http://threebean.org/patches/python-bugzilla/0002-Replace-str-foo-with-si...
 
http://threebean.org/patches/python-bugzilla/0003-Rewrite-two-loops-as-on...
  http://threebean.org/patches/python-bugzilla/0004-Regression-fix.patch
 
http://threebean.org/patches/python-bugzilla/0005-Always-return-bytestrin...

-Ralph

[Toshio Kuratomi]

On Tue, Jun 26, 2012 at 01:08:02PM -0400, Ralph Bean wrote:
>  On Mon, Jun 25, 2012 at 10:39:48PM -0700, Toshio Kuratomi wrote:
>  
>  Here's my list of updated patches (including a rename of my original).
>  I kept them separated out instead of rebasing just to preserve
>  history and make them easier to cherry-pick.
>  
>    http://threebean.org/patches/python-bugzilla/0001-Python3-support.patch
>   
> http://threebean.org/patches/python-bugzilla/0002-Replace-str-foo-with-si...
>   
> http://threebean.org/patches/python-bugzilla/0003-Rewrite-two-loops-as-on...
>    http://threebean.org/patches/python-bugzilla/0004-Regression-fix.patch
>   
> http://threebean.org/patches/python-bugzilla/0005-Always-return-bytestrin...
>   
These look good to me.

>  > * Needs python-2.6 or greater (as python-2.5 doesn't have
> from __future__
>  >   import print_function, the new format of "except Foo as bar:", and
>  >   relative imports)
>  
>  This is correct.  Can anyone comment on what python versions
>  python-bugzilla is required to support?  EPEL6 is on python-2.6 now.
>  
So this question and whether adding the dep on python-six is okay are the
questions that need answering.

-Toshio

[Ralph Bean]

On Tue, Jun 26, 2012 at 11:20:30AM -0700, Toshio Kuratomi wrote:
>  On Tue, Jun 26, 2012 at 01:08:02PM -0400, Ralph Bean wrote:
>  > On Mon, Jun 25, 2012 at 10:39:48PM -0700, Toshio Kuratomi wrote:
>  > 
>  > Here's my list of updated patches (including a rename of my original).
>  > I kept them separated out instead of rebasing just to preserve
>  > history and make them easier to cherry-pick.
>  > 
>  >   http://threebean.org/patches/python-bugzilla/0001-Python3-support.patch
>  >  
> http://threebean.org/patches/python-bugzilla/0002-Replace-str-foo-with-si...
>  >  
> http://threebean.org/patches/python-bugzilla/0003-Rewrite-two-loops-as-on...
>  >   http://threebean.org/patches/python-bugzilla/0004-Regression-fix.patch
>  >  
> http://threebean.org/patches/python-bugzilla/0005-Always-return-bytestrin...
>  > 
>  These look good to me.
>  
>  > > * Needs python-2.6 or greater (as python-2.5 doesn't have from __future__
>  > >   import print_function, the new format of "except Foo as bar:", and
>  > >   relative imports)
>  > 
>  > This is correct.  Can anyone comment on what python versions
>  > python-bugzilla is required to support?  EPEL6 is on python-2.6 now.
>  >
>  So this question and whether adding the dep on python-six is okay are the
>  questions that need answering. 
It has been over a month, so I'm bumping this discussion.

I understand in retrospect that I introduced a patch that could
potentially destabilize python-bugzilla thus giving everybody
nightmares after the BZ upgrade earlier in the Summer.

I think this work maintains all the backwards compatibility it can,
with exceptions to what Toshio has raised: no more python-2.5 and a
new dependency on python-six.

Someday, python2 will be unmaintained.  We won't be able to start
porting things to python3 that depend on python-bugzilla until it
itself is ported.  Better to get a head start.

[Ralph Bean]

On Tue, Jun 26, 2012 at 11:20:30AM -0700, Toshio Kuratomi wrote:
>  On Tue, Jun 26, 2012 at 01:08:02PM -0400, Ralph Bean wrote:
>  > On Mon, Jun 25, 2012 at 10:39:48PM -0700, Toshio Kuratomi wrote:
>  > 
>  > Here's my list of updated patches (including a rename of my original).
>  > I kept them separated out instead of rebasing just to preserve
>  > history and make them easier to cherry-pick.
>  > 
>  >   http://threebean.org/patches/python-bugzilla/0001-Python3-support.patch
>  >  
> http://threebean.org/patches/python-bugzilla/0002-Replace-str-foo-with-si...
>  >  
> http://threebean.org/patches/python-bugzilla/0003-Rewrite-two-loops-as-on...
>  >   http://threebean.org/patches/python-bugzilla/0004-Regression-fix.patch
>  >  
> http://threebean.org/patches/python-bugzilla/0005-Always-return-bytestrin...
>  > 
>  These look good to me.
>
>  > > * Needs python-2.6 or greater (as python-2.5 doesn't have from __future__
>  > >   import print_function, the new format of "except Foo as bar:", and
>  > >   relative imports)
>  > 
>  > This is correct.  Can anyone comment on what python versions
>  > python-bugzilla is required to support?  EPEL6 is on python-2.6 now.
>  >
>  So this question and whether adding the dep on python-six is okay are the
>  questions that need answering. 
It has been over a month, so I'm bumping this discussion.

I understand in retrospect that I introduced a patch that could
potentially destabilize python-bugzilla thus giving everybody
nightmares after the BZ upgrade earlier in the Summer.

I think this work maintains all the backwards compatibility it can,
with exceptions to what Toshio has raised: no more python-2.5 and a
new dependency on python-six.

Someday, python2 will be unmaintained.  We won't be able to start
porting things to python3 that depend on python-bugzilla until it
itself is ported.  Better to get a head start.

[Cole Robinson]

On 08/01/2012 08:55 AM, Ralph Bean wrote:
>  On Tue, Jun 26, 2012 at 11:20:30AM -0700, Toshio Kuratomi wrote:
> > On Tue, Jun 26, 2012 at 01:08:02PM -0400, Ralph Bean wrote:
> >> On Mon, Jun 25, 2012 at 10:39:48PM -0700, Toshio Kuratomi wrote:
> >>
> >> Here's my list of updated patches (including a rename of my original).
> >> I kept them separated out instead of rebasing just to preserve
> >> history and make them easier to cherry-pick.
> >>
> >>   http://threebean.org/patches/python-bugzilla/0001-Python3-support.patch
> >>  
> http://threebean.org/patches/python-bugzilla/0002-Replace-str-foo-with-si...
> >>  
> http://threebean.org/patches/python-bugzilla/0003-Rewrite-two-loops-as-on...
> >>   http://threebean.org/patches/python-bugzilla/0004-Regression-fix.patch
> >>  
> http://threebean.org/patches/python-bugzilla/0005-Always-return-bytestrin...
> >>
> > These look good to me.
> >
> >>> * Needs python-2.6 or greater (as python-2.5 doesn't have from
> __future__
> >>>   import print_function, the new format of "except Foo as bar:",
> and
> >>>   relative imports)
> >>
> >> This is correct.  Can anyone comment on what python versions
> >> python-bugzilla is required to support?  EPEL6 is on python-2.6 now.
> >>
> > So this question and whether adding the dep on python-six is okay are the
> > questions that need answering.
>  
>  It has been over a month, so I'm bumping this discussion.
>  
>  I understand in retrospect that I introduced a patch that could
>  potentially destabilize python-bugzilla thus giving everybody
>  nightmares after the BZ upgrade earlier in the Summer.
>  
>  I think this work maintains all the backwards compatibility it can,
>  with exceptions to what Toshio has raised: no more python-2.5 and a
>  new dependency on python-six.
>  
>  Someday, python2 will be unmaintained.  We won't be able to start
>  porting things to python3 that depend on python-bugzilla until it
>  itself is ported.  Better to get a head start. 
Hey Ralph, sorry for the general lack of response (excluding Toshio being
awesome).

Originally my thought was to cut another python-bugzilla release or two, with
a whole bunch more unit tests and squashing most of the reports in bugzilla.
That way we could push a more solid version out to epel 5, commit your patches
the next day, and then declare that we aren't ever rebasing again in epel 5.

And then I didn't find the time/energy/motivation to do anything with
python-bugzilla for months :)

During that time, epel5 /usr/bin/bugzilla has been completely busted due to a
python2.5-ism:

https://bugzilla.redhat.com/show_bug.cgi?id=854979

It took 3 months for anyone to file a bug which leads me to believe that there
aren't enough epel5 users to hold up taking this patch.

Ralph and/or Toshio, can you clarify a couple things for me?

1) How can I test future changes to make sure they work on both python2 and
python3?

2) If packaging this new version for epel6, I'd need to add a dep on python26
and python-six. How to ensure that python-bugzilla is only used with python26
in that case? (or does python26 basically replace the stock RHEL6 python version?)

Also cc-ing wwoods, the main man in python-bugzilla land. Will, any issues
with taking this patch soon-ish?

Thanks,
Cole

[Toshio Kuratomi]

On Thu, Oct 18, 2012 at 05:40:05PM -0400, Cole Robinson wrote:
>  
>  During that time, epel5 /usr/bin/bugzilla has been completely busted due to a
>  python2.5-ism:
>  
>  https://bugzilla.redhat.com/show_bug.cgi?id=854979
>  
>  It took 3 months for anyone to file a bug which leads me to believe that there
>  aren't enough epel5 users to hold up taking this patch.
>  
>  Ralph and/or Toshio, can you clarify a couple things for me?
>  
>  1) How can I test future changes to make sure they work on both python2 and
>  python3?
>   
I'll let ralph answer this one.

>  2) If packaging this new version for epel6, I'd need to add a dep
> on python26
>  and python-six. How to ensure that python-bugzilla is only used with python26
>  in that case? (or does python26 basically replace the stock RHEL6 python version?)
>  
For epel6, the system python is at version 2.6 already so no worries there.

For epel5, you'd need to package a separate python26-bugzilla package.  That
package will build and install into the python2.6/site-packages directory
and you'd change the requires to use python2.6 instead of python.

There's some information here: https://fedoraproject.org/wiki/Python26 and
I'd probably also take a look at some of the packages that are already
submitted.

>  Also cc-ing wwoods, the main man in python-bugzilla land. Will, any
> issues
>  with taking this patch soon-ish?
>  
>  Thanks,
>  Cole

[Ralph Bean]

Cole, thanks for looking into this.  

On Thu, Oct 18, 2012 at 05:40:05PM -0400, Cole Robinson wrote:
>  1) How can I test future changes to make sure they work on both
> python2 and
>  python3? 
While developing, I'll have two terminals open each with a different
virtualenv[1] activated: one for python-2.7 and the other for python-3.2.
I'll use nosy.py[2] in each terminal to re-run the test suite each
time I change a file.  This helps catch errors early.

A popular tool for testing a library against multiple python versions
is tox[3].  I have used it on other people's projects but haven't yet
taken the time to set it up on any of my own; it looks more robust than
my nosy.py setup, but I can't really vouch for it yet.

Lastly, I rely pretty heavily on external continuous integration
systems like travis-ci[4].  If python-bugzilla has %check enabled for
both python and python3-bugzilla, then koji can provide some
safeguards.  I think there is some activity in Fedora Infra to set up a
jenkins instance in our hosted cloud -- that can probably help with
cross-python testing soon, too.

-Ralph

[1] http://www.virtualenv.org
[2] http://github.com/ralphbean/threebean-dot-files/blob/master/content/bin/n...
[3] http://tox.testrun.org
[4] http://travis-ci.org

[Cole Robinson]

On 10/29/2012 09:49 AM, Ralph Bean wrote:
>  Cole, thanks for looking into this.  
>  
>  On Thu, Oct 18, 2012 at 05:40:05PM -0400, Cole Robinson wrote:
> > 1) How can I test future changes to make sure they work on both python2 and
> > python3?
>  
>  While developing, I'll have two terminals open each with a different
>  virtualenv[1] activated: one for python-2.7 and the other for python-3.2.
>  I'll use nosy.py[2] in each terminal to re-run the test suite each
>  time I change a file.  This helps catch errors early.
>  
>  A popular tool for testing a library against multiple python versions
>  is tox[3].  I have used it on other people's projects but haven't yet
>  taken the time to set it up on any of my own; it looks more robust than
>  my nosy.py setup, but I can't really vouch for it yet.
>  
>  Lastly, I rely pretty heavily on external continuous integration
>  systems like travis-ci[4].  If python-bugzilla has %check enabled for
>  both python and python3-bugzilla, then koji can provide some
>  safeguards.  I think there is some activity in Fedora Infra to set up a
>  jenkins instance in our hosted cloud -- that can probably help with
>  cross-python testing soon, too.
>  
>  -Ralph
>  
>  [1] http://www.virtualenv.org
>  [2] http://github.com/ralphbean/threebean-dot-files/blob/master/content/bin/n...
>  [3] http://tox.testrun.org
>  [4] http://travis-ci.org 
Thanks for the info Ralph. I still haven't forgotten about this.

However given that there's an RH bugzilla upgrade coming soon that will again
break the majority of python-bugzilla usage, I'm holding off on this work
until after the next release, since I still want to get an update into epel
for RHEL5.

Hopefully this is the last time an rhbz upgrade causes these types of
problems, since we are dropping most of the RH specific code from python-bugzilla.

- Cole

[Ralph Bean]

>  However given that there's an RH bugzilla upgrade coming soon
> that will again
>  break the majority of python-bugzilla usage, I'm holding off on this work
>  until after the next release, since I still want to get an update into epel
>  for RHEL5. 
No problem.  Thanks for keeping me updated!  :)

[Cole Robinson]

On 01/02/2013 08:16 AM, Cole Robinson wrote:
>  On 10/29/2012 09:49 AM, Ralph Bean wrote:
> > Cole, thanks for looking into this.  
> >
> > On Thu, Oct 18, 2012 at 05:40:05PM -0400, Cole Robinson wrote:
> >> 1) How can I test future changes to make sure they work on both python2 and
> >> python3?
> >
> > While developing, I'll have two terminals open each with a different
> > virtualenv[1] activated: one for python-2.7 and the other for python-3.2.
> > I'll use nosy.py[2] in each terminal to re-run the test suite each
> > time I change a file.  This helps catch errors early.
> >
> > A popular tool for testing a library against multiple python versions
> > is tox[3].  I have used it on other people's projects but haven't yet
> > taken the time to set it up on any of my own; it looks more robust than
> > my nosy.py setup, but I can't really vouch for it yet.
> >
> > Lastly, I rely pretty heavily on external continuous integration
> > systems like travis-ci[4].  If python-bugzilla has %check enabled for
> > both python and python3-bugzilla, then koji can provide some
> > safeguards.  I think there is some activity in Fedora Infra to set up a
> > jenkins instance in our hosted cloud -- that can probably help with
> > cross-python testing soon, too.
> >
> > -Ralph
> >
> > [1] http://www.virtualenv.org
> > [2] http://github.com/ralphbean/threebean-dot-files/blob/master/content/bin/n...
> > [3] http://tox.testrun.org
> > [4] http://travis-ci.org
>  
>  Thanks for the info Ralph. I still haven't forgotten about this.
>  
>  However given that there's an RH bugzilla upgrade coming soon that will again
>  break the majority of python-bugzilla usage, I'm holding off on this work
>  until after the next release, since I still want to get an update into epel
>  for RHEL5.
>  
>  Hopefully this is the last time an rhbz upgrade causes these types of
>  problems, since we are dropping most of the RH specific code from python-bugzilla.
>   
I think python-bugzilla is in pretty good shape after the current release. Any
future issues shouldn't be world breaking, so I'm fine with never rebasing
epel5 again. This opens the door for bumping the dep to python-2.7 and
committing python3 support.

TBH it's not a priority for me so I don't intend to actively work on it, but
if you revive the patches I'll be happy to review + push + handle the
packaging and testing side of things. Unfortunately the code has changed a
lot, so it may be easier to just reimplement the whole thing than to try and
get the old patch to apply.

Thanks,
Cole

[Ralph Bean]

On Tue, Jun 25, 2013 at 10:42:05AM -0400, Cole Robinson wrote:
>  I think python-bugzilla is in pretty good shape after the current
> release. Any
>  future issues shouldn't be world breaking, so I'm fine with never rebasing
>  epel5 again. This opens the door for bumping the dep to python-2.7 and
>  committing python3 support.
>  
>  TBH it's not a priority for me so I don't intend to actively work on it, but
>  if you revive the patches I'll be happy to review + push + handle the
>  packaging and testing side of things. Unfortunately the code has changed a
>  lot, so it may be easier to just reimplement the whole thing than to try and
>  get the old patch to apply. 
Cool.  It's not really a priority here either, just a side project.
I'll revive them some point in the next year and send them back to the
list.  Thanks again!