[BUG] Server error: buf must be a byte string

[Jani Nikula]

The command-line client fails with python 2.7.8 as follows:

Traceback (most recent call last):
  File "/usr/local/bin/bugzilla", line 1253, in <module>
    main()
  File "/usr/local/bin/bugzilla", line 1212, in main
    _do_info(bz, opt)
  File "/usr/local/bin/bugzilla", line 728, in _do_info
    products = bz.getproducts(include_fields=include_fields)
  File "/usr/local/lib/python2.7/dist-packages/bugzilla/base.py", line 757, in
getproducts
    self._products = self._getproducts(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/bugzilla/base.py", line 891, in
_getproducts
    product_ids = self._proxy.Product.get_accessible_products()
  File "/usr/lib/python2.7/xmlrpclib.py", line 1224, in __call__
    return self.__send(self.__name, args)
  File "/usr/local/lib/python2.7/dist-packages/bugzilla/base.py", line 168, in
_ServerProxy__request
    ret = ServerProxy._ServerProxy__request(self, methodname, params)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1578, in __request
    verbose=self.__verbose
  File "/usr/local/lib/python2.7/dist-packages/bugzilla/base.py", line 263, in
request
    return self._request_helper(url, request_body)
  File "/usr/local/lib/python2.7/dist-packages/bugzilla/base.py", line 250, in
_request_helper
    raise e
BugzillaError: buf must be a byte string

Server error: buf must be a byte string

I've bisected this to 

commit e3f17d7db5907ea629f31fbc892576fdf9ccec4e
Author: Josef Stribny <jstribny(a)redhat.com&gt;
Date:   Tue Jun 3 15:27:08 2014 +0200

    Convert request_body to string for Python 3

and confirmed that reverting the commit from current master fixes the
issue.

BR,
Jani.

-- 
Jani Nikula, Intel Open Source Technology Center

[Cole Robinson]

On 11/14/2014 04:15 AM, Jani Nikula wrote:
>  The command-line client fails with python 2.7.8 as follows:
>
>  Traceback (most recent call last):
>     File "/usr/local/bin/bugzilla", line 1253, in <module>
>       main()
>     File "/usr/local/bin/bugzilla", line 1212, in main
>       _do_info(bz, opt)
>     File "/usr/local/bin/bugzilla", line 728, in _do_info
>       products = bz.getproducts(include_fields=include_fields)
>     File "/usr/local/lib/python2.7/dist-packages/bugzilla/base.py", line 757, in
> getproducts
>       self._products = self._getproducts(**kwargs)
>     File "/usr/local/lib/python2.7/dist-packages/bugzilla/base.py", line 891, in
> _getproducts
>       product_ids = self._proxy.Product.get_accessible_products()
>     File "/usr/lib/python2.7/xmlrpclib.py", line 1224, in __call__
>       return self.__send(self.__name, args)
>     File "/usr/local/lib/python2.7/dist-packages/bugzilla/base.py", line 168, in
> _ServerProxy__request
>       ret = ServerProxy._ServerProxy__request(self, methodname, params)
>     File "/usr/lib/python2.7/xmlrpclib.py", line 1578, in __request
>       verbose=self.__verbose
>     File "/usr/local/lib/python2.7/dist-packages/bugzilla/base.py", line 263, in
> request
>       return self._request_helper(url, request_body)
>     File "/usr/local/lib/python2.7/dist-packages/bugzilla/base.py", line 250, in
> _request_helper
>       raise e
>  BugzillaError: buf must be a byte string
>
>  Server error: buf must be a byte string
>
>  I've bisected this to
>
>  commit e3f17d7db5907ea629f31fbc892576fdf9ccec4e
>  Author: Josef Stribny <jstribny(a)redhat.com&gt;
>  Date:   Tue Jun 3 15:27:08 2014 +0200
>
>       Convert request_body to string for Python 3
>
>  and confirmed that reverting the commit from current master fixes the
>  issue.
>
>  BR,
>  Jani.
>  
Thanks for the report. Is this against bugzilla.redhat.com? If not, is it 
against a public bugzilla instance I can test against? And if so, can you give 
an example command line that reproduces?

- Cole

[Jani Nikula]

On Sun, 16 Nov 2014, Cole Robinson <crobinso(a)redhat.com&gt; wrote:
>  On 11/14/2014 04:15 AM, Jani Nikula wrote:
> >
> > The command-line client fails with python 2.7.8 as follows:
> >
> > Traceback (most recent call last):
> >    File "/usr/local/bin/bugzilla", line 1253, in <module>
> >      main()
> >    File "/usr/local/bin/bugzilla", line 1212, in main
> >      _do_info(bz, opt)
> >    File "/usr/local/bin/bugzilla", line 728, in _do_info
> >      products = bz.getproducts(include_fields=include_fields)
> >    File "/usr/local/lib/python2.7/dist-packages/bugzilla/base.py", line
> 757, in getproducts
> >      self._products = self._getproducts(**kwargs)
> >    File "/usr/local/lib/python2.7/dist-packages/bugzilla/base.py", line
> 891, in _getproducts
> >      product_ids = self._proxy.Product.get_accessible_products()
> >    File "/usr/lib/python2.7/xmlrpclib.py", line 1224, in __call__
> >      return self.__send(self.__name, args)
> >    File "/usr/local/lib/python2.7/dist-packages/bugzilla/base.py", line
> 168, in _ServerProxy__request
> >      ret = ServerProxy._ServerProxy__request(self, methodname, params)
> >    File "/usr/lib/python2.7/xmlrpclib.py", line 1578, in __request
> >      verbose=self.__verbose
> >    File "/usr/local/lib/python2.7/dist-packages/bugzilla/base.py", line
> 263, in request
> >      return self._request_helper(url, request_body)
> >    File "/usr/local/lib/python2.7/dist-packages/bugzilla/base.py", line
> 250, in _request_helper
> >      raise e
> > BugzillaError: buf must be a byte string
> >
> > Server error: buf must be a byte string
> >
> > I've bisected this to
> >
> > commit e3f17d7db5907ea629f31fbc892576fdf9ccec4e
> > Author: Josef Stribny <jstribny(a)redhat.com&gt;
> > Date:   Tue Jun 3 15:27:08 2014 +0200
> >
> >      Convert request_body to string for Python 3
> >
> > and confirmed that reverting the commit from current master fixes the
> > issue.
> >
> > BR,
> > Jani.
> >
>
>  Thanks for the report. Is this against bugzilla.redhat.com? If not, is it 
>  against a public bugzilla instance I can test against? And if so, can you give 
>  an example command line that reproduces? 
Seems to be reproducible with simply:

$ bugzilla info -p
$ bugzilla --bugzilla=https://bugs.freedesktop.org/xmlrpc.cgi info -p
$ bugzilla --bugzilla=https://bugzilla.kernel.org/xmlrpc.cgi info -p

This is behind a proxy; haven't had a chance to try without yet.

(Not sure if it's related in any way, but some of my scripts doing login
and bug modification seem to be broken as well. Debugging saga
continues.)

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

[Arun Babu Neelicattu]

Jani; for what it is worth, could not reproduce using Python 2.7.5, 2.7.8 and 3.3.2. Tried
using both requests 1.x and 2.x too. This might have to do with the proxy issues requests
had via urllib3. Think it should be fixed in 2.x.

https://github.com/kennethreitz/requests/pull/478
https://github.com/shazow/urllib3/pull/68

Without knowing your environment that is my best guess. Can you provide what version of
python-bugzilla and requests you are using? And probably will be good to see what happens
without the proxy.

-arun


----- Original Message -----
>  From: "Jani Nikula" <jani.nikula(a)intel.com&gt;
>  To: "Cole Robinson" <crobinso(a)redhat.com&gt;,
> python-bugzilla(a)lists.fedorahosted.org
>  Cc: "Josef Stribny" <jstribny(a)redhat.com&gt;
>  Sent: Monday, November 17, 2014 11:21:58 PM
>  Subject: Re: [python-bugzilla] [BUG] Server error: buf must be a byte string
>  
>  On Sun, 16 Nov 2014, Cole Robinson <crobinso(a)redhat.com&gt; wrote:
>  > On 11/14/2014 04:15 AM, Jani Nikula wrote:
>  >>
>  >> The command-line client fails with python 2.7.8 as follows:
>  >>
>  >> Traceback (most recent call last):
>  >>    File "/usr/local/bin/bugzilla", line 1253, in <module>
>  >>      main()
>  >>    File "/usr/local/bin/bugzilla", line 1212, in main
>  >>      _do_info(bz, opt)
>  >>    File "/usr/local/bin/bugzilla", line 728, in _do_info
>  >>      products = bz.getproducts(include_fields=include_fields)
>  >>    File "/usr/local/lib/python2.7/dist-packages/bugzilla/base.py",
> line
>  >>    757, in getproducts
>  >>      self._products = self._getproducts(**kwargs)
>  >>    File "/usr/local/lib/python2.7/dist-packages/bugzilla/base.py",
> line
>  >>    891, in _getproducts
>  >>      product_ids = self._proxy.Product.get_accessible_products()
>  >>    File "/usr/lib/python2.7/xmlrpclib.py", line 1224, in __call__
>  >>      return self.__send(self.__name, args)
>  >>    File "/usr/local/lib/python2.7/dist-packages/bugzilla/base.py",
> line
>  >>    168, in _ServerProxy__request
>  >>      ret = ServerProxy._ServerProxy__request(self, methodname, params)
>  >>    File "/usr/lib/python2.7/xmlrpclib.py", line 1578, in __request
>  >>      verbose=self.__verbose
>  >>    File "/usr/local/lib/python2.7/dist-packages/bugzilla/base.py",
> line
>  >>    263, in request
>  >>      return self._request_helper(url, request_body)
>  >>    File "/usr/local/lib/python2.7/dist-packages/bugzilla/base.py",
> line
>  >>    250, in _request_helper
>  >>      raise e
>  >> BugzillaError: buf must be a byte string
>  >>
>  >> Server error: buf must be a byte string
>  >>
>  >> I've bisected this to
>  >>
>  >> commit e3f17d7db5907ea629f31fbc892576fdf9ccec4e
>  >> Author: Josef Stribny <jstribny(a)redhat.com&gt;
>  >> Date:   Tue Jun 3 15:27:08 2014 +0200
>  >>
>  >>      Convert request_body to string for Python 3
>  >>
>  >> and confirmed that reverting the commit from current master fixes the
>  >> issue.
>  >>
>  >> BR,
>  >> Jani.
>  >>
>  >
>  > Thanks for the report. Is this against bugzilla.redhat.com? If not, is it
>  > against a public bugzilla instance I can test against? And if so, can you
>  > give
>  > an example command line that reproduces?
>  
>  Seems to be reproducible with simply:
>  
>  $ bugzilla info -p
>  $ bugzilla --bugzilla=https://bugs.freedesktop.org/xmlrpc.cgi info -p
>  $ bugzilla --bugzilla=https://bugzilla.kernel.org/xmlrpc.cgi info -p
>  
>  This is behind a proxy; haven't had a chance to try without yet.
>  
>  (Not sure if it's related in any way, but some of my scripts doing login
>  and bug modification seem to be broken as well. Debugging saga
>  continues.)
>  
>  BR,
>  Jani.
>  
>  
>  --
>  Jani Nikula, Intel Open Source Technology Center
>  _______________________________________________
>  python-bugzilla mailing list
>  python-bugzilla(a)lists.fedorahosted.org
>  https://lists.fedorahosted.org/mailman/listinfo/python-bugzilla
>

[Jani Nikula]

On Mon, 17 Nov 2014, Arun Babu Neelicattu <abn(a)redhat.com&gt; wrote:
>  Jani; for what it is worth, could not reproduce using Python 2.7.5,
> 2.7.8 and 3.3.2. Tried using both requests 1.x and 2.x too. This might have to do with the
> proxy issues requests had via urllib3. Think it should be fixed in 2.x.
>
>  https://github.com/kennethreitz/requests/pull/478
>  https://github.com/shazow/urllib3/pull/68
>
>  Without knowing your environment that is my best guess. Can you
>  provide what version of python-bugzilla and requests you are using?
>  And probably will be good to see what happens without the proxy. 
I ruled out proxy, same thing without.

python-bugzilla is current git master.

The dependencies are from Debian testing, python-requests version
2.4.3-2 and python-urllib3 version 1.9.1-2.

BR,
Jani.



-- 
Jani Nikula, Intel Open Source Technology Center

[Cole Robinson]

On 11/17/2014 09:37 AM, Jani Nikula wrote:
>  On Mon, 17 Nov 2014, Arun Babu Neelicattu <abn(a)redhat.com&gt;
> wrote:
> > Jani; for what it is worth, could not reproduce using Python 2.7.5, 2.7.8 and 3.3.2.
> Tried using both requests 1.x and 2.x too. This might have to do with the proxy issues
> requests had via urllib3. Think it should be fixed in 2.x.
> >
> > https://github.com/kennethreitz/requests/pull/478
> > https://github.com/shazow/urllib3/pull/68
> >
> > Without knowing your environment that is my best guess. Can you
> > provide what version of python-bugzilla and requests you are using?
> > And probably will be good to see what happens without the proxy.
>
>  I ruled out proxy, same thing without.
>
>  python-bugzilla is current git master.
>
>  The dependencies are from Debian testing, python-requests version
>  2.4.3-2 and python-urllib3 version 1.9.1-2.
>
>  BR,
>  Jani.
>
>
>  
I see you've installed bugzilla locally, maybe try running directly from the 
git checkout using ./bugzilla-cli, might rule out potential installation issues

If you get the same result, stick a 'print requests' in bugzilla/base.py and 
retry, ensure you are using the expected requests module path. I've hit issues 
before with having an older version of a library somewhere in pythonpath

FWIW the bug that came with the python-bugzilla commit you identified (I 
should have stuck this in the changelog): 
https://bugzilla.redhat.com/show_bug.cgi?id=1104197

Should be easy enough to conditionalize that bit on older python-requests, but 
I'd still like to figure out what's going on first.

- Cole

[Arun Babu Neelicattu]

Hmm, reproduced this on a debian:testing container. The core issue seems to be from
pyopenssl. Based on [1] this is being handled at [2]. The expected fix is in pyopenssl
0.15 (unreleased).

  File "/tmp/python-bugzilla/bugzilla/base.py", line 223, in _request_helper
    url, data=request_body, **self.request_defaults)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 94, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 49, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 457, in
request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 569, in
send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 362, in
send
    timeout=timeout
  File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 516,
in urlopen
    body=body, headers=headers)
  File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 308,
in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python2.7/httplib.py", line 1001, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1035, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 997, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 854, in _send_output
    self.send(message_body)
  File "/usr/lib/python2.7/httplib.py", line 826, in send
    self.sock.sendall(data)
  File "/usr/lib/python2.7/dist-packages/urllib3/contrib/pyopenssl.py", line
203, in sendall
    return self.connection.sendall(data)
  File "/usr/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 969, in sendall
    raise TypeError("buf must be a byte string")

[1] https://github.com/kennethreitz/requests/issues/2113
[2] https://github.com/pyca/pyopenssl/issues/15

----- Original Message -----
>  From: "Cole Robinson" <crobinso(a)redhat.com&gt;
>  To: "Jani Nikula" <jani.nikula(a)intel.com&gt;, "Arun Babu
> Neelicattu" <abn(a)redhat.com&gt;
>  Cc: python-bugzilla(a)lists.fedorahosted.org, "Josef Stribny"
> <jstribny(a)redhat.com&gt;
>  Sent: Tuesday, November 18, 2014 2:43:46 AM
>  Subject: Re: [python-bugzilla] [BUG] Server error: buf must be a byte string
>  
>  On 11/17/2014 09:37 AM, Jani Nikula wrote:
>  > On Mon, 17 Nov 2014, Arun Babu Neelicattu <abn(a)redhat.com&gt; wrote:
>  >> Jani; for what it is worth, could not reproduce using Python 2.7.5, 2.7.8
>  >> and 3.3.2. Tried using both requests 1.x and 2.x too. This might have to
>  >> do with the proxy issues requests had via urllib3. Think it should be
>  >> fixed in 2.x.
>  >>
>  >> https://github.com/kennethreitz/requests/pull/478
>  >> https://github.com/shazow/urllib3/pull/68
>  >>
>  >> Without knowing your environment that is my best guess. Can you
>  >> provide what version of python-bugzilla and requests you are using?
>  >> And probably will be good to see what happens without the proxy.
>  >
>  > I ruled out proxy, same thing without.
>  >
>  > python-bugzilla is current git master.
>  >
>  > The dependencies are from Debian testing, python-requests version
>  > 2.4.3-2 and python-urllib3 version 1.9.1-2.
>  >
>  > BR,
>  > Jani.
>  >
>  >
>  >
>  
>  I see you've installed bugzilla locally, maybe try running directly from the
>  git checkout using ./bugzilla-cli, might rule out potential installation
>  issues
>  
>  If you get the same result, stick a 'print requests' in bugzilla/base.py and
>  retry, ensure you are using the expected requests module path. I've hit
>  issues
>  before with having an older version of a library somewhere in pythonpath
>  
>  FWIW the bug that came with the python-bugzilla commit you identified (I
>  should have stuck this in the changelog):
>  https://bugzilla.redhat.com/show_bug.cgi?id=1104197
>  
>  Should be easy enough to conditionalize that bit on older python-requests,
>  but
>  I'd still like to figure out what's going on first.
>  
>  - Cole
>

[Arun Babu Neelicattu]

fwiw, the issue is not reproducible on fedora as f20 ships pyOpenSSL 0.13.1 while debial
testing ships 0.14.

----- Original Message -----
>  From: "Arun Babu Neelicattu" <abn(a)redhat.com&gt;
>  To: "Cole Robinson" <crobinso(a)redhat.com&gt;
>  Cc: "Jani Nikula" <jani.nikula(a)intel.com&gt;,
> python-bugzilla(a)lists.fedorahosted.org, "Josef Stribny"
>  <jstribny(a)redhat.com&gt;
>  Sent: Tuesday, November 18, 2014 4:03:53 AM
>  Subject: Re: [python-bugzilla] [BUG] Server error: buf must be a byte string
>  
>  Hmm, reproduced this on a debian:testing container. The core issue seems to
>  be from pyopenssl. Based on [1] this is being handled at [2]. The expected
>  fix is in pyopenssl 0.15 (unreleased).
>  
>    File "/tmp/python-bugzilla/bugzilla/base.py", line 223, in _request_helper
>      url, data=request_body, **self.request_defaults)
>    File "/usr/lib/python2.7/dist-packages/requests/api.py", line 94, in post
>      return request('post', url, data=data, json=json, **kwargs)
>    File "/usr/lib/python2.7/dist-packages/requests/api.py", line 49, in
>    request
>      return session.request(method=method, url=url, **kwargs)
>    File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 457, in
>    request
>      resp = self.send(prep, **send_kwargs)
>    File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 569, in
>    send
>      r = adapter.send(request, **kwargs)
>    File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 362, in
>    send
>      timeout=timeout
>    File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line
>    516, in urlopen
>      body=body, headers=headers)
>    File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line
>    308, in _make_request
>      conn.request(method, url, **httplib_request_kw)
>    File "/usr/lib/python2.7/httplib.py", line 1001, in request
>      self._send_request(method, url, body, headers)
>    File "/usr/lib/python2.7/httplib.py", line 1035, in _send_request
>      self.endheaders(body)
>    File "/usr/lib/python2.7/httplib.py", line 997, in endheaders
>      self._send_output(message_body)
>    File "/usr/lib/python2.7/httplib.py", line 854, in _send_output
>      self.send(message_body)
>    File "/usr/lib/python2.7/httplib.py", line 826, in send
>      self.sock.sendall(data)
>    File "/usr/lib/python2.7/dist-packages/urllib3/contrib/pyopenssl.py", line
>    203, in sendall
>      return self.connection.sendall(data)
>    File "/usr/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 969, in
>    sendall
>      raise TypeError("buf must be a byte string")
>  
>  [1] https://github.com/kennethreitz/requests/issues/2113
>  [2] https://github.com/pyca/pyopenssl/issues/15
>  
>  ----- Original Message -----
>  > From: "Cole Robinson" <crobinso(a)redhat.com&gt;
>  > To: "Jani Nikula" <jani.nikula(a)intel.com&gt;, "Arun Babu
> Neelicattu"
>  > <abn(a)redhat.com&gt;
>  > Cc: python-bugzilla(a)lists.fedorahosted.org, "Josef Stribny"
>  > <jstribny(a)redhat.com&gt;
>  > Sent: Tuesday, November 18, 2014 2:43:46 AM
>  > Subject: Re: [python-bugzilla] [BUG] Server error: buf must be a byte
>  > string
>  > 
>  > On 11/17/2014 09:37 AM, Jani Nikula wrote:
>  > > On Mon, 17 Nov 2014, Arun Babu Neelicattu <abn(a)redhat.com&gt; wrote:
>  > >> Jani; for what it is worth, could not reproduce using Python 2.7.5,
>  > >> 2.7.8
>  > >> and 3.3.2. Tried using both requests 1.x and 2.x too. This might have to
>  > >> do with the proxy issues requests had via urllib3. Think it should be
>  > >> fixed in 2.x.
>  > >>
>  > >> https://github.com/kennethreitz/requests/pull/478
>  > >> https://github.com/shazow/urllib3/pull/68
>  > >>
>  > >> Without knowing your environment that is my best guess. Can you
>  > >> provide what version of python-bugzilla and requests you are using?
>  > >> And probably will be good to see what happens without the proxy.
>  > >
>  > > I ruled out proxy, same thing without.
>  > >
>  > > python-bugzilla is current git master.
>  > >
>  > > The dependencies are from Debian testing, python-requests version
>  > > 2.4.3-2 and python-urllib3 version 1.9.1-2.
>  > >
>  > > BR,
>  > > Jani.
>  > >
>  > >
>  > >
>  > 
>  > I see you've installed bugzilla locally, maybe try running directly from
>  > the
>  > git checkout using ./bugzilla-cli, might rule out potential installation
>  > issues
>  > 
>  > If you get the same result, stick a 'print requests' in bugzilla/base.py
>  > and
>  > retry, ensure you are using the expected requests module path. I've hit
>  > issues
>  > before with having an older version of a library somewhere in pythonpath
>  > 
>  > FWIW the bug that came with the python-bugzilla commit you identified (I
>  > should have stuck this in the changelog):
>  > https://bugzilla.redhat.com/show_bug.cgi?id=1104197
>  > 
>  > Should be easy enough to conditionalize that bit on older python-requests,
>  > but
>  > I'd still like to figure out what's going on first.
>  > 
>  > - Cole
>  > 
>