[Fedora-suds-list] Gzip compression
Daniel Rodriguez
danjrod at gmail.com
Thu May 6 21:51:23 UTC 2010
Hi Jeff,
As stated I have worked in the transport module as I said, as I believed
there was room for improvement. Please find it attached for your
consideration and for anyone else that may be willing to test it (the
options.py and http.py can directly replace the existing files in
suds/transport and be used straight away)
I have run a series of tests with compression enabled and disabled, tested
sites with gzip and deflate (found none with bzip2) and wsdl in files or in
urls. So far, everything worked and since they have many more changes than
just the compression, I think they deserve their own ticket, for you to
choose which one to accept (if any)
In options.py
- Variable: "compression" added.
It may be assigned the following values:
- yes: request compression and decompress incoming compressed content
- no: do not request compression and do not decompress content even if
present in the reply
- auto: do not request compression but decompress it if present in the
reply
Default: yes
- Variable: compmethods
A list with the compression methods to be supported.
So far: gzip, deflate, bzip2
The changes in http.py are:
- open
The function is now urllib2 agnostic. It will try to detect if the passed
url is a file, a url or data and act accordingly.
This allows passing a variable with the WSDL content inside, besides the
current functionality (a url)
Still returning a "file-like" object
- send
The function is now urllib2 agnostic. It simply calls the appropriate
function
I think that both could be moved upward to __init__.py to the Transport
class, given its library independence
- invoke (possibly a better name can be found)
New function that is called by open and send to perform http. This should be
in the base class Transport and be redefined by new transports like this
This function is the one working with urllib2 directly or through the
functions you had already provided
It may return a suds.transport.Reply or a file-like object for open
It does call two new functions that in my opinion could make it to the
standard interface of Transport
- prerequest
- postreply
This should be called before sending the request to the network and before
returning the reply to the caller.
I have used them to "request" compression and decompress if needed. Another
example would be "HttpAuthenticated", where you redefine open and send to
add the credentials. This could go into "prerequest"
- addcookies and getcookies
I think the could be removed since they are one-liners
- u2opener
As stated in my comments in the code, I think there was a "bug" in the
original code since self.urlopener was never being used and a new urlopener
was being created for each and every call
Now a urlopener is created if needed or if the proxy options change during
execution.
This could be improved by just checking if the proxy for a given scheme
(http, https) has changed, but would require passing the url to this
function
- u2handlers
Now a copy of self.options.proxy is made (if needed) to detect if they
change during execution
- prerequest and postreply as aforementioned
Let me know what you think
Best regards
Daniel
On Thu, May 6, 2010 at 10:35, Daniel Rodriguez <danjrod at gmail.com> wrote:
> Hi Jeff,
>
> The following will let you test it.
>
> import suds
> l_wsdl = 'http://webservices.daehosting.com/services/isbnservice.wso?WSDL'
> l_client = suds.client.Client(l_wsdl, gzip = False)
>
> l_result = l_client.service.IsValidISBN10('0123456789')
> print l_result
>
> # default behaviour is gzip = True as below
> l_client2 = suds.client.Client(l_wsdl, gzip = True)
>
> l_result = l_client2.service.IsValidISBN10('0123456789')
> print l_result
>
>
> Anyhow, I think that the changes can be implemented also for
> Transport.open, so that if you have to download large WSDL files, the
> download will also happen compressed (if supported by the server)
>
> I even think that the compression request and processing could be part of
> the Transport class, to let people implement transport with libraries that
> may not have native gzip support.
>
> Since I have some free time in the next minutes, I will possibly add a new
> patch to the ticket, so you may decide which one you like more
>
> Best regards
>
> Daniel
>
> On Wed, May 5, 2010 at 20:52, Jeff Ortel <jortel at redhat.com> wrote:
>
>> Hey Daniel,
>>
>> Thanks for the great suggestion, patch and for creating a ticket for this.
>> I'll review for inclusion in 0.4. Can you recommend a public test
>> site/service for this?
>>
>> Regards,
>>
>> Jeff
>>
>>
>> On 04/29/2010 07:53 AM, Daniel Rodriguez wrote:
>>
>>> Hello,
>>>
>>> I have added ticket 311 requesting this enhancement with attached
>>> patches for transport/http.py and transport/options.py
>>>
>>> Requesting compression can be turned off and on with the "gzip" option
>>> (a bool)
>>>
>>> The gzip decompression code works always. The reasoning behind this is:
>>>
>>> * Some network providers may compress the data in HTTP responses in
>>> order to decrease network traffic, expecting all clients to be
>>> ready to decompress data
>>>
>>> Best regards
>>>
>>> Daniel
>>>
>>> On Sat, Apr 3, 2010 at 21:07, Daniel Rodriguez <danjrod at gmail.com
>>> <mailto:danjrod at gmail.com>> wrote:
>>>
>>> A "break" at the end of the "if" statement will not hurt and avoid
>>> checking unnecessary headers.
>>>
>>> Best regards
>>>
>>>
>>> On Sat, Apr 3, 2010 at 02:14, Daniel Rodriguez <danjrod at gmail.com
>>> <mailto:danjrod at gmail.com>> wrote:
>>>
>>> Hi Jeff et al,
>>>
>>> If I am not wrong (I may have missed a configuration option),
>>> suds is not using or requesting gzip compression.
>>>
>>> I have added a few lines and tested at least requesting and
>>> accepting gzip compression. My changes have been trivial and
>>> this could be possibly added as an option to suds.
>>>
>>> The changes I have made (against 3.9):
>>>
>>> _File_: suds/transport/http.py:
>>> _
>>> _
>>> _Extra imports_:
>>>
>>> import gzip
>>> import StringIO
>>>
>>> _Class_: HttpTransport
>>> _Method_: send
>>>
>>> _Added:_
>>>
>>> After:
>>>
>>> * headers = request.headers
>>>
>>> added the following line
>>>
>>> * headers.update([('Accept-Encoding', 'gzip')])
>>>
>>> _Changed:_
>>>
>>> * result = Reply(200, fp.headers.dict, fp.read())
>>>
>>> to
>>>
>>> * replydata = fp.read()
>>> * for header in fp.headers.dict.keys():
>>> * if header.lower() == 'content-encoding' and
>>> fp.headers.dict[header] == 'gzip':
>>> * replydatacompressedstream =
>>> StringIO.StringIO(replydata)
>>> * gzipper =
>>> gzip.GzipFile(fileobj=replydatacompressedstream)
>>> * replydata = gzipper.read()
>>> * result = Reply(200, fp.headers.dict, replydata)
>>>
>>> Best regards
>>>
>>> Daniel
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> suds mailing list
>>> suds at lists.fedoraproject.org
>>> https://admin.fedoraproject.org/mailman/listinfo/suds
>>>
>>
>>
>
On Thu, May 6, 2010 at 10:35, Daniel Rodriguez <danjrod at gmail.com> wrote:
> Hi Jeff,
>
> The following will let you test it.
>
> import suds
> l_wsdl = 'http://webservices.daehosting.com/services/isbnservice.wso?WSDL'
> l_client = suds.client.Client(l_wsdl, gzip = False)
>
> l_result = l_client.service.IsValidISBN10('0123456789')
> print l_result
>
> # default behaviour is gzip = True as below
> l_client2 = suds.client.Client(l_wsdl, gzip = True)
>
> l_result = l_client2.service.IsValidISBN10('0123456789')
> print l_result
>
>
> Anyhow, I think that the changes can be implemented also for
> Transport.open, so that if you have to download large WSDL files, the
> download will also happen compressed (if supported by the server)
>
> I even think that the compression request and processing could be part of
> the Transport class, to let people implement transport with libraries that
> may not have native gzip support.
>
> Since I have some free time in the next minutes, I will possibly add a new
> patch to the ticket, so you may decide which one you like more
>
> Best regards
>
> Daniel
>
> On Wed, May 5, 2010 at 20:52, Jeff Ortel <jortel at redhat.com> wrote:
>
>> Hey Daniel,
>>
>> Thanks for the great suggestion, patch and for creating a ticket for this.
>> I'll review for inclusion in 0.4. Can you recommend a public test
>> site/service for this?
>>
>> Regards,
>>
>> Jeff
>>
>>
>> On 04/29/2010 07:53 AM, Daniel Rodriguez wrote:
>>
>>> Hello,
>>>
>>> I have added ticket 311 requesting this enhancement with attached
>>> patches for transport/http.py and transport/options.py
>>>
>>> Requesting compression can be turned off and on with the "gzip" option
>>> (a bool)
>>>
>>> The gzip decompression code works always. The reasoning behind this is:
>>>
>>> * Some network providers may compress the data in HTTP responses in
>>> order to decrease network traffic, expecting all clients to be
>>> ready to decompress data
>>>
>>> Best regards
>>>
>>> Daniel
>>>
>>> On Sat, Apr 3, 2010 at 21:07, Daniel Rodriguez <danjrod at gmail.com
>>> <mailto:danjrod at gmail.com>> wrote:
>>>
>>> A "break" at the end of the "if" statement will not hurt and avoid
>>> checking unnecessary headers.
>>>
>>> Best regards
>>>
>>>
>>> On Sat, Apr 3, 2010 at 02:14, Daniel Rodriguez <danjrod at gmail.com
>>> <mailto:danjrod at gmail.com>> wrote:
>>>
>>> Hi Jeff et al,
>>>
>>> If I am not wrong (I may have missed a configuration option),
>>> suds is not using or requesting gzip compression.
>>>
>>> I have added a few lines and tested at least requesting and
>>> accepting gzip compression. My changes have been trivial and
>>> this could be possibly added as an option to suds.
>>>
>>> The changes I have made (against 3.9):
>>>
>>> _File_: suds/transport/http.py:
>>> _
>>> _
>>> _Extra imports_:
>>>
>>> import gzip
>>> import StringIO
>>>
>>> _Class_: HttpTransport
>>> _Method_: send
>>>
>>> _Added:_
>>>
>>> After:
>>>
>>> * headers = request.headers
>>>
>>> added the following line
>>>
>>> * headers.update([('Accept-Encoding', 'gzip')])
>>>
>>> _Changed:_
>>>
>>> * result = Reply(200, fp.headers.dict, fp.read())
>>>
>>> to
>>>
>>> * replydata = fp.read()
>>> * for header in fp.headers.dict.keys():
>>> * if header.lower() == 'content-encoding' and
>>> fp.headers.dict[header] == 'gzip':
>>> * replydatacompressedstream =
>>> StringIO.StringIO(replydata)
>>> * gzipper =
>>> gzip.GzipFile(fileobj=replydatacompressedstream)
>>> * replydata = gzipper.read()
>>> * result = Reply(200, fp.headers.dict, replydata)
>>>
>>> Best regards
>>>
>>> Daniel
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> suds mailing list
>>> suds at lists.fedoraproject.org
>>> https://admin.fedoraproject.org/mailman/listinfo/suds
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.fedoraproject.org/pipermail/suds/attachments/20100506/f9511d42/attachment-0001.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: options.py
Type: application/octet-stream
Size: 2953 bytes
Desc: not available
Url : http://lists.fedoraproject.org/pipermail/suds/attachments/20100506/f9511d42/attachment-0002.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: http.py
Type: application/octet-stream
Size: 11045 bytes
Desc: not available
Url : http://lists.fedoraproject.org/pipermail/suds/attachments/20100506/f9511d42/attachment-0003.obj
More information about the suds
mailing list