[Fedora-suds-list] HTTP1.1 keep-alive persistent connections

[Rod Montgomery]

Inspired by David Robinson's pattern for using httplib2 (quoted at the end of this message), I hacked out an extension to his 
pattern that adds two capabilities:

1. An "open" method, so it can load WSDL from the Net rather than from a local file, and

2. Capturing and using two particular cookies that I need for my current application.

Generalizing to arbitrary cookies would be somewhat tricky, because of the way httplib2 handles multiple "set-cookie" 
headers, to wit: by ",".join()-ing them. Since "," can appear within a "set-cookie" header, the cookie-catching code would 
have to figure out which ","s are the separators, and I don't quite see how to do that with full generality. See the comment 
in the code and the Web-posting to which that comment points.

My hack does not keep track of which cookies should go to what domains: it just sends all cookies received on every 
subsequent transaction.

One thing puzzles me: Suds does not let me re-use the same transport instance for a second client. If I try, it throws 
 >>Exception: Duplicate domain "suds.options" found<<. For my particular application, this is an annoyance rather than a 
show-stopper: I just have to copy the cookies from the transport instance for the client that collects them to the transport 
instance for the client that needs to use them.

I haven't thoroughly tested my hack yet, but it seems to pass its self-test, i.e. the code after "if __name__...".

I hope this hack has some inspirational value.

Here it is:

# mw3trTransport.py

# Very minimal cookie-handling: assumes that every cookie set is to be sent on every subsequent request

# Based on http://lists.fedoraproject.org/pipermail/suds/2010-March/000749.html
#      and http://code.google.com/p/httplib2/wiki/Examples section "Cookies"

import sys

sudsdir = '/home/rod/mw3/webServices/suds0' # ...suds for my mods, ...suds0 for unmodified from repo

for px in range(len(sys.path)):
   if 'suds' in sys.path[px]:
     print 'sys.path[' + repr(px) + ']: found ' + sys.path[px]
     sys.path[px] = sudsdir
     print '   changed to ' + sys.path[px]
     break
   else:
     pass
else:
   sys.path += [sudsdir]
   print 'sys.path: appended ', sys.path[-1]

import re, StringIO, httplib2, suds.transport, suds.client

class MW3TRResponse:
     pass

# cookie_setting_RE = re.compile('([^=]+)=([^;]*);')
MW3TRauthCookieValueRE = re.compile('\.CLG_AUTHFORMS=([^;]*);')
MW3TRsessCookieValueRE = re.compile('ASP\.NET_SessionId=([^;]*);')

class MW3TRTransport(suds.transport.Transport):

     def __init__(self, **kwargs):
         suds.transport.Transport.__init__(self)
         self.http = httplib2.Http()
         self.cookieValue = {}

     def send(self, request):
         url = request.url
         message = request.message
         headers = request.headers
         if self.cookieValue:
           cookieList = []
           if 'auth' in self.cookieValue: cookieList.append('.CLG_AUTHFORMS='    + self.cookieValue['auth'])
           if 'sess' in self.cookieValue: cookieList.append('ASP.NET_SessionId=' + self.cookieValue['sess'])
           headers["Cookie"] = ';'.join(cookieList)
           print "Cookie:", headers["Cookie"]
         response = MW3TRResponse()
         print "headers:", headers
         print "message:", message
         response.headers, response.message = self.http.request(url, "POST", body=message, headers=headers)
         print "response.headers:", response.headers
         if "set-cookie" in response.headers:
           # Problem: response.headers is a dict but
           # multiple Set-Cookie constructs cannot be stuffed into a dict!!!
           # However, one of the answers to
           # http://stackoverflow.com/questions/1738227/httplib2-how-to-set-more-than-one-cookie
           # says httplib2 actually ','.join()s the contents of multiple Set-Cookie headers
           # to make the response.headers["Set-Cookie"] value
           print "set-cookie:", response.headers["set-cookie"]
           mm = MW3TRsessCookieValueRE.search(response.headers["set-cookie"])
           if mm: print "mm/sess:", mm.groups()
           if mm: self.cookieValue['sess'] = mm.groups()[0]
           mm = MW3TRauthCookieValueRE.search(response.headers["set-cookie"])
           if mm: print "mm/auth:", mm.groups()
           if mm: self.cookieValue['auth'] = mm.groups()[0]
           print 'self.cookieValue:', self.cookieValue
         return response

     def open(self, rqst):
       #print "MW3TRTransport.open rqst.url:", rqst.url
       #print "    rqst.headers:", rqst.headers
       #print "    rqst.message:", rqst.message
       response = MW3TRResponse()
       response.headers, response.message = self.http.request(rqst.url, "GET", body=rqst.message, headers=rqst.headers)
       #print "MW3TRTransport.open response.message:", response.message
       return StringIO.StringIO(response.message)

#wsdl = "file:///yourWsdl.wsdl"
#http = MW3TRTransport()
#client = suds.client.Client(wsdl, transport=http)
#...

if __name__ == '__main__':
   import sys
   if len(sys.argv) < 4:
     print "Usage (testing): python mw3trTransport.py warNumber userId password"
   else:
     warId, userId, password = sys.argv[1:]
     import logging
     logging.basicConfig(level=logging.INFO)

     logging.getLogger('suds.transport').setLevel(logging.DEBUG)

     sudsdir = '/home/rod/mw3/webServices/suds0' # ...suds for my mods, ...suds0 for unmodified from repo

     # This should work whether suds was installed from PyPI or not:
     for px in range(len(sys.path)):
       if 'suds' in sys.path[px]:
         print 'sys.path[' + repr(px) + ']: found ' + sys.path[px]
         sys.path[px] = sudsdir
         print '   changed to ' + sys.path[px]
         break
       else:
         pass
     else:
       sys.path += [sudsdir]
       print 'sys.path: appended ', sys.path[-1]

     import suds.client
     import pdb

     mw3HttpAS = MW3TRTransport()
     asurl = 'http://test.crimsonleafgames.com/WebSvc/AuthenticationService.svc?wsdl'
     asc = suds.client.Client(asurl, cache=None, transport=mw3HttpAS)
     #pdb.run('asc = suds.client.Client(asurl, cache=None)')
     print asc
     print "Login:", asc.service.Login(userId, password)
     print "last_received:"
     print asc.last_received()
     print "mw3HttpAS.cookieValue:", mw3HttpAS.cookieValue

     mw3surl = 'http://test.crimsonleafgames.com/WebSvc/MW3TRServices.svc?wsdl'
     mw3HttpMW3 = MW3TRTransport()
     mw3HttpMW3.cookieValue = mw3HttpAS.cookieValue
     mw3sc = suds.client.Client(mw3surl, cache=None, transport=mw3HttpMW3)
     #pdb.run('mw3sc = suds.client.Client(mw3surl, cache=None, transport=mw3Http)')
     print mw3sc

     print "Web Service Version:", mw3sc.service.GetWebServiceVersion()

     sdd = mw3sc.service.GetSpaceDockData(warId)
     print "Space Dock Data:"
     print sdd

     ecr = mw3sc.service.ExecuteCommand(warId, 'fly')
     print 'fly => type:', type(ecr), 'val:', ecr

     #print 'nav a =>', mw3sc.service.ExecuteCommand(warId, 'nav a')

     print 'end =>',  mw3sc.service.ExecuteCommand(warId, 'end')

     lor = asc.service.Logout()
     print "Logout: type:", type(lor), 'val:', lor

David Robinson wrote, On 03/09/2010 03:12 AM:
>> I am interested in getting keep-alive connections working with suds
>> because it has large performance benefits for my application. I
>> noticed that there was some discussion in October about persistent
>> connections:
>> https://www.redhat.com/archives/fedora-suds-list/2009-October/msg00038.html
>>
>> Rod ->  you were going to follow up to the list about your results.
>> Have you had any luck till now in getting that working? If you did, is
>> it possible to share your solution?
>
> Hi all,
>
> It seems like a few people are interested in persistent connections...
> below is how I got suds working with httplib2. I don't have a need for
> cookies or proxies etc so the solution below is lacking them (if
> anyone needs them take a look in suds/transport/http.py and the
> httplib2 docs - it doesn't look like it would be too difficult to
> add). Its also lacking any exception handling (although you'd probably
> just catch them in your application). Reusing connections gives a nice
> performance boost for my use-case. Anyway, thought I'd share it since
> there seems to be some demand :-)
>
> -Dave
>
> #!/usr/bin/env python
>
> from httplib2 import Http
> from suds.transport import Transport
> from suds.client import Client
>
> class Httplib2Response:
>      pass
>
> class Httplib2Transport(Transport):
>
>      def __init__(self, **kwargs):
>          Transport.__init__(self)
>          self.http = Http()
>
>      def send(self, request):
>          url = request.url
>          message = request.message
>          headers = request.headers
>          response = Httplib2Response()
>          response.headers, response.message = self.http.request(url,
> "PUT", body=message, headers=headers)
>          return response
>
> wsdl = "file:///yourWsdl.wsdl"
> http = Httplib2Transport()
> client = Client(wsdl, transport=http)
> ...