[Fedora-suds-list] Persistent connections - anyone else interested / suggest howto / working on it?

Fri Nov 6 19:13:10 UTC 2009

Matt,

First a disclaimer, I too am not convinced that Twisted would be a good 
fit for creating persistent connections in suds.  My reasoning is that I 
would assume suds is generally used from within other python projects 
(ie django), so adding the complication of running twisted inside 
another framework seems to be the wrong direction to go.

However, if a project is already running on twisted, then being able to 
switch out the tcp connection to a twisted managed one might prove 
useful.  I make use of twisted pretty extensively, so I'd be happy to 
help with any twisted questions you might have.  To give you some 
insight, the class you're probably interested in for getting the 
TCPClient to automatically reconnect is 
twisted.internet.protocol.ReconnectingClientFactory.  One thing I do 
when I make use of this class is set the maxDelay attribute to put a cap 
on the scaled back reconnects (or else the soap server may come up and 
you'll find yourself waiting up to the default of 3600 seconds before 
trying to reconnect again).

In order to get at the underlying socket to set socket options, the 
following should help:

The reactor.connectTCP (which internet.TCPClient is a wrapper around if 
your using the application framework) takes a factory (that produces 
protocols) and returns back a Connector (who was handed the factory 
through __init__).  The Connector has a _makeTransport method which 
creates a new Client (which is the tcp client, the connector passes a 
handle to itself to the Client through __init__).  Client asks the 
connector for a protocol via connector.buildProtocol, which delegates to 
the factory buildProtocol.   The protocol returned by the factory allows 
you to grab it's underlying transport, which is actually an instance of 
a Connection.  You can then set socket level options on this Connection.

The proper way to set the socket options (at least as far as I've been 
able to tell) is to override the Protocol connectionMade method and set 
the options on the transport, ala (If I want to set set the tcp 
keepalive and keep alive interval for a socket):

 def connectionMade(self):

   # Set the keep alive.
   self.transport.setTcpKeepAlive(1)
   self.transport.socket.setsockopt(socket.SOL_TCP, 
socket.TCP_KEEPINTVL, 200)

One thing to remember is that the reactor is a singleton, so you'll need 
to take into account the reactor already being started (again if suds is 
being used by code already running inside twisted).  Feel free to run by 
me what you come up with for plugging twisted connections in and I'll 
see if I can add anything to it.

Thanks,

Mike

Matt C wrote:
> Twisted definitely takes some getting used to. That said, it's got
> some advantages that I'm very interested in using for a project later
> on down the road. For example, it's pretty easy to get at socket level
> options to, say, enable or disable TCP_NODELAY which can have
> performance implications for this type of traffic...
>
> I'll post anything useful that I come up with using twisted for transport.
>
> -Matt
>
> On Wed, Oct 21, 2009 at 9:55 PM, Rod Montgomery <monty at starfief.com> wrote:
>   
>> Jeff Ortel wrote, On 10/21/2009 01:45 PM:
>>     
>>> On 10/21/2009 12:20 PM, Waldemar Osuch wrote:
>>>       
>>>> On Wed, Oct 21, 2009 at 8:05 AM, Rod Montgomery<monty at starfief.com>
>>>> wrote:
>>>>         
>>>>> Currently (i.e. r580 / 0.3.7 release candidate) Suds uses urllib2, which
>>>>> uses urllib, both from the standard Python library.
>>>>>
>>>>> I'm interested in making Suds able to use persistent connections.
>>>>>
>>>>> Twisted (twistedmatrix.com) seems to have an elaborate HTTP/1.1 Client
>>>>>           
>>>>> Another possibility is httplib2:
>>>>>           
>>>> Yes, it looks like the Client is using urllib2 by default but you should
>>>> be able to replace it with a custom transport.
>>>>
>>>> Client.options.transport = YourFancyPersistentConnection()
>>>>
>>>> As long as YourFancyPersistentConnection confirms to Transport interface
>>>> from suds.transport it should work.
>>>>         
>>> Yup.  The transport functionality as factored out into the Transport
>>> (interface) and urllib2 based implementations for just this reason.
>>>       
>> Aha! That sounds promising! I had not grasped that from the Documentation.
>> Thanks!
>>
>>     
>>>> I think you will have more luck with httplib2 than Twisted version
>>>> unless you are versed in Twisted ways.
>>>>         
>> You are now the second person who has warned me that Twisted may be more
>> challenging than I want.
>>
>>     
>>>> By the way you probably want:
>>>> http://httplib2.googlecode.com/files/httplib2-0.5.0.tar.gz
>>>> and not the Python3 version you have linked above.
>>>>         
>> It seems httplib2 is also available in a repository from my Linux
>> distribution, Ubuntu.
>>
>> There is also apparently a urllib3, which also claims to be thread-safe:
>>
>> http://pypi.python.org/pypi/urllib3/0.2
>>
>> I think I'll try both httplib2 and urllib3, and report my results back to
>> the List.
>>
>> Thanks for the prompt, informative responses!
>>
>> _______________________________________________
>> fedora-suds-list mailing list
>> fedora-suds-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/fedora-suds-list
>>
>>     
>
> _______________________________________________
> fedora-suds-list mailing list
> fedora-suds-list at redhat.com
> https://www.redhat.com/mailman/listinfo/fedora-suds-list
>
>   