Complex Return Types (Again?)

[Dan Sommers]

Greetings,

I'm calling a web service whose return value is a complex type, and I'm
trying to recover the "original" XML from the SOAP response rather than
the python object (so that I can store that XML in my database for
further analysis later).  I thumbed through the archives, and found
these:

    https://www.redhat.com/archives/fedora-suds-list/2009-November/msg00021.html,
    https://www.redhat.com/archives/fedora-suds-list/2009-September/msg00009....,
    https://www.redhat.com/archives/fedora-suds-list/2009-August/msg00022.html,
    https://www.redhat.com/archives/fedora-suds-list/2009-August/msg00022.html

but no real solution.

I can recover something close by calling last_received method of the
client, or equivalently by setting the retxml option of the client, but
I'm having an awful time fixing up all of the namespaces and getting rid
of the elements with xsi:nil="true" attributes.

Is there a (simple?) solution to this issue?  I can provide WSDL and an
actual SOAP response if that would help, but I think that this is a
generic issue (and I'd certainly like to solve it that way).

It would seem that if suds can parse WSDL and SOAP messages, then there
should be enough infrastructure to recover the XML from the server.  (In
fact, it seems that suds almost has to recover that XML in order to
build the python object that it does.)  Or maybe it's just easy for me
to say that because I'm relatively new at SOAP.  ;-)

Thanks,
Dan

[Jeff Ortel]

On 11/18/2009 02:44 PM, Dan Sommers wrote:
>  Greetings, 
Hey Dan :)

>  I'm calling a web service whose return value is a complex type, and I'm
>  trying to recover the "original" XML from the SOAP response rather than
>  the python object (so that I can store that XML in my database for
>  further analysis later).  I thumbed through the archives, and found
>  these:
>
>       https://www.redhat.com/archives/fedora-suds-list/2009-November/msg00021.html,
>       https://www.redhat.com/archives/fedora-suds-list/2009-September/msg00009....,
>       https://www.redhat.com/archives/fedora-suds-list/2009-August/msg00022.html,
>       https://www.redhat.com/archives/fedora-suds-list/2009-August/msg00022.html
>
>  but no real solution.
>
>  I can recover something close by calling last_received method of the
>  client, or equivalently by setting the retxml option of the client, but
>  I'm having an awful time fixing up all of the namespaces and getting rid
>  of the elements with xsi:nil="true" attributes. 
Both last_received() and using setting the 'retxml' option results in giving you
access to 
the sax parsed XML returned by the server.  The suds.sax.element.Element object can be 
easily converted to a string for storage by:

d = client.last_received()
s = str(d)

client.set_options(retxml=True)
d = client.service.foo()
s = str(d)

but, I'm assuming that is somehow not what you're looking for.  Can you be more
specific 
about how this does not meet you needs?  Is it because the str() of the DOM tree is a 
reconstruction of the string XML and not the raw string XML?  There should be almost no 
difference them.  Also, won't you have the same problems with:
<snip>
>  I'm having an awful time fixing up all of the namespaces and
> getting rid
>  of the elements with xsi:nil="true" attributes. 
</snip>
even if you had the raw string XML?

>  Is there a (simple?) solution to this issue?  I can provide WSDL and an
>  actual SOAP response if that would help, but I think that this is a
>  generic issue (and I'd certainly like to solve it that way). 
Generic, yes.  That is where the two methods for getting the returned XML you mentioned 
above came from.

>  It would seem that if suds can parse WSDL and SOAP messages, then there
>  should be enough infrastructure to recover the XML from the server.  (In
>  fact, it seems that suds almost has to recover that XML in order to
>  build the python object that it does.)  Or maybe it's just easy for me
>  to say that because I'm relatively new at SOAP.  ;-) 
The exact XML (stream) returned by the server is consumed by the sax parser (inline) and 
isn't really available.  To capture it, I would have to read the entire document into

memory then sax parse the string.

>  Thanks,
>  Dan
>
>  _______________________________________________
>  fedora-suds-list mailing list
>  fedora-suds-list(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-suds-list

[Dan Sommers]

On 2009-11-18 at 17:29:00 -0600,
Jeff Ortel <jortel(a)redhat.com&gt; wrote:

>  On 11/18/2009 02:44 PM, Dan Sommers wrote: 
>  d = client.last_received()
>  s = str(d) 
That was my first try, and it worked smashingly until one of my users
found some XML that contained an ampersand ("&") and a less than symbol
("<") within its text nodes.  Somewhere in there, suds decoded the
entities twice, and str returned ill-formed XML (i.e., XML containing
actual ampersands and less than symbols in its text nodes instead of
"&amp;"s and "&lt;"s).

>  client.set_options(retxml=True)
>  d = client.service.foo()
>  s = str(d) 
That's what my code looks like now, but without that last string
conversion (which apparently triggers that extra level of decoding?):

    client.set_options( retxml = True )
    s = client.service.foo( ... )
    x = lxml.etree.XML( s )
    ...

>  ... Also, won't you have the same problems with:
>  <snip>
>  > I'm having an awful time fixing up all of the namespaces and getting
>  > rid of the elements with xsi:nil="true" attributes.
>  </snip>
>  even if you had the raw string XML? 
My tactical mistake was prematurely extracting an XML fragment from the
response and losing an essential xmlns attribute or three.  Oops.  ;-)

So now I take the *whole* response (i.e., the soapenv:Envelope element)
and run it through an XSL transform that pulls out the fragment I need,
re-wraps that fragment in a new element, and removes elements with
xsi:nil="true" attributes.  The namespace issue went away once I fixed
my approach.

>  The exact XML (stream) returned by the server is consumed by the sax
>  parser (inline) and isn't really available.  To capture it, I would
>  have to read the entire document into memory then sax parse the
>  string. 
What you're doing is definitely the right approach (not that you need my
approval!).  My use of the word "original" referred to the character
entity issue I described above.

Thank you, for both suds and your reply.

Dan

[Jeff Ortel]

On 11/19/2009 06:54 AM, Dan Sommers wrote:
>  On 2009-11-18 at 17:29:00 -0600,
>  Jeff Ortel<jortel(a)redhat.com>  wrote:
>
> > On 11/18/2009 02:44 PM, Dan Sommers wrote:
>
> > d = client.last_received()
> > s = str(d)
>
>  That was my first try, and it worked smashingly until one of my users
>  found some XML that contained an ampersand ("&") and a less than symbol
>  ("<") within its text nodes.  Somewhere in there, suds decoded the
>  entities twice, and str returned ill-formed XML (i.e., XML containing
>  actual ampersands and less than symbols in its text nodes instead of
>  "&amp;"s and "&lt;"s). 
Ah, very interesting.  Let's fix this!  Can you send me sample XML that the suds sax 
package does not handle correctly?  Also, what version of suds are you using?

> > client.set_options(retxml=True)
> > d = client.service.foo()
> > s = str(d)
>
>  That's what my code looks like now, but without that last string
>  conversion (which apparently triggers that extra level of decoding?):
>
>       client.set_options( retxml = True )
>       s = client.service.foo( ... )
>       x = lxml.etree.XML( s )
>       ...
>
> > ... Also, won't you have the same problems with:
> > <snip>
> >> I'm having an awful time fixing up all of the namespaces and getting
> >> rid of the elements with xsi:nil="true" attributes.
> > </snip>
> > even if you had the raw string XML?
>
>  My tactical mistake was prematurely extracting an XML fragment from the
>  response and losing an essential xmlns attribute or three.  Oops.  ;-)
>
>  So now I take the *whole* response (i.e., the soapenv:Envelope element)
>  and run it through an XSL transform that pulls out the fragment I need,
>  re-wraps that fragment in a new element, and removes elements with
>  xsi:nil="true" attributes.  The namespace issue went away once I fixed
>  my approach.
>
> > The exact XML (stream) returned by the server is consumed by the sax
> > parser (inline) and isn't really available.  To capture it, I would
> > have to read the entire document into memory then sax parse the
> > string.
>
>  What you're doing is definitely the right approach (not that you need my
>  approval!).  My use of the word "original" referred to the character
>  entity issue I described above.
>
>  Thank you, for both suds and your reply.
>
>  Dan

[Dan Sommers]

On 2009-11-19 at 08:49:34 -0600,
Jeff Ortel <jortel(a)redhat.com&gt; wrote:

>  On 11/19/2009 06:54 AM, Dan Sommers wrote:
> > On 2009-11-18 at 17:29:00 -0600,
> > Jeff Ortel&lt;jortel(a)redhat.com&gt;  wrote:
> >
> >> On 11/18/2009 02:44 PM, Dan Sommers wrote:
> >
> >> d = client.last_received()
> >> s = str(d)
> >
> > That was my first try, and it worked smashingly until one of my users
> > found some XML that contained an ampersand ("&") and a less than
> > symbol ("<") within its text nodes.  Somewhere in there, suds decoded
> > the entities twice, and str returned ill-formed XML (i.e., XML
> > containing actual ampersands and less than symbols in its text nodes
> > instead of "&amp;"s and "&lt;"s).
>
>  Ah, very interesting.  Let's fix this!  Can you send me sample XML
>  that the suds sax package does not handle correctly?  Also, what
>  version of suds are you using? 
It's suds 0.3.7 (and Python 2.6.2 on Ubuntu).

For the moment, it's not my XML to send.  That said, you should be able
to confirm this behavior with any SOAP response that contains an
XML-special character within a text node.  Meanwhile, I will verify the
legal status of the XML and go from there.

Thanks,
Dan

[Waldemar Osuch]

On Wed, Nov 18, 2009 at 1:44 PM, Dan Sommers <sommersd(a)nimssc.net&gt; wrote:
>  Greetings,
>
>  I'm calling a web service whose return value is a complex type, and I'm
>  trying to recover the "original" XML from the SOAP response rather than
>  the python object (so that I can store that XML in my database for
>  further analysis later). 
If you really, really have to have the xml in the raw form maybe you should
look into a different library.

Before suds came along (thanks Jeff) I have been successfully using
elementsoap by effbot http://effbot.org/zone/element-soap.htm.

It is much lower level library than suds (no WSDL processing) but you have
full control over what is happening.

Waldemar