Null/optional items in complex types

[Martin Aspeli]

Hi,

We are calling a service getUserDetails(). It returns a complex type
"userDetails" with various fields, let's say first/middle/last name to
keep it simple. The WSDL says middleName is optional (it has
<xs:element minOccurs='0' name='middleName'
type='xs:string'/>).

Our code expects the return value to have a middleName attribute with
a value of None if the middle name is not set.

However, what's happening is that the service (being written by a
third party, I believe using JBoss and Axis) is not returning anything
at all for this attribute. The complex type simply doesn't have it, so
when we do details.middleName, we get an AttributeError.

I can't post the WSDL or response XML here on a public list, but
basically it looks to me like the response is omitting the tag for the
middle name altogether and Suds just ignores it, rather than
consulting the WSDL and saying "this property should be here, it's not
in the response, set it to None", as we were expecting.

Actually, there's one other problem: In the userDetails complex type,
we have an attribute that's of another complex type, say "address". If
This is not set, it comes through as an empty tag "<address />" in the
response body. Suds treats that as an empty string, instead of None,
as I would've expected.

So:

 - Is this a problem with our understanding of Suds?
 - Is this a bug in Suds?
 - Is this in effect an invalid response from the server?

Cheers,
lMartin

[Jeff Ortel]

Hey Martin,

Handling NULL and nil is tricky in XML.  There is a big difference between the 
presents/absence of a element and having the element be empty <middleName/>. 
Handling 
these nodes depends on how they are defined in the XSD.

Suds follows these rules which (I believe) directly reflect XSD/XML specifications:

An /optional/ element that is omitted from the document is just that - not specified.  So,

suds will not create an attribute for it in the unmarshalled object.

A /required/ element that is omitted from the document is represented in the unmarshalled

object as an attribute with a value = None.  It's supposed to be there and isn't. 
It's 
absence is interpreted as nil.  Although technically, it's required by the XSD so
it's 
absence makes the document invalid.

An /empty/ (nil) element <middleName xsi:nil="1"/> contained in the
document is 
represented in the unmarshalled object as an attribute with a value = None.  But, this 
element must be defined in the xsd as <xs:element xs:nillable="1"/>.

An /empty/ (non-nil) element <middleName/> contained in the document is represented
in the 
unmarshalled object as an attribute with a value = ''.  None (nil) must be
represented in 
the document with xsi:nil="1".

See inline below.

Hope this helps :)

-jeff

On 05/06/2010 02:20 AM, Martin Aspeli wrote:
>  Hi,
>
>  We are calling a service getUserDetails(). It returns a complex type
>  "userDetails" with various fields, let's say first/middle/last name to
>  keep it simple. The WSDL says middleName is optional (it has
>  <xs:element minOccurs='0' name='middleName'
> type='xs:string'/>).
>
>  Our code expects the return value to have a middleName attribute with
>  a value of None if the middle name is not set. 
Because it is optional (minOccurs='0'), suds will not create an attribute when the
node is 
not in the document.

>  However, what's happening is that the service (being written by a
>  third party, I believe using JBoss and Axis) is not returning anything
>  at all for this attribute. The complex type simply doesn't have it, so
>  when we do details.middleName, we get an AttributeError. 
Right.  It's optional so you need to check first.

>  I can't post the WSDL or response XML here on a public list, but
>  basically it looks to me like the response is omitting the tag for the
>  middle name altogether and Suds just ignores it, rather than
>  consulting the WSDL and saying "this property should be here, it's not
>  in the response, set it to None", as we were expecting. 
Since <middleName/> is optional, it's not necessarily /supposed/ to be there. 
It's 
optional so it /may/ be there.

>  Actually, there's one other problem: In the userDetails complex type,
>  we have an attribute that's of another complex type, say "address". If
>  This is not set, it comes through as an empty tag "<address />" in the
>  response body. Suds treats that as an empty string, instead of None,
>  as I would've expected. 
As described above, empty does not always = None.

>  So:
>
>    - Is this a problem with our understanding of Suds? 
I think so.

>    - Is this a bug in Suds? 
Don't think so ;)

>    - Is this in effect an invalid response from the server? 
Seems valid to me.

>  Cheers,
>  lMartin
>  _______________________________________________
>  suds mailing list
>  suds(a)lists.fedoraproject.org
>  https://admin.fedoraproject.org/mailman/listinfo/suds

[Martin Aspeli]

Hi Jeff,

On 6 May 2010 23:03, Jeff Ortel <jortel(a)redhat.com&gt; wrote:
>  Hey Martin,
>
>  Handling NULL and nil is tricky in XML.  There is a big difference between
>  the presents/absence of a element and having the element be empty
>  <middleName/>.  Handling these nodes depends on how they are defined in the
>  XSD. 
Yes. However, if the WSDL says the complex type has a given attribute,
I think Suds should make sure that particular complex type *always*
has that attribute (even if set to None or possibly some other "no
value" marker). Getting an attribute error is pretty disconcerting.

>  Suds follows these rules which (I believe) directly reflect XSD/XML
>  specifications:
>
>  An /optional/ element that is omitted from the document is just that - not
>  specified.  So, suds will not create an attribute for it in the unmarshalled
>  object. 
Okay. I guess that runs contrary to my expectations, though I can see
how this may be the right interpretation.

>  A /required/ element that is omitted from the document is represented
> in the
>  unmarshalled object as an attribute with a value = None.  It's supposed to
>  be there and isn't.  It's absence is interpreted as nil.  Although
>  technically, it's required by the XSD so it's absence makes the document
>  invalid. 
Right. So really, there's no way for the service to say "this is
optional" and the code using Suds to do a check like "if response.foo
is not None". You have to do hasattr/getattr tricks. That makes the
code more brittle, since you're relying on the string names of the
attributes when doing the check.

>  An /empty/ (nil) element <middleName xsi:nil="1"/>
> contained in the document
>  is represented in the unmarshalled object as an attribute with a value =
>  None.  But, this element must be defined in the xsd as <xs:element
>  xs:nillable="1"/>. 
Okay. I suspect our WSDL doesn't do this. Maybe it should.

>  An /empty/ (non-nil) element <middleName/> contained in the
> document is
>  represented in the unmarshalled object as an attribute with a value = ''.
>   None (nil) must be represented in the document with xsi:nil="1". 
That's actually seems entirely wrong to me. The WSDL has specified a
type for this attribute. In the example below, the attribute is of a
type 'address', another complex type. For that to be a string is just
plain wrong, and makes for some pretty awkward duck typing.

> > We are calling a service getUserDetails(). It returns a complex
> type
> > "userDetails" with various fields, let's say first/middle/last name to
> > keep it simple. The WSDL says middleName is optional (it has
> > <xs:element minOccurs='0' name='middleName'
> type='xs:string'/>).
> >
> > Our code expects the return value to have a middleName attribute with
> > a value of None if the middle name is not set.
>
>  Because it is optional (minOccurs='0'), suds will not create an attribute
>  when the node is not in the document. 
I can see why Suds does this, though I think it makes client code more
awkward to write. So, in our case we're copying some user data to the
session. Before, we'd do:

session['middlename'] = response.middlename

and we'd expect that session['middlename'] could be None.

Since middlename has minOccurs=0, we actually need:

session['middlename'] = getattr(response, 'middlename', None)

Which is ugly. But even more so, if I'd done:

session['middlename'] = getattr(response, 'middlenmae', None)

(i.e. a typo) I'd get None always and this error could go silent unnoticed.

> > However, what's happening is that the service (being written
> by a
> > third party, I believe using JBoss and Axis) is not returning anything
> > at all for this attribute. The complex type simply doesn't have it, so
> > when we do details.middleName, we get an AttributeError.
>
>  Right.  It's optional so you need to check first. 
I wish I could check using something other than getattr/hasattr, though.

> > I can't post the WSDL or response XML here on a public list,
> but
> > basically it looks to me like the response is omitting the tag for the
> > middle name altogether and Suds just ignores it, rather than
> > consulting the WSDL and saying "this property should be here, it's not
> > in the response, set it to None", as we were expecting.
>
>  Since <middleName/> is optional, it's not necessarily /supposed/ to be
>  there.  It's optional so it /may/ be there. 
Would there be a better way for the WSDL to say "not required"?

> > Actually, there's one other problem: In the userDetails
> complex type,
> > we have an attribute that's of another complex type, say "address". If
> > This is not set, it comes through as an empty tag "<address />" in
> the
> > response body. Suds treats that as an empty string, instead of None,
> > as I would've expected.
>
>  As described above, empty does not always = None. 
Yes, I appreciate that. In some frameworks, we have e.g.

OMITTED = object()

and then you basically can do:

if response.middlename is OMITTED:
    ...

or something like that. Of course, this has cons, too.

Cheers,
Martin

[Jeff Ortel]

On 05/06/2010 10:19 AM, Martin Aspeli wrote:
>  Hi Jeff,
>
>  On 6 May 2010 23:03, Jeff Ortel<jortel(a)redhat.com>  wrote:
> > Hey Martin,
> >
> > Handling NULL and nil is tricky in XML.  There is a big difference between
> > the presents/absence of a element and having the element be empty
> > <middleName/>.  Handling these nodes depends on how they are defined in the
> > XSD.
>
>  Yes. However, if the WSDL says the complex type has a given attribute,
>  I think Suds should make sure that particular complex type *always*
>  has that attribute (even if set to None or possibly some other "no
>  value" marker). Getting an attribute error is pretty disconcerting. 
I can understand you're thinking here and most well designed services stay away from 
optional elements for just this reason.  It seems intuitive to me that /optional/ elements

be represented optionally in suds objects.  If it rubs you the wrong way to do

session['middlename'] = getattr(response, 'middlename', None)

, I understand :)

But there are other circumstances where elements may be optionally present.  And, in these

cases, suds adding an attribute with a value of (None) could be confusing and unintuitive.

  For example, <xs:choice>.  Say my schema has:

<xs:element name="foo">
   <xs:sequence>
     <xs:choice>
       <xs:element nullable="1" name='name'/>
       <xs:element nullable="1" name='age'/>
       <xs:element nullable="1" name='phone'/>
     </xs:choice>
   </xs:sequence>
</xs:element>

Eg:

<foo>
   <age xsi:nil="1"/>
</foo>

Now, if suds umnarshalled as:

(Foo){
   name=None
   age=None
   phone=None
}

And the intention is to communicate to the caller that only the value of age has changed 
and is not (None). The meaning here would be lost.  And, I'd like to keep suds 
unmarshalling rules consistent.

> > Suds follows these rules which (I believe) directly reflect XSD/XML
> > specifications:
> >
> > An /optional/ element that is omitted from the document is just that - not
> > specified.  So, suds will not create an attribute for it in the unmarshalled
> > object.
>
>  Okay. I guess that runs contrary to my expectations, though I can see
>  how this may be the right interpretation.
>
> > A /required/ element that is omitted from the document is represented in the
> > unmarshalled object as an attribute with a value = None.  It's supposed to
> > be there and isn't.  It's absence is interpreted as nil.  Although
> > technically, it's required by the XSD so it's absence makes the document
> > invalid.
>
>  Right. So really, there's no way for the service to say "this is
>  optional" and the code using Suds to do a check like "if response.foo
>  is not None". You have to do hasattr/getattr tricks. That makes the
>  code more brittle, since you're relying on the string names of the
>  attributes when doing the check.
>
> > An /empty/ (nil) element<middleName xsi:nil="1"/>  contained in the
> document
> > is represented in the unmarshalled object as an attribute with a value =
> > None.  But, this element must be defined in the xsd as<xs:element
> > xs:nillable="1"/>.
>
>  Okay. I suspect our WSDL doesn't do this. Maybe it should.
>
> > An /empty/ (non-nil) element<middleName/>  contained in the document is
> > represented in the unmarshalled object as an attribute with a value = ''.
> >   None (nil) must be represented in the document with xsi:nil="1".
>
>  That's actually seems entirely wrong to me. The WSDL has specified a
>  type for this attribute. In the example below, the attribute is of a
>  type 'address', another complex type. For that to be a string is just
>  plain wrong, and makes for some pretty awkward duck typing. 
Sorry, I should have been more specific here.  We were talking about <middleName/>
which 
is an xs:string.  Only empty xs:string elements are interpreted as ''.  All other
types 
such as (int, bool, custom, ..) which cannot by definition be ('') - will be
interpreted 
as (None) unless a default is specified in the XSD.

> >> We are calling a service getUserDetails(). It returns a complex type
> >> "userDetails" with various fields, let's say first/middle/last name
> to
> >> keep it simple. The WSDL says middleName is optional (it has
> >> <xs:element minOccurs='0' name='middleName'
> type='xs:string'/>).
> >>
> >> Our code expects the return value to have a middleName attribute with
> >> a value of None if the middle name is not set.
> >
> > Because it is optional (minOccurs='0'), suds will not create an attribute
> > when the node is not in the document.
>
>  I can see why Suds does this, though I think it makes client code more
>  awkward to write. 
Agreed.

So, in our case we're copying some user data to the
>  session. Before, we'd do:
>
>  session['middlename'] = response.middlename
>
>  and we'd expect that session['middlename'] could be None.
>
>  Since middlename has minOccurs=0, we actually need:
>
>  session['middlename'] = getattr(response, 'middlename', None)
>
>  Which is ugly. But even more so, if I'd done:
>
>  session['middlename'] = getattr(response, 'middlenmae', None)
>
>  (i.e. a typo) I'd get None always and this error could go silent unnoticed. 
True.

This is uglier but safer:

key = 'middlename'
if key in response:
   session[key] = response[key]
else:
   session[key] = None

> >> However, what's happening is that the service (being written by a
> >> third party, I believe using JBoss and Axis) is not returning anything
> >> at all for this attribute. The complex type simply doesn't have it, so
> >> when we do details.middleName, we get an AttributeError.
> >
> > Right.  It's optional so you need to check first.
>
>  I wish I could check using something other than getattr/hasattr, though.
>
> >> I can't post the WSDL or response XML here on a public list, but
> >> basically it looks to me like the response is omitting the tag for the
> >> middle name altogether and Suds just ignores it, rather than
> >> consulting the WSDL and saying "this property should be here, it's not
> >> in the response, set it to None", as we were expecting.
> >
> > Since<middleName/>  is optional, it's not necessarily /supposed/ to be
> > there.  It's optional so it /may/ be there.
>
>  Would there be a better way for the WSDL to say "not required"?
>
> >> Actually, there's one other problem: In the userDetails complex type,
> >> we have an attribute that's of another complex type, say "address".
> If
> >> This is not set, it comes through as an empty tag "<address />"
> in the
> >> response body. Suds treats that as an empty string, instead of None,
> >> as I would've expected.
> >
> > As described above, empty does not always = None.
>
>  Yes, I appreciate that. In some frameworks, we have e.g.
>
>  OMITTED = object()
>
>  and then you basically can do:
>
>  if response.middlename is OMITTED:
>       ...
>
>  or something like that. Of course, this has cons, too.
>
>  Cheers,
>  Martin

[Martin Aspeli]

Hi Jeff,

On 7 May 2010 00:32, Jeff Ortel <jortel(a)redhat.com&gt; wrote:

> >> An /empty/ (nil) element<middleName
> xsi:nil="1"/>  contained in the
> >> document
> >> is represented in the unmarshalled object as an attribute with a value =
> >> None.  But, this element must be defined in the xsd as<xs:element
> >> xs:nillable="1"/>.
> >
> > Okay. I suspect our WSDL doesn't do this. Maybe it should.
> >
> >> An /empty/ (non-nil) element<middleName/>  contained in the document is
> >> represented in the unmarshalled object as an attribute with a value =
> ''.
> >>  None (nil) must be represented in the document with xsi:nil="1".
> >
> > That's actually seems entirely wrong to me. The WSDL has specified a
> > type for this attribute. In the example below, the attribute is of a
> > type 'address', another complex type. For that to be a string is just
> > plain wrong, and makes for some pretty awkward duck typing.
>
>  Sorry, I should have been more specific here.  We were talking about
>  <middleName/> which is an xs:string.  Only empty xs:string elements are
>  interpreted as ''.  All other types such as (int, bool, custom, ..) which
>  cannot by definition be ('') - will be interpreted as (None) unless a
>  default is specified in the XSD. 
Ok, that may be a bug then (or more weirdness in the WSDL). We have an
element like <address /> in the response and it's coming back as "".
When it's set (so we get
<address><street>...</street>...</address>),
it comes back as the correct complex data structure.

>  This is uglier but safer:
>
>  key = 'middlename'
>  if key in response:
>   session[key] = response[key]
>  else:
>   session[key] = None 
Ah, I didn't realise it supported __getitem__ as well as __getattr__.
Does it also support .get()?

Cheers,
Martin

[Jeff Ortel]

On 05/06/2010 08:07 PM, Martin Aspeli wrote:
>  Hi Jeff,
>
>  On 7 May 2010 00:32, Jeff Ortel<jortel(a)redhat.com>  wrote:
>
> >>> An /empty/ (nil) element<middleName xsi:nil="1"/>   
> contained in the
> >>> document
> >>> is represented in the unmarshalled object as an attribute with a value =
> >>> None.  But, this element must be defined in the xsd as<xs:element
> >>> xs:nillable="1"/>.
> >>
> >> Okay. I suspect our WSDL doesn't do this. Maybe it should.
> >>
> >>> An /empty/ (non-nil) element<middleName/>    contained in the document
> is
> >>> represented in the unmarshalled object as an attribute with a value =
> ''.
> >>>   None (nil) must be represented in the document with xsi:nil="1".
> >>
> >> That's actually seems entirely wrong to me. The WSDL has specified a
> >> type for this attribute. In the example below, the attribute is of a
> >> type 'address', another complex type. For that to be a string is just
> >> plain wrong, and makes for some pretty awkward duck typing.
> >
> > Sorry, I should have been more specific here.  We were talking about
> > <middleName/>  which is an xs:string.  Only empty xs:string elements are
> > interpreted as ''.  All other types such as (int, bool, custom, ..) which
> > cannot by definition be ('') - will be interpreted as (None) unless a
> > default is specified in the XSD.
>
>  Ok, that may be a bug then (or more weirdness in the WSDL). We have an
>  element like<address />  in the response and it's coming back as "".
>  When it's set (so we
> get<address><street>...</street>...</address>),
>  it comes back as the correct complex data structure. 
I'll check on this.
I'm more confident that it works this way for the builtin (xs) types I mentioned then
the 
complex types.

> > This is uglier but safer:
> >
> > key = 'middlename'
> > if key in response:
> >   session[key] = response[key]
> > else:
> >   session[key] = None
>
>  Ah, I didn't realise it supported __getitem__ as well as __getattr__.
>  Does it also support .get()? 
No, sorry.  Although rare, I didn't want to hide a real <get/> attribute.

>  Cheers,
>  Martin

[Martin Aspeli]

Hi Jeff,

On 7 May 2010 20:51, Jeff Ortel <jortel(a)redhat.com&gt; wrote:

> > Ok, that may be a bug then (or more weirdness in the WSDL). We
> have an
> > element like<address />  in the response and it's coming back as
> "".
> > When it's set (so we
> get<address><street>...</street>...</address>),
> > it comes back as the correct complex data structure.
>
>  I'll check on this.
>  I'm more confident that it works this way for the builtin (xs) types I
>  mentioned then the complex types. 
Thanks!

> >> This is uglier but safer:
> >>
> >> key = 'middlename'
> >> if key in response:
> >>  session[key] = response[key]
> >> else:
> >>  session[key] = None
> >
> > Ah, I didn't realise it supported __getitem__ as well as __getattr__.
> > Does it also support .get()?
>
>  No, sorry.  Although rare, I didn't want to hide a real <get/> attribute.
Yeah, makes sense.

To be honest, I think it'd have been better to choose either attribute
or dict notation. Having both is confusing and leads to divergent
practices and documentation. Zope 2 made that mistake with various
things (like request keys) and it was never a good idea.

Martin

[Joshua J. Kugler]

On Friday 07 May 2010, Martin Aspeli elucidated thus:
>  >> Ah, I didn't realise it supported __getitem__ as well
> as
>  >> __getattr__. Does it also support .get()?
>  >
>  > No, sorry.  Although rare, I didn't want to hide a real <get/>
>  > attribute.
>
>  Yeah, makes sense.
>
>  To be honest, I think it'd have been better to choose either
>  attribute or dict notation. Having both is confusing and leads to
>  divergent practices and documentation. Zope 2 made that mistake with
>  various things (like request keys) and it was never a good idea. 
FWIW, I like the choice.  Key notation is useful for cycling through a 
list of keys, and attribute notation is nice when you know your list of 
attributes before hand, and makes for cleaner code.  But you are right: 
it does require a coding standard to be in place.

j

-- 
Joshua Kugler
Part-Time System Admin/Programmer
http://www.eeinternet.com
PGP Key: http://pgp.mit.edu/  ID 0x73B13B6A