An Introduction to XSO

This document shall serve as an introduction to the aioxmpp.xso subpackage. This is intentionally separate from the API documentation and the glossary, since it should provide a high- and user-level introduction for those who first get into using it.

About aioxmpp.xso

Let us give you an introduction to the aioxmpp.xso package in form of answers to a few quick questions:

What is aioxmpp.xso?

It is a mapping layer between XML structured data (elements, text and attributes) and python objects. It is built with streaming in mind. SAX-compatible events are interpreted and converted to python objects on-the-fly. No DOM is needed or used.

If you have ever worked with Object-Relational Mappers for databases, such as sqlalchemy, XSO will feel familiar.

What is aioxmpp.xso not?

A replacement for a full-blown XML library. If you need the full XML 1.0+ DOM, XPath, XQuery and/or possibly XSLT to work with your data, XSO is not the right thing for you.

Specifically, the following XML 1.0 features are decidedly not supported in aioxmpp.xso:

  • Non-namespace-well-formed documents: All documents processed and generated by XSO are namespace well-formed.

  • Processing Insturctions

  • Comments

  • Document Type Declarations

  • Preservation of qualified names / namespace prefixes. They are semantically irrelevant: only the Namespace URI and Local Name matter.

  • Preservation of ordering between some elements. The following relative orders are specifically violated:

    • Text nodes vs. non-text nodes within the same parent element

    • Child elements which are handled by different descriptors. Often, only elements with the same Namespace URI and Local Name are handled by the same descriptor (see also XSO descriptors).

There may be other edge-case features we do not support.

How much Magic is inside aioxmpp.xso?

Hopefully not too much, but there’s still a bit. I’ll let you know that there is at least one metaclass involved, to handle the processing of descriptors at class-definition time and enforcing invariants during inheritance. Sorry for that.

Oh, and the use of generators as suspendable functions to make the parsing code easier to read.

Other than that, I think, it’s pretty standard Python though.

Into the Deep End / Very Quick Start

Let us jump right in:

>>> data = \
...   b"<node xmlns='urn:uuid:203ef66e-4423-49f2-90c9-3cb160986734'" \
...   b" a1='foo'>" \
...   b"<child>some text</child>" \
...   b"</node>"
>>> namespace = "urn:uuid:203ef66e-4423-49f2-90c9-3cb160986734"
>>> class Node(aioxmpp.xso.XSO):
...   TAG = namespace, "node"
...   attr = aioxmpp.xso.Attr("a1")
...   data = aioxmpp.xso.ChildText((namespace, "child"))
...
>>> buf = io.BytesIO(data)
>>> n = aioxmpp.xml.read_single_xso(buf, Node)
>>> isinstance(n, Node)
True
>>> n.attr
'foo'
>>> n.data
'some text'

Look, you just parsed your first XSO!

Note

The aioxmpp.xml module, which is technically not part of aioxmpp.xso, was also involved. This is because driving the XSO parser with SAX events from a bytes object requires quite some setup, and there are shorthands for that in aioxmpp.xml.

Let us walk through this step-by-step.

  1. data = ...: We simply set up a blob of data for us to parse. There should be nothing or at least not much special in there. It is simply an XML fragment with an element which has a single child element.

  2. class Node: This declares the XSO class. Inheriting from aioxmpp.xso.XSO is how you say “I want this to be parseable and serialisable from/to XML”. It is required for the descriptors to work.

    1. TAG = ...: This sets the namespace-uri/local-name pair which identifies this XSO. The identification is not global; thus, it is allowed to declare multiple XSO descendant classes with the same TAG.

    2. attr = aioxmpp.xso.Attr(...): aioxmpp.xso.Attr is a descriptor. It is understood by the aioxmpp.xso.XSO class and collected into bookkeeping attributes at class definition time. When an element needs to be parsed and it has attributes, the parsing function looks up the attribute tag in the bookkeeping and delegates processing of the attribute to the descriptor.

    3. data = aioxmpp.xso.ChildText(...): aioxmpp.xso.ChildText is another descriptor. In contrast to the Attr descriptor, this one handles child element events (and not attribute events). If a child element event matching the tag given as first argument to this descriptor, the parser delegates parsing of that element to the descriptor.

  3. buf = ...: Create a file-like from which the parser function can read.

  4. n = aioxmpp.xml.read_single_xso: Read a single XSO from a file-like object and save it into n.

  5. The following attribute accesses show how data has arrived in the instance of Node.

Again, if you have used an ORM before, how we declared Node should be very familiar to you.

A Bit of XSO Terminology

Now after the plunge into the deep end, let us get a bit of terminology straight so that it is clear what we’re talking about:

Character Data

Text or CDATA nodes in the XML document. Text and CDATA are treated the same by XSO (after the decoding handled by the XML library).

Element

An element node in an XML tree. An element node may hold child nodes, such as text nodes, other elements and attributes.

Tag

A tag is a pair consisting of a namespace-uri and a local-name. It is a fully-qualified name for an XML element. A common notation for tags is Clark’s Notation. For example {uri:foo}bar for a local name bar and a namespace URI uri:foo.

In XSO, tags are represented as tuples with two strings, reflecting the structure of the aforementioned pair.

XSO Type

Describes how to map XML data (character data or element subtrees) to python types and vice versa. Examples are aioxmpp.xso.Integer and aioxmpp.xso.EnumElementType.

XSO types can be categorized in two classes:

  1. Character Data Types, which map character data to python data structures (e.g. aioxmpp.xso.Integer).

  2. Element Types, which map XML subtrees to python data structures and vice versa (e.g. aioxmpp.xso.EnumElementType).

Not to be confused with a descendant of aioxmpp.xso.

Writing XSO classes

To write your own XSO class, you simply need a class which inherits (directly or indirectly) from aioxmpp.xso.XSO. Inheriting from that class allows the descriptors to work.

Note

Despite its intricacy, inheritance involving aioxmpp.xso.XSO descendants is fully supported. There are a few invariants which have to be maintained, however. Violating those invariants will raise an error at class definition time. In general, those invariants are common sense, but if you want to dig into the details, see aioxmpp.xso.model.XMLStreamClass.

XSO descriptors

The descriptors are the main component a user will come in contact with. They can be categorized into four categories:

Attribute Descriptors

which handle attribute nodes, i.e. attributes on the element which the XSO describes.

Text Descriptors

which handle text nodes, i.e. text content (including CDATA sections) inside the element which the XSO describes.

Scalar Child Descriptors

which handle (possibly different) child elements, but at most one of them.

For example, a scalar descriptor which captures one child element of either of two different types will at any time hold at most one child element; it cannot hold one of each type. Two different descriptors, or a non-scalar descriptor is needed for that.

Non-scalar Child Descriptors

which handle multiple child elements. These are then aggregated in different types of containers depending on the specific descriptor.

An overview of all descriptors, grouped by their category, follows. Please click through to the full classes at one point, because the one-liner description shown in this summary (as well as the abbreviated argument list) cannot describe the full potential.

Attribute Descriptors

Attr(tag, *[, type_, missing])

A single XML attribute.

LangAttr([default])

Special handler for the xml:lang attribute.

Text Descriptors

Text(*[, type_, erroneous_as_absent])

Character data contents of an XSO.

Scalar Child Descriptors

Child(classes[, required, strict])

A single child element of any of the given XSO types.

ChildTag(tags, *[, default_ns, text_policy, …])

Tag of a single child element with one of the given tags.

ChildFlag(tag[, text_policy, child_policy, …])

Presence of a child element with the given tag, as boolean.

ChildText(tag, *[, child_policy, …])

Character data of a single child element matching the given tag.

ChildValue(type_)

Child element parsed using an Element Type.

Non-scalar Child Descriptors

ChildList(classes)

List of child elements of any of the given XSO classes.

ChildMap(classes, *[, key])

Dictionary holding child elements of one or more XSO classes.

ChildValueList(type_, *[, container_type])

List of child elements parsed using the given Element Type.

ChildValueMap(type_, *[, mapping_type])

Dictiorary of child elements parsed using the given Element Type.

ChildValueMultiMap(type_, *[, mapping_type])

Multi-dict of child elements parsed using the given Element Type.

ChildLangMap(classes, **kwargs)

Shorthand for a dictionary of child elements keyed by the language attribute.

ChildTextMap(xso_type)

Dictionary of character data in child elements keyed by the language attribute.

Collector()

Catch-all descriptor collecting unhandled elements in an lxml element tree.

Handling of unexpected attributes and child elements

The handling of unexpected attributes and child elements on an XSO can be controlled at class definition time using two special attributes:

Note

Unexpected text is always treated as an error.

Character Data Types

XML data (beyond the structure) is strings only. However, most protocols built on top of XML will have types which are used for attributes and text content more specific than “string”.

For example, you’ll commonly find attributes which are integers or booleans and character data payloads which are base64-encoded binary. For the common types, aioxmpp.xso ships with type definitions:

aioxmpp.xso.String([prepfunc])

String Character Data Type, optionally with string preparation.

aioxmpp.xso.Float()

Floating point or decimal Character Data Type.

aioxmpp.xso.Integer()

Integer Character Data Type, to the base 10.

aioxmpp.xso.Bool()

XML boolean Character Data Type.

aioxmpp.xso.Base64Binary(*[, empty_as_equal])

Character Data Type for bytes encoded as base64.

aioxmpp.xso.HexBinary()

Character Data Type for bytes encoded as hexadecimal.

aioxmpp.xso.LanguageTag()

Character Data Type for language tags.

aioxmpp.xso.JSON()

Character Data Type for JSON formatted data.

Some more XMPP specific types are:

aioxmpp.xso.DateTime(*[, legacy])

ISO datetime Character Data Type.

aioxmpp.xso.Date()

ISO date Character Data Type.

aioxmpp.xso.Time()

ISO time Character Data Type.

aioxmpp.xso.JID(*[, strict])

Character Data Type for aioxmpp.JID objects.

aioxmpp.xso.ConnectionLocation()

Character Data Type for a hostname-port pair.

Note

“What is XMPP-specific about the date types?” you may very well ask. They do not implement the full syntax of xml schema date, datetime and time data type definitions.

They should work for most of those values, but some edge-cases (such as years outside of the range 0..9999) are not handled. See also XEP-0082.

The types above can be used anywhere where XSO character data types are needed. Which in turn is every place where XSO handles XML character data, so that’s attributes (Attr) and text nodes (e.g. ChildText and Text).

Combining the Above in an Example

We’ve given you lots of theoretical stuff to chew on. Let us put this in practice with a more sophisticated example.

Hopefully, with the above explanations and the links into the reference documentation, you will be able to understand this example. If you are not, I did a bad job at writing this documentation. In that case, I very much would like to hear about it to improve it in the future!

Take this bit of code:

import aioxmpp.xso

namespace = "urn:uuid:39ba7586-fb65-4ec8-80ce-f3a9f2890490"

class Chapter(aioxmpp.xso.XSO):
    TAG = namespace, "chapter"

    title = aioxmpp.xso.ChildTextMap((namespace, "title"))
    start_page = aioxmpp.xso.Attr(
        "start-page",
        type_=aioxmpp.xso.Integer()
    )


class TableOfContents(aioxmpp.xso.XSO):
    TAG = namespace, "toc"

    chapters = aioxmpp.xso.ChildList([Chapter])


class Book(aioxmpp.xso.XSO):
    TAG = namespace, "book"

    id_ = aioxmpp.xso.Attr("id")
    author = aioxmpp.xso.ChildText((namespace, "author"))
    npages = aioxmpp.xso.ChildText(
        (namespace, "pages"),
        type_=aioxmpp.xso.Integer(),
    )
    published = aioxmpp.xso.ChildText(
        (namespace, "published"),
        type_=aioxmpp.xso.Date(),
    )
    title = aioxmpp.xso.ChildTextMap((namespace, "title"))
    toc = aioxmpp.xso.Child([TableOfContents])


class Library(aioxmpp.xso.XSO):
    TAG = namespace, "library"

    books = aioxmpp.xso.ChildList([Book])

It declares one of the classic examples of XML teaching: a book collection. Save the above snippet as library_demo.py. Then we can read an XML file with a Library shaped root element using the following snippet:

import sys
import aioxmpp.xml
import library_demo

with open(sys.argv[1], "r") as f:
    library = aioxmpp.xml.read_single_xso(f, library_demo.Library)

for book in library.books:
    print("book (id = {!r}):".format(book.id_))
    print("  author:", book.author)
    print("  published:", book.published)
    print("  npages:", book.npages)
    print("  title:")
    for lang, title in book.title.items():
    print("    [{!s}] {!r}".format(lang, title))
    print("  table of contents:")
    for i, chapter in enumerate(book.toc.chapters, 1):
        print("    {}.    (page {})".format(i, chapter.start_page))
        for lang, title in chapter.title.items():
            print("      [{!s}] {!r}".format(lang, title))

Save that file as library_load.py and try it on the following XML file (library_test.xml):

<?xml version="1.0"?>
<library xmlns="urn:uuid:39ba7586-fb65-4ec8-80ce-f3a9f2890490">
  <book id="foo">
    <title xml:lang="en">The Amazing Life of Foo</title>
    <title xml:lang="de">Das Faszinierende Leben des Foo</title>
    <author>F. Nord</author>
    <published>2099-01-01</published>
    <pages>23</pages>
    <toc>
      <chapter start-page="1">
        <title xml:lang="en">The Birth of Foo</title>
        <title xml:lang="de">Die Geburt des Foo</title>
      </chapter>
      <chapter start-page="3">
        <title xml:lang="en">The Death of Foo</title>
        <title xml:lang="de">Der Tod des Foo</title>
      </chapter>
    </toc>
  </book>
  <book id="pink-flamingos">
    <title xml:lang="en">The Relevance of Pink Flamingos to Computer Science</title>
    <title xml:lang="de">Die Relevanz von rosa Flamingos für die Informatik</title>
    <author>O. L. Bilderrahmen</author>
    <published>2007-01-01</published>
    <pages>42</pages>
    <toc/>
  </book>
</library>

Try it:

$ python3 library_load.py library_test.xml
book (id = 'foo'):
  author: F. Nord
  published: 2099-01-01
  npages: 23
  title:
    [en] 'The Amazing Life of Foo'
    [de] 'Das Faszinierende Leben des Foo'
  table of contents:
    1.    (page 1)
      [en] 'The Birth of Foo'
      [de] 'Die Geburt des Foo'
    2.    (page 3)
      [en] 'The Death of Foo'
      [de] 'Der Tod des Foo'
book (id = 'pink-flamingos'):
  author: O. L. Bilderrahmen
  published: 2007-01-01
  npages: 42
  title:
    [en] 'The Relevance of Pink Flamingos to Computer Science'
    [de] 'Die Relevanz von rosa Flamingos für die Informatik'
  table of contents:

Reference Documentation

To learn more about XSO and how to use it, check out the reference documentation in aioxmpp.xso. The remainder of this documentation will now dive deeper into the details on how XSO works.