.. _ug-introduction-to-xso:
An Introduction to XSO
######################
This document shall serve as an introduction to the :mod:`aioxmpp.xso`
subpackage. This is intentionally separate from the API documentation and
the glossary, since it should provide a high- and user-level introduction for
those who first get into using it.
About :mod:`aioxmpp.xso`
========================
Let us give you an introduction to the :mod:`aioxmpp.xso` package in form of
answers to a few quick questions:
What is :mod:`aioxmpp.xso`?
---------------------------
It is a mapping layer between XML structured data (elements, text and
attributes) and python objects. It is built with streaming in mind.
SAX-compatible events are interpreted and converted to python objects
on-the-fly. No DOM is needed or used.
If you have ever worked with Object-Relational Mappers for databases, such as
:mod:`sqlalchemy`, XSO will feel familiar.
What is :mod:`aioxmpp.xso` **not**?
-----------------------------------
A replacement for a full-blown XML library. If you need the full XML 1.0+ DOM,
XPath, XQuery and/or possibly XSLT to work with your data, XSO is not the right
thing for you.
Specifically, the following XML 1.0 features are decidedly **not supported** in
:mod:`aioxmpp.xso`:
- Non-namespace-well-formed documents: All documents processed and generated by
XSO are namespace well-formed.
- Processing Insturctions
- Comments
- Document Type Declarations
- Preservation of qualified names / namespace prefixes. They are semantically
irrelevant: only the :term:`Namespace URI` and :term:`Local Name` matter.
- Preservation of ordering between some elements. The following relative orders
are specifically violated:
- Text nodes vs. non-text nodes within the same parent element
- Child elements which are handled by different descriptors. Often, only
elements with the same :term:`Namespace URI` and :term:`Local Name` are
handled by the same descriptor (see also
:ref:`ug-introduction-to-xso-descriptors`).
There may be other edge-case features we do not support.
How much `Magic`_ is inside :mod:`aioxmpp.xso`?
-----------------------------------------------
Hopefully not too much, but there’s still a bit. I’ll let you know that there
is at least one metaclass involved, to handle the processing of descriptors at
class-definition time and enforcing invariants during inheritance. Sorry for
that.
Oh, and the use of generators as suspendable functions to make the parsing code
easier to read.
Other than that, I think, it’s pretty standard Python though.
.. _Magic: http://www.catb.org/jargon/html/M/magic.html
Into the Deep End / Very Quick Start
====================================
Let us jump right in:
.. code-block:: python
>>> data = \
... b"" \
... b"some text" \
... b""
>>> namespace = "urn:uuid:203ef66e-4423-49f2-90c9-3cb160986734"
>>> class Node(aioxmpp.xso.XSO):
... TAG = namespace, "node"
... attr = aioxmpp.xso.Attr("a1")
... data = aioxmpp.xso.ChildText((namespace, "child"))
...
>>> buf = io.BytesIO(data)
>>> n = aioxmpp.xml.read_single_xso(buf, Node)
>>> isinstance(n, Node)
True
>>> n.attr
'foo'
>>> n.data
'some text'
Look, you just parsed your first XSO!
.. note::
The :mod:`aioxmpp.xml` module, which is technically not part of
:mod:`aioxmpp.xso`, was also involved. This is because *driving* the XSO
parser with SAX events from a bytes object requires quite some setup, and
there are shorthands for that in :mod:`aioxmpp.xml`.
Let us walk through this step-by-step.
1. ``data = ...``: We simply set up a blob of data for us to parse. There
should be nothing or at least not much special in there. It is simply an
XML fragment with an element which has a single child element.
2. ``class Node``: This declares the XSO class. Inheriting from
:class:`aioxmpp.xso.XSO` is how you say "I want this to be parseable and
serialisable from/to XML". It is required for the descriptors to work.
1. ``TAG = ...``: This sets the :term:`namespace-uri`/:term:`local-name`
pair which identifies this XSO. The identification is not global; thus,
it is allowed to declare multiple XSO descendant classes with the same
TAG.
2. ``attr = aioxmpp.xso.Attr(...)``: :class:`aioxmpp.xso.Attr` is a
descriptor. It is understood by the :class:`aioxmpp.xso.XSO` class and
collected into bookkeeping attributes at class definition time. When
an element needs to be parsed and it has attributes, the parsing function
looks up the attribute tag in the bookkeeping and delegates processing of
the attribute to the descriptor.
3. ``data = aioxmpp.xso.ChildText(...)``: :class:`aioxmpp.xso.ChildText` is
another descriptor. In contrast to the :class:`~aioxmpp.xso.Attr`
descriptor, this one handles child element events (and not attribute
events). If a child element event matching the tag given as first
argument to this descriptor, the parser delegates parsing of that element
to the descriptor.
3. ``buf = ...``: Create a file-like from which the parser function can read.
4. ``n = aioxmpp.xml.read_single_xso``: Read a single XSO from a file-like
object and save it into ``n``.
5. The following attribute accesses show how data has arrived in the instance
of ``Node``.
Again, if you have used an ORM before, how we declared `Node` should be very
familiar to you.
.. _ug-introduction-to-xso-terminology:
A Bit of XSO Terminology
========================
Now after the plunge into the deep end, let us get a bit of terminology
straight so that it is clear what we're talking about:
Character Data
Text or CDATA nodes in the XML document. Text and CDATA are treated the
same by XSO (after the decoding handled by the XML library).
Element
An element node in an XML tree. An element node may hold child nodes,
such as text nodes, other elements and attributes.
Tag
A tag is a pair consisting of a :term:`namespace-uri` and a
:term:`local-name`. It is a fully-qualified name for an XML element. A
common notation for tags is
`Clark’s Notation `_. For example
``{uri:foo}bar`` for a local name ``bar`` and a namespace URI
``uri:foo``.
In XSO, tags are represented as tuples with two strings, reflecting the
structure of the aforementioned pair.
XSO Type
Describes how to map XML data (character data or element subtrees) to
python types and vice versa. Examples are :class:`aioxmpp.xso.Integer`
and :class:`aioxmpp.xso.EnumElementType`.
XSO types can be categorized in two classes:
1. :term:`Character Data Types `, which map character
data to python data structures (e.g. :class:`aioxmpp.xso.Integer`).
2. :term:`Element Types `, which map XML subtrees to python
data structures and vice versa (e.g.
:class:`aioxmpp.xso.EnumElementType`).
Not to be confused with a descendant of :mod:`aioxmpp.xso`.
Writing XSO classes
===================
To write your own XSO class, you simply need a class which inherits (directly
or indirectly) from :class:`aioxmpp.xso.XSO`. Inheriting from that class allows
the descriptors to work.
.. note::
Despite its intricacy, inheritance involving :class:`aioxmpp.xso.XSO`
descendants is fully supported. There are a few invariants which have to be
maintained, however. Violating those invariants will raise an error at
class definition time. In general, those invariants are common sense, but
if you want to dig into the details, see
:class:`aioxmpp.xso.model.XMLStreamClass`.
.. _ug-introduction-to-xso-descriptors:
XSO descriptors
---------------
The descriptors are the main component a user will come in contact with. They
can be categorized into four categories:
*Attribute Descriptors*
which handle attribute nodes, i.e. attributes on the element which the XSO
describes.
*Text Descriptors*
which handle text nodes, i.e. text content (including CDATA sections)
inside the element which the XSO describes.
*Scalar Child Descriptors*
which handle (possibly different) child elements, but at most one of them.
For example, a scalar descriptor which captures one child element of either
of two different types will at any time hold at most one child element; it
cannot hold one of each type. Two different descriptors, or a non-scalar
descriptor is needed for that.
*Non-scalar Child Descriptors*
which handle multiple child elements. These are then aggregated in
different types of containers depending on the specific descriptor.
An overview of all descriptors, grouped by their category, follows. Please
click through to the full classes at one point, because the one-liner
description shown in this summary (as well as the abbreviated argument list)
cannot describe the full potential.
Attribute Descriptors
^^^^^^^^^^^^^^^^^^^^^
.. autosummary::
~aioxmpp.xso.Attr
~aioxmpp.xso.LangAttr
Text Descriptors
^^^^^^^^^^^^^^^^
.. autosummary::
~aioxmpp.xso.Text
Scalar Child Descriptors
^^^^^^^^^^^^^^^^^^^^^^^^
.. autosummary::
~aioxmpp.xso.Child
~aioxmpp.xso.ChildTag
~aioxmpp.xso.ChildFlag
~aioxmpp.xso.ChildText
~aioxmpp.xso.ChildValue
Non-scalar Child Descriptors
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. autosummary::
~aioxmpp.xso.ChildList
~aioxmpp.xso.ChildMap
~aioxmpp.xso.ChildValueList
~aioxmpp.xso.ChildValueMap
~aioxmpp.xso.ChildValueMultiMap
~aioxmpp.xso.ChildLangMap
~aioxmpp.xso.ChildTextMap
~aioxmpp.xso.Collector
Handling of unexpected attributes and child elements
----------------------------------------------------
The handling of unexpected attributes and child elements on an XSO can be
controlled at class definition time using two special attributes:
* :attr:`aioxmpp.xso.XSO.UNKNOWN_CHILD_POLICY` to control how unknown children
are handled. The possible values are :attr:`.UnknownChildPolicy.DROP` (the
default), which simply ignores such child elements and
:attr:`.UnknownChildPolicy.FAIL` which raises an exception.
* :attr:`aioxmpp.xso.XSO.UNKNOWN_ATTR_POLICY` to control how unknown attributes
are handled. The possible values are :attr:`.UnknownAttrPolicy.DROP` (the
default), which simply ignores such attributes and
:attr:`.UnknownAttrPolicy.FAIL` which raises an exception.
.. note::
Unexpected text is always treated as an error.
Character Data Types
--------------------
XML data (beyond the structure) is strings only. However, most protocols built
on top of XML will have types which are used for attributes and text content
more specific than "string".
For example, you’ll commonly find attributes which are integers or booleans and
character data payloads which are base64-encoded binary. For the common types,
:mod:`aioxmpp.xso` ships with type definitions:
.. autosummary::
aioxmpp.xso.String
aioxmpp.xso.Float
aioxmpp.xso.Integer
aioxmpp.xso.Bool
aioxmpp.xso.Base64Binary
aioxmpp.xso.HexBinary
aioxmpp.xso.LanguageTag
aioxmpp.xso.JSON
Some more XMPP specific types are:
.. autosummary::
aioxmpp.xso.DateTime
aioxmpp.xso.Date
aioxmpp.xso.Time
aioxmpp.xso.JID
aioxmpp.xso.ConnectionLocation
.. note::
"What is XMPP-specific about the date types?" you may very well ask. They
do not implement the full syntax of xml schema date, datetime and time
data type definitions.
They should work for most of those values, but some edge-cases (such as
years outside of the range 0..9999) are not handled. See also :xep:`82`.
The types above can be used anywhere where XSO character data types are needed.
Which in turn is every place where XSO handles XML character data, so that’s
attributes (:class:`~aioxmpp.xso.Attr`) and text nodes (e.g. :class:`~aioxmpp.xso.ChildText` and :class:`~aioxmpp.xso.Text`).
Combining the Above in an Example
---------------------------------
We’ve given you lots of theoretical stuff to chew on. Let us put this in
practice with a more sophisticated example.
Hopefully, with the above explanations and the links into the reference
documentation, you will be able to understand this example. If you are not, I
did a bad job at writing this documentation. In that case, I very much would
like to `hear about it `_ to
improve it in the future!
Take this bit of code:
.. code-block:: python
import aioxmpp.xso
namespace = "urn:uuid:39ba7586-fb65-4ec8-80ce-f3a9f2890490"
class Chapter(aioxmpp.xso.XSO):
TAG = namespace, "chapter"
title = aioxmpp.xso.ChildTextMap((namespace, "title"))
start_page = aioxmpp.xso.Attr(
"start-page",
type_=aioxmpp.xso.Integer()
)
class TableOfContents(aioxmpp.xso.XSO):
TAG = namespace, "toc"
chapters = aioxmpp.xso.ChildList([Chapter])
class Book(aioxmpp.xso.XSO):
TAG = namespace, "book"
id_ = aioxmpp.xso.Attr("id")
author = aioxmpp.xso.ChildText((namespace, "author"))
npages = aioxmpp.xso.ChildText(
(namespace, "pages"),
type_=aioxmpp.xso.Integer(),
)
published = aioxmpp.xso.ChildText(
(namespace, "published"),
type_=aioxmpp.xso.Date(),
)
title = aioxmpp.xso.ChildTextMap((namespace, "title"))
toc = aioxmpp.xso.Child([TableOfContents])
class Library(aioxmpp.xso.XSO):
TAG = namespace, "library"
books = aioxmpp.xso.ChildList([Book])
It declares one of the classic examples of XML teaching: a book collection.
Save the above snippet as ``library_demo.py``. Then we can read an XML file
with a ``Library`` shaped root element using the following snippet:
.. code-block:: python
import sys
import aioxmpp.xml
import library_demo
with open(sys.argv[1], "r") as f:
library = aioxmpp.xml.read_single_xso(f, library_demo.Library)
for book in library.books:
print("book (id = {!r}):".format(book.id_))
print(" author:", book.author)
print(" published:", book.published)
print(" npages:", book.npages)
print(" title:")
for lang, title in book.title.items():
print(" [{!s}] {!r}".format(lang, title))
print(" table of contents:")
for i, chapter in enumerate(book.toc.chapters, 1):
print(" {}. (page {})".format(i, chapter.start_page))
for lang, title in chapter.title.items():
print(" [{!s}] {!r}".format(lang, title))
Save that file as ``library_load.py`` and try it on the following XML file
(``library_test.xml``):
.. code-block:: xml
The Amazing Life of Foo
Das Faszinierende Leben des Foo
F. Nord
2099-01-01
23
The Birth of Foo
Die Geburt des Foo
The Death of Foo
Der Tod des Foo
The Relevance of Pink Flamingos to Computer Science
Die Relevanz von rosa Flamingos für die Informatik
O. L. Bilderrahmen
2007-01-01
42
Try it:
.. code-block:: console
$ python3 library_load.py library_test.xml
book (id = 'foo'):
author: F. Nord
published: 2099-01-01
npages: 23
title:
[en] 'The Amazing Life of Foo'
[de] 'Das Faszinierende Leben des Foo'
table of contents:
1. (page 1)
[en] 'The Birth of Foo'
[de] 'Die Geburt des Foo'
2. (page 3)
[en] 'The Death of Foo'
[de] 'Der Tod des Foo'
book (id = 'pink-flamingos'):
author: O. L. Bilderrahmen
published: 2007-01-01
npages: 42
title:
[en] 'The Relevance of Pink Flamingos to Computer Science'
[de] 'Die Relevanz von rosa Flamingos für die Informatik'
table of contents:
Reference Documentation
=======================
To learn more about XSO and how to use it, check out the reference
documentation in :mod:`aioxmpp.xso`. The remainder of this documentation will
now dive deeper into the details on how XSO works.