.. _ug-introduction-to-xso: An Introduction to XSO ###################### This document shall serve as an introduction to the :mod:`aioxmpp.xso` subpackage. This is intentionally separate from the API documentation and the glossary, since it should provide a high- and user-level introduction for those who first get into using it. About :mod:`aioxmpp.xso` ======================== Let us give you an introduction to the :mod:`aioxmpp.xso` package in form of answers to a few quick questions: What is :mod:`aioxmpp.xso`? --------------------------- It is a mapping layer between XML structured data (elements, text and attributes) and python objects. It is built with streaming in mind. SAX-compatible events are interpreted and converted to python objects on-the-fly. No DOM is needed or used. If you have ever worked with Object-Relational Mappers for databases, such as :mod:`sqlalchemy`, XSO will feel familiar. What is :mod:`aioxmpp.xso` **not**? ----------------------------------- A replacement for a full-blown XML library. If you need the full XML 1.0+ DOM, XPath, XQuery and/or possibly XSLT to work with your data, XSO is not the right thing for you. Specifically, the following XML 1.0 features are decidedly **not supported** in :mod:`aioxmpp.xso`: - Non-namespace-well-formed documents: All documents processed and generated by XSO are namespace well-formed. - Processing Insturctions - Comments - Document Type Declarations - Preservation of qualified names / namespace prefixes. They are semantically irrelevant: only the :term:`Namespace URI` and :term:`Local Name` matter. - Preservation of ordering between some elements. The following relative orders are specifically violated: - Text nodes vs. non-text nodes within the same parent element - Child elements which are handled by different descriptors. Often, only elements with the same :term:`Namespace URI` and :term:`Local Name` are handled by the same descriptor (see also :ref:`ug-introduction-to-xso-descriptors`). There may be other edge-case features we do not support. How much `Magic`_ is inside :mod:`aioxmpp.xso`? ----------------------------------------------- Hopefully not too much, but there’s still a bit. I’ll let you know that there is at least one metaclass involved, to handle the processing of descriptors at class-definition time and enforcing invariants during inheritance. Sorry for that. Oh, and the use of generators as suspendable functions to make the parsing code easier to read. Other than that, I think, it’s pretty standard Python though. .. _Magic: http://www.catb.org/jargon/html/M/magic.html Into the Deep End / Very Quick Start ==================================== Let us jump right in: .. code-block:: python >>> data = \ ... b"" \ ... b"some text" \ ... b"" >>> namespace = "urn:uuid:203ef66e-4423-49f2-90c9-3cb160986734" >>> class Node(aioxmpp.xso.XSO): ... TAG = namespace, "node" ... attr = aioxmpp.xso.Attr("a1") ... data = aioxmpp.xso.ChildText((namespace, "child")) ... >>> buf = io.BytesIO(data) >>> n = aioxmpp.xml.read_single_xso(buf, Node) >>> isinstance(n, Node) True >>> n.attr 'foo' >>> n.data 'some text' Look, you just parsed your first XSO! .. note:: The :mod:`aioxmpp.xml` module, which is technically not part of :mod:`aioxmpp.xso`, was also involved. This is because *driving* the XSO parser with SAX events from a bytes object requires quite some setup, and there are shorthands for that in :mod:`aioxmpp.xml`. Let us walk through this step-by-step. 1. ``data = ...``: We simply set up a blob of data for us to parse. There should be nothing or at least not much special in there. It is simply an XML fragment with an element which has a single child element. 2. ``class Node``: This declares the XSO class. Inheriting from :class:`aioxmpp.xso.XSO` is how you say "I want this to be parseable and serialisable from/to XML". It is required for the descriptors to work. 1. ``TAG = ...``: This sets the :term:`namespace-uri`/:term:`local-name` pair which identifies this XSO. The identification is not global; thus, it is allowed to declare multiple XSO descendant classes with the same TAG. 2. ``attr = aioxmpp.xso.Attr(...)``: :class:`aioxmpp.xso.Attr` is a descriptor. It is understood by the :class:`aioxmpp.xso.XSO` class and collected into bookkeeping attributes at class definition time. When an element needs to be parsed and it has attributes, the parsing function looks up the attribute tag in the bookkeeping and delegates processing of the attribute to the descriptor. 3. ``data = aioxmpp.xso.ChildText(...)``: :class:`aioxmpp.xso.ChildText` is another descriptor. In contrast to the :class:`~aioxmpp.xso.Attr` descriptor, this one handles child element events (and not attribute events). If a child element event matching the tag given as first argument to this descriptor, the parser delegates parsing of that element to the descriptor. 3. ``buf = ...``: Create a file-like from which the parser function can read. 4. ``n = aioxmpp.xml.read_single_xso``: Read a single XSO from a file-like object and save it into ``n``. 5. The following attribute accesses show how data has arrived in the instance of ``Node``. Again, if you have used an ORM before, how we declared `Node` should be very familiar to you. .. _ug-introduction-to-xso-terminology: A Bit of XSO Terminology ======================== Now after the plunge into the deep end, let us get a bit of terminology straight so that it is clear what we're talking about: Character Data Text or CDATA nodes in the XML document. Text and CDATA are treated the same by XSO (after the decoding handled by the XML library). Element An element node in an XML tree. An element node may hold child nodes, such as text nodes, other elements and attributes. Tag A tag is a pair consisting of a :term:`namespace-uri` and a :term:`local-name`. It is a fully-qualified name for an XML element. A common notation for tags is `Clark’s Notation `_. For example ``{uri:foo}bar`` for a local name ``bar`` and a namespace URI ``uri:foo``. In XSO, tags are represented as tuples with two strings, reflecting the structure of the aforementioned pair. XSO Type Describes how to map XML data (character data or element subtrees) to python types and vice versa. Examples are :class:`aioxmpp.xso.Integer` and :class:`aioxmpp.xso.EnumElementType`. XSO types can be categorized in two classes: 1. :term:`Character Data Types `, which map character data to python data structures (e.g. :class:`aioxmpp.xso.Integer`). 2. :term:`Element Types `, which map XML subtrees to python data structures and vice versa (e.g. :class:`aioxmpp.xso.EnumElementType`). Not to be confused with a descendant of :mod:`aioxmpp.xso`. Writing XSO classes =================== To write your own XSO class, you simply need a class which inherits (directly or indirectly) from :class:`aioxmpp.xso.XSO`. Inheriting from that class allows the descriptors to work. .. note:: Despite its intricacy, inheritance involving :class:`aioxmpp.xso.XSO` descendants is fully supported. There are a few invariants which have to be maintained, however. Violating those invariants will raise an error at class definition time. In general, those invariants are common sense, but if you want to dig into the details, see :class:`aioxmpp.xso.model.XMLStreamClass`. .. _ug-introduction-to-xso-descriptors: XSO descriptors --------------- The descriptors are the main component a user will come in contact with. They can be categorized into four categories: *Attribute Descriptors* which handle attribute nodes, i.e. attributes on the element which the XSO describes. *Text Descriptors* which handle text nodes, i.e. text content (including CDATA sections) inside the element which the XSO describes. *Scalar Child Descriptors* which handle (possibly different) child elements, but at most one of them. For example, a scalar descriptor which captures one child element of either of two different types will at any time hold at most one child element; it cannot hold one of each type. Two different descriptors, or a non-scalar descriptor is needed for that. *Non-scalar Child Descriptors* which handle multiple child elements. These are then aggregated in different types of containers depending on the specific descriptor. An overview of all descriptors, grouped by their category, follows. Please click through to the full classes at one point, because the one-liner description shown in this summary (as well as the abbreviated argument list) cannot describe the full potential. Attribute Descriptors ^^^^^^^^^^^^^^^^^^^^^ .. autosummary:: ~aioxmpp.xso.Attr ~aioxmpp.xso.LangAttr Text Descriptors ^^^^^^^^^^^^^^^^ .. autosummary:: ~aioxmpp.xso.Text Scalar Child Descriptors ^^^^^^^^^^^^^^^^^^^^^^^^ .. autosummary:: ~aioxmpp.xso.Child ~aioxmpp.xso.ChildTag ~aioxmpp.xso.ChildFlag ~aioxmpp.xso.ChildText ~aioxmpp.xso.ChildValue Non-scalar Child Descriptors ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. autosummary:: ~aioxmpp.xso.ChildList ~aioxmpp.xso.ChildMap ~aioxmpp.xso.ChildValueList ~aioxmpp.xso.ChildValueMap ~aioxmpp.xso.ChildValueMultiMap ~aioxmpp.xso.ChildLangMap ~aioxmpp.xso.ChildTextMap ~aioxmpp.xso.Collector Handling of unexpected attributes and child elements ---------------------------------------------------- The handling of unexpected attributes and child elements on an XSO can be controlled at class definition time using two special attributes: * :attr:`aioxmpp.xso.XSO.UNKNOWN_CHILD_POLICY` to control how unknown children are handled. The possible values are :attr:`.UnknownChildPolicy.DROP` (the default), which simply ignores such child elements and :attr:`.UnknownChildPolicy.FAIL` which raises an exception. * :attr:`aioxmpp.xso.XSO.UNKNOWN_ATTR_POLICY` to control how unknown attributes are handled. The possible values are :attr:`.UnknownAttrPolicy.DROP` (the default), which simply ignores such attributes and :attr:`.UnknownAttrPolicy.FAIL` which raises an exception. .. note:: Unexpected text is always treated as an error. Character Data Types -------------------- XML data (beyond the structure) is strings only. However, most protocols built on top of XML will have types which are used for attributes and text content more specific than "string". For example, you’ll commonly find attributes which are integers or booleans and character data payloads which are base64-encoded binary. For the common types, :mod:`aioxmpp.xso` ships with type definitions: .. autosummary:: aioxmpp.xso.String aioxmpp.xso.Float aioxmpp.xso.Integer aioxmpp.xso.Bool aioxmpp.xso.Base64Binary aioxmpp.xso.HexBinary aioxmpp.xso.LanguageTag aioxmpp.xso.JSON Some more XMPP specific types are: .. autosummary:: aioxmpp.xso.DateTime aioxmpp.xso.Date aioxmpp.xso.Time aioxmpp.xso.JID aioxmpp.xso.ConnectionLocation .. note:: "What is XMPP-specific about the date types?" you may very well ask. They do not implement the full syntax of xml schema date, datetime and time data type definitions. They should work for most of those values, but some edge-cases (such as years outside of the range 0..9999) are not handled. See also :xep:`82`. The types above can be used anywhere where XSO character data types are needed. Which in turn is every place where XSO handles XML character data, so that’s attributes (:class:`~aioxmpp.xso.Attr`) and text nodes (e.g. :class:`~aioxmpp.xso.ChildText` and :class:`~aioxmpp.xso.Text`). Combining the Above in an Example --------------------------------- We’ve given you lots of theoretical stuff to chew on. Let us put this in practice with a more sophisticated example. Hopefully, with the above explanations and the links into the reference documentation, you will be able to understand this example. If you are not, I did a bad job at writing this documentation. In that case, I very much would like to `hear about it `_ to improve it in the future! Take this bit of code: .. code-block:: python import aioxmpp.xso namespace = "urn:uuid:39ba7586-fb65-4ec8-80ce-f3a9f2890490" class Chapter(aioxmpp.xso.XSO): TAG = namespace, "chapter" title = aioxmpp.xso.ChildTextMap((namespace, "title")) start_page = aioxmpp.xso.Attr( "start-page", type_=aioxmpp.xso.Integer() ) class TableOfContents(aioxmpp.xso.XSO): TAG = namespace, "toc" chapters = aioxmpp.xso.ChildList([Chapter]) class Book(aioxmpp.xso.XSO): TAG = namespace, "book" id_ = aioxmpp.xso.Attr("id") author = aioxmpp.xso.ChildText((namespace, "author")) npages = aioxmpp.xso.ChildText( (namespace, "pages"), type_=aioxmpp.xso.Integer(), ) published = aioxmpp.xso.ChildText( (namespace, "published"), type_=aioxmpp.xso.Date(), ) title = aioxmpp.xso.ChildTextMap((namespace, "title")) toc = aioxmpp.xso.Child([TableOfContents]) class Library(aioxmpp.xso.XSO): TAG = namespace, "library" books = aioxmpp.xso.ChildList([Book]) It declares one of the classic examples of XML teaching: a book collection. Save the above snippet as ``library_demo.py``. Then we can read an XML file with a ``Library`` shaped root element using the following snippet: .. code-block:: python import sys import aioxmpp.xml import library_demo with open(sys.argv[1], "r") as f: library = aioxmpp.xml.read_single_xso(f, library_demo.Library) for book in library.books: print("book (id = {!r}):".format(book.id_)) print(" author:", book.author) print(" published:", book.published) print(" npages:", book.npages) print(" title:") for lang, title in book.title.items(): print(" [{!s}] {!r}".format(lang, title)) print(" table of contents:") for i, chapter in enumerate(book.toc.chapters, 1): print(" {}. (page {})".format(i, chapter.start_page)) for lang, title in chapter.title.items(): print(" [{!s}] {!r}".format(lang, title)) Save that file as ``library_load.py`` and try it on the following XML file (``library_test.xml``): .. code-block:: xml The Amazing Life of Foo Das Faszinierende Leben des Foo F. Nord 2099-01-01 23 The Birth of Foo Die Geburt des Foo The Death of Foo Der Tod des Foo The Relevance of Pink Flamingos to Computer Science Die Relevanz von rosa Flamingos für die Informatik O. L. Bilderrahmen 2007-01-01 42 Try it: .. code-block:: console $ python3 library_load.py library_test.xml book (id = 'foo'): author: F. Nord published: 2099-01-01 npages: 23 title: [en] 'The Amazing Life of Foo' [de] 'Das Faszinierende Leben des Foo' table of contents: 1. (page 1) [en] 'The Birth of Foo' [de] 'Die Geburt des Foo' 2. (page 3) [en] 'The Death of Foo' [de] 'Der Tod des Foo' book (id = 'pink-flamingos'): author: O. L. Bilderrahmen published: 2007-01-01 npages: 42 title: [en] 'The Relevance of Pink Flamingos to Computer Science' [de] 'Die Relevanz von rosa Flamingos für die Informatik' table of contents: Reference Documentation ======================= To learn more about XSO and how to use it, check out the reference documentation in :mod:`aioxmpp.xso`. The remainder of this documentation will now dive deeper into the details on how XSO works.