Myths

(Draft blog post)

Myths and Legends

From where I sit typing this, I can look out of my window and see Bryn Myrddin - Merlin's Hill - underneath which, legend says, Merlin lies asleep, awaiting a time when England in is peril once more. This is a problematic myth, since Bryn Myrddin isn't in England, but nevertheless, it's enough so I can convince myself somewhat that I'm something of an expert on the mythical.

XMPP seems to have some interesting myths, too. But just as Bryn Myrddin's real interest is the ancient hill fort on top, many of these are made up. Here's some of my (least) favourite:

Myth One: XMPP is XML, so it's too slow.

The hypothesis: XML is inarguably slower than JSON and other similar syntaxes, and XMPP will subsequently suffer in terms of performance.

The fact: XML is just as fast as JSON, and therefore XMPP pays no penalty.

JSON, in particular, is often heralded as being better in all possible ways than XML. In reality, while JSON can be constructed in slightly fewer bytes, that's really it's only advantage. You may think this is a key advantage for mobile, in which case hold on there for Myth Three.

In "typical" parsers, XML and JSON perform more or less identically in practice. It's worth noting that there are a number of high-performance XML parsers, but comparatively few high-performance JSON parsers, too.

But even ignoring the high performance, specialist end, XML still holds its own. Consider these findings, for example: http://balisage.net/Proceedings/vol10/html/Lee01/BalisageVol10-Lee01.html

Myth Two: The baseline is minimal, therefore XMPP is useless.

The hypothesis: The core of XMPP is so small is has no practical use, and features that are essential for my use case have been bolted on as an afterthought.

The fact: XMPP is designed to be extensible, and many extensions have very broad deployment.

"It's a feature, not a bug" is the obvious thing to say here. XMPP's core, RFC 6120, doesn't even include the "classic" IM features, really; those are present in it's companion specification, RFC 6121. As a protocol that's nearly fifteen years old, had the core been constantly redesigned to add new capabilities as new mandatory, baked-in features, lots of existing servers would have been redeclared as broken.

Instead, what we have is a pretty large number of extensions to a clean core. This core defines, amongst other things, how to extend the core properties of the connection - but in fact the community designed an end-to-end extension system early on.

While you can still connect a client that supports no extensions at all to a server and have it work, nearly every client supports a wide range of extensions suited to its purpose, and every server supports several.

Myth Three: It's too bandwidth-inefficient for mobile.

The hypothesis: XMPP stanzas are just too verbose to work on mobile.

The fact: XMPP has been successfully deployed over SATCOM and HF.

You might think that, because XML is known to be relatively verbose compared to other serialization formats, that this would be a problem in low bandwidth. However, XMPP's use of XML is actually quite concise - it's really nothing like the uses you see in, say, SOAP - and XMPP has been used (and is still used) over some ridiculously slow links.

SATCOM links tend to run at around 9600 - that's the same speed as a GSM data call, which won't help you because you'll have never made one. The latency is around 30s. HF, on the other hand, runs considerably lower, and is half-duplex to boot - it'll run down to around 9 bits/s, and with radio turnaround times at the 2 minute mark every time you need to switch direction, that's almost glacial.

Yet XMPP still works - at least as well as anything else.

On mobile, the most concerning cost is the stream startup time - because it uses a sequence of round trips - but extensions such as XEP-0198 can help reduce this substantially.

Finally, XMPP is a NAK protocol - traffic doesn't get acknowledged in the success case normally, only on error. If you examine protocols specifically designed for low-bandwidth, you'll find the same design prevails.

Compare this with any protocol that is based on JSON over HTTP - HTTP/1.1 is an amazingly verbose protocol, so the overhead of each transaction is going to be quite astonishing. HTTP/2.0 may help with this, of course.

Myth Four: XMPP is unreliable without a bunch of extensions.

The hypothesis: XMPP stanzas can be lost, therefore the system is unreliable.

The fact: XMPP typically uses reliable stream transports, so even without extensions it's pretty reliable.

There's a classic problem in computer communications called the Two generals problem. I'll wait here for you to read the Wikipedia page.

Back? OK.

On the TCP binding - the normal one people use - TCP gives XMPP a slew of features in order to minimize any failure case. If you receive a stanza over TCP, you can be totally assured that every stanza beforehand in the stream has arrived safely - mitigating the Two Generals to the point that only the last few messages sent are subject to it. This is important, and I'll come back to it. WebSockets are the same; they, too, run over a continuous TCP stream.

It's easy to take this for granted, but take a look at BOSH - the HTTP binding for XMPP - here, the transport has no such reliability to offer, so the community had to add in the same guarantees, with resend timers, buffering, and so on. In fact, any system based around HTTP polling must do something fairly intensive in order to get basic reliability, so it's no surprise that when viewed from that angle, the utter lack of visible support for reliability seems like there's something crucial missing.

So if this is the case, then the question becomes why did we need any extensions anyway? There's two cases, each addressed in a different way - though both involve "acks".

The first is that XMPP is, by default, a NAK protocol (see Myth Three). This is great for bandwidth, but it's a little disconcerting since it's nice to know people are getting your messages - their client hasn't crashed, their connection hasn't dropped, and so on. So XEP-0184 exists to fill that gap - you can ask the other client for an Ack, and when the client has received your message, it'll send one back.

But if you don't get one back, there's no way to tell if it was the message that was lost or the ack, and there's no very obvious way to handle the situation anyway. Do you send another? Or will the client get the message from offline storage? Do you care?

XEP-0198 extends TCP's reliability across multiple sessions, and also extends it through to the application. It, too, sends acks on request, but only across a single link. So a client and server can maintain a buffer of send stanzas, and if they're not acked (and they must be acked in sequence, just like TCP) then when the client reconnects, it can resume the XMPP session (and both ends can resend any missed stanzas).

This mitigates the Two Generals to the point that only the last few messages sent within a session are subject to it, and only then if resuming the session was somehow impossible.

The community developed this because TCP sessions are a lot less reliable over mobile - now a number of servers support it, and many mobile clients (particularly the Android ones) will use it if available.