Formerly known as Jabber, XMPP is the 'Extensible Messaging and Presence Protocol.' This page discusses how PSYC relates to XMPP and how psyced implements it. But you probably came here because we describe its fundamental architectural problems. Documentation on how to use Jabber with psyced and PSYC is kept at http://www.psyced.org/psyc-jabber.en.html

Siehe auch FAQ.

Contents

[edit]

1 psyced and the XMPP

The psyced implements XMPP according to its RFCs for interserver gatewaying and client access. Please note that XMPP client access isn't as stable as IRC's. Here's a list of XMPP extension protocols that we support:

  • XEP-0012: Last Activity
  • XEP-0030: Service Discovery
  • XEP-0045: Multi-User Chat
  • XEP-0054: vcard-temp
  • XEP-0077: In-Band Registration
  • XEP-0091: Legacy Delayed Delivery
  • XEP-0092: Software Version
  • XEP-0107: User Mood
  • XEP-0154: User Profile
  • XEP-0178: Best Practices for Use of SASL EXTERNAL with Certificates (up to revision 1.0)
  • XEP-0185: Dialback Key Generation and Validation
  • XEP-0190: Best Practice for Closing Idle Streams
  • XEP-0199: XMPP Ping
  • XEP-0203: Delayed Delivery
  • XEP-0212: XMPP Basic Server 2008
  • XEP-0220: Server Dialback (up to revision 0.5)
[edit]

2 Technical Problems with XMPP

[edit]

2.1 Wrong use of XML

It's not a question of religion – it's not true that one format is like another – there are clear and logical technical reasons why XML and specifically the XMPP dialect of XML are unsuitable for instant messaging applications. These deficiencies can be compensated by using more processing power and network bandwidth, but still they could be avoided (look at syntax for an example):

[edit]

2.1.1 Major disadvantage: Performance inefficiency

Many people are currently turning away from XML and turning to JSON for efficiency gains, but if you look at the libpsyc benchmarks comparing XML, JSON and PSYC you will find out that they both aren't very performant formats unless you really make use of their respective strengths. Check those benchmark results out, they're really quite enlightening. PSYC turns out several factors faster than both, whatever input you feed it.

[edit]

2.1.2 Major disadvantage: Binary Data Transfer

Just like HTTP, PSYC has the ability to say, the following n bytes are binary data - transfer them without trying to interpret them. This may sound totally unspectacular, but several protocols cannot do that. It makes File Transfers or embedding of a photograph or cryptographic key as simple as a fingersnap. XML instead has no simple solution for binary transfers. You either have to escape all XML control characters, or encode the data using something like base64 that needlessly consumes computation power and bandwidth, or you have to encode each message as a separate XML document, and put a binary capable framing protocol around it. XMPP doesn't do that, either.

It also means that it can't just deliver XML data. Ironically, any non-XML protocol is much better suited for delivery of XML than XMPP – because XMPP itself looks like XML so it can't handle anything similar to it. Just like HTTP, PSYC can simply transmit XML as is while XMPP has to encode it to avoid collision with its own syntax. The result will also be quite hard to read, should anyone need to do so. Either escaped or base64-encoded.

Alternatively, if you twist the XML data in a way that the XML document becomes itself part of the XMPP protocol, then it works. But that's not simple, not transparent, not spontaneous at all: you have to define an XMPP protocol extension with a custom namespace for that. You can't say you're delivering an XML document at the snap of a finger as you would when using HTTP.

XMPP extensions like SOAP over XMPP or ATOM over XMPP re-implement the foreign XML format into XMPP stanzas. This means that whenever you want to receive those packets, you have to parse them, then render the tree structure into a regular XML document, then pass it on to the software library which implements SOAP, ATOM (or whatever).

On the other side when you want to publish a document you received via SOAP or ATOM you have to parse it, then re-render it into XMPP. This is waste of processing power! The only way to avoid this is either to not use libraries and implement everything yourself, or to not use XMPP. Guess what PSYC does: It knows there is an XML thing in its message body. It can pass it on without even looking at it.

[edit]

2.1.3 Major problem with XMPP: Missing framing

Being able to provide the length of a packet isn't an advantage only for binary packets. It generally allows to wait for the complete packet before starting to work on it, which allows for more efficient low-level read operations. In the case of PSYC nodes providing routing services only, it is even better as parsing of the packet can stop immediately after the routing header of the packet. Everything else is just read into a buffer at once - very processing-friendly.

In XML you have to parse the complete tree before you can find out if something has reached completion, so you have to try parsing every little chunk of data coming from the network, because it might just be the last one you've been waiting for - this makes processing more expensive. XML parser implementations have gotten pretty smart at guessing the number of closed braces to figure out when it is worthwhile to start parsing, still it is an unnecessarily complicated heuristic approach. Several protocols solve this by wrapping a non-XML length-prefixed framing protocol around XML. In the case of XMPP this is however not being done:

Dave Cridland writes:

"You're correct in your assertion that framed data would mean clean binary transfers, but that isn't a goal of XMPP - anyone shipping binary data directly through XMPP is simply doing something inefficient. If you do want to ship large binary objects, it's more efficient to send them either via email, or send a URL via XMPP, and ship the data by a more suitable protocol."

And we would prefer to transfer any data across our PSYC links once we have them up and ready for use, not be limited to XML packets, especially since an HTTP file transfer cannot be multicast.

Flash and some other applications introduced null bytes into the XML stream to frame packets. This doesn't help optimizing read operations, but at least gives you a chance to postpone parsing. XMPP could adopt that.

[edit]

2.1.4 XMPP isn't even proper XML

Some say, in the light of today's processing power, it doesn't matter if XMPP uses an inefficient syntax, and yet performance optimizations were important enough for XMPP not to fully embrace the XML standard.

One advantage of XML is the existence of ready to use parsers, and by now, many of them handle the XMPP dialect pretty well. In fact XMPP developers prefer if you don't roll your own. In order to employ such an XML parser, Jabber developers need to tweak certain things before and after parsing:

See also elmex' AnyEvent-XMPP Parser comments.

[edit]

2.2 Federation vs Scalability

From a protocol and architectural point of view, Jabber/XMPP generates a lot more traffic than it necessarily should and has not performed well when users had many friends or entered huge chatrooms in the past. This may be compensated to a certain degree by processing power and network bandwidth. Unfortunately we have no solid numbers to prove this.

Here are some statistics that Matthias Wimmer, the friendly systems administrator of amessage, has provided us (Thank you!):

"I just checked the log of the three biggest domains (amessage.be, amessage.de, and amessage.info) where for the last 9,211,845 stanzas that have been delivered to these three domains (either from remote or locale JIDs) it's the following:" 
71.4 % are presence stanzas (not including subscription)
14.5 % are message stanzas
13.4 % are iq stanzas
0.6% are subscription stanzas

Then, he investigated a bit further, by looking at how many presence stanzas are actually identical copies being retransmitted over and over, and found out that:

"6200 out of 10374 presence stanzas have been dupes (59.76 %)."

That means that 42.66% of packets sent across the amessage servers were redundant. These numbers are from 2006. Would be cool if there were newer stats than these, but no-one has provided any. Maybe the situation is worse now.

Why is XMPP sending the same information again and again? Presence, like groupchat, is information that needs to be distributed to many recipients, so XMPP has to do it somehow. Currently, XMPP simply generates a copy of each message to each and every recipient and even resubmits the same information onto the same TCP connection for each recipient.

At least, when both sides of the XMPP communication support XEP 0033, the list of recipients can be delivered in an SMTP-like fashion, but that is only a slight improvement, since the Cc: lists create overhead, too.

True bandwidth benefits appear only if

  1. subscription state is not retransmitted with every communication,
  2. the burden of transmission is not going from the source to every single recipient, but using a distribution tree instead.

The latter strategy is called multicast and is what IRC and PSYC do.

Some examples:

When you have hundreds of friends on your roster and distributed on several remote servers, that's when XMPP servers start connecting each of the recipient servers and send a presence update to each person one by one.

The same is done for pubsub distribution. Let's say a news source has a million subscribers. Each time it has news, the publishing server needs to address all of these people one by one. In PSYC, by contrast, the server will contact a set of servers, which will forward the news to another set of servers each, until all the recipients receive the news. Since each of these servers knows who of their users have subscribed to this distribution context, the list of recipients does not need to be retransmitted with the message.

It's a protocol design issue that cannot easily be circumvented by server design. Some XEP drafts have been written to address the problem, but they didn't find enough acceptance in the XMPP community. If you're interested:

Although these proposals do not address true multicasting yet, they can be extended to provide for it. See also http://hancke.name/jabber/

The SIMPLE working group has published an Internet Draft on Optimizing Federated Presence with View Sharing. Their approach is similar to what we proposed for XMPP, but they have an explicit negotation of distribution context while we intended to couple that with presence subscriptions. See presence for more details.

Ironically, even if Jabber was using IRC as a kind of multicast overlay network to distribute its presence, it would probably perform better in the whole. fippo wrote up a little proposal: Jabber Relay Chat. IRC has its weaknesses, but some things can be learnt from it.

XMPP is generally claimed to be scalable, but its scalability is best achieved within a single domain, in a closed cloud silo, that uses its own decentralization strategies, like the Erlang language provides, not trying to use S2S XMPP internally. The open federation network of XMPP instead would benefit greatly from more elaborate routing strategies like stanza repeaters and multicasting.

By not providing those, XMPP cements the likelihood that people will just stick to large popular servers as XMPP works best when your contacts are on the same server as you.

Still, a distribution network based on established TCP lines should generally be superior to similar efforts using HTTP POST requests (see also XMPP To The Rescue), so if you are still doing things using HTTP, you may of course experience an improvement switching to XMPP. Then again, in the light of REST-ful APIs, web servers have improved a lot on handling small notifications in recent years. HTTP/2.0 is probably going to be several times more efficient than XMPP. Especially if you deploy a multicast solution over HTTP like FETHR has tentatively done.

When asked about multicast, Ralph Meijer, member of the XSF Council, stated in a conversation at the FSW2011 that you cannot entrust subscriber nodes to correctly forward your message to further subscribers. They might censor or fake messages. FETHR approaches this problem by applying digital signatures and using hash chains. It's not such an attractive option for XMPP however, since all this cryptographic data cannot be transmitted in native binary form – inefficient base64 encoding is needed instead, but it would probably still be better than giving up on efficient distribution altogether.

Interestingly however, since 2009 Ralph Meijer's XEP-0253 exists, which provides for multicast distribution at least for pubsub. In combination with XEP-0163: Personal Eventing Protocol this could actually turn into a scalable replacement for traditional <presence/>.

[edit]

2.2.1 What if PSYC and XMPP were to team up?

It is being said((r)) many large XMPP deployments like Google Talk, Wave or Facebook Chat use custom tree-based distribution protocols within the cluster, while only interfacing to outer XMPP federation using the XMPP protocols.

At least when Google Talk started, its first experiments weren't using XMPP. Also Facebook Chat's first incarnation was no XMPP. Both implementations are custom and didn't start out from an XMPP codebase. Since it is in both companies' interest not to tell that they might be using more efficient inter-server communications, we will probably not be told. In any case their existence is a proof, that you can do XMPP in a scalable way if you solve internal communications efficiently somehow. Even ejabberd uses its own XML-free internal protocol between processes, the one provided by Erlang, so it can't be a bad idea.

It would be good for the federation, decentralized communications and open source in general, if such strengths weren't limited to cloud-based commercial offerings (here's an example for that – don't try this without a cloud at hand!), but rather generally available and evenly integrated into the XMPP experience. The ejabberd solution is freely available, but it only works in homogenous network clusters.

In PSYC as Jabber S2S we have laid out a plan on how to use PSYC as an interserver federation protocol for XMPP applications. This would combine the respective strengths of the two technologies. psyced already implements several parts of that, but it sure would be fruitful to extend solid XMPP server technologies to also process PSYC as a more effective network serialisation format and distribution system for the same content (see below for benchmarks comparing PSYC and XMPP syntax).

We are however not putting further effort into this, as we have come to the conclusion that federation is the wrong architecture for private communications. We still do multicast, but we no longer expect our users to trust a server (see secushare).

[edit]

2.3 Presence

Presence itself can be a problem for the protocol that was specifically designed to solve it. Examples:

There has been a scalability analysis shoot-out between SIMPLE and XMPP developers, we documented and commented that on the Presence page – including the PSYC way.

[edit]

2.4 Having to Guess the Meaning of a Packet

The PSYC protocol provides you with a method (or message code, message type) for every protocol message, which immediately makes it clear what the meaning (semantics) of the packet is, whereas an XMPP server, even psyced, has to closely look at what a client or server sends to figure out what it wants. This means parsing the complete XML packet, no matter who it is intended for, then inspecting a lot of individual aspects of each packet. For example, 

 <presence from='foo' to='bar'/>

is what we call a _notice_friend_present, while the XMPP construct 

 <presence from='foo' to='bar'>
    <x xmlns='jabber:x:delay stamp='20051015T20:07:42' from='foo'/>
 </presence>

is a _status_friend_present in PSYC terminology. While in PSYC we can quickly glance at the method name, then decide what other parts of the packet we need, in XMPP we have to look at all the attributes and children to figure out what semantic meaning the packet has, before we can start acting upon it.

The lack of a method name as a unified speedway to packet interpretation makes programming XMPP applications harder than it needs to be. Oh, since XMPP developers reading this paragraph mentioned that it's bad if a client needs to know all methods – no, just because this page is criticizing XMPP doesn't mean you get a complete introduction to PSYC. Clients make use of method inheritance and only need to understand three or four basic method families to already provide for all basic functionality, so PSYC achieves the same with a lot less code complexity.

[edit]

2.5 MUC the "Multi-User Conference"

MUCs are organized in the most natural way you would expect a chatroom to be organized: It is running somewhere on a server, all users register with it, all communication flows through the server of the chatroom. You could say XMPP MUC implements the most straightforward way to implement a chatroom, similarely to most webchats.

In situations of either highly distributed topology or large audiences, this approach has the same scalability issues we have seen above with presence and pubsub. Other technologies have more elaborate strategies.

IRC hosts its channels on each and every server of its network, multicasts the messages to each other in a close to optimal way, then each server takes care of letting its users take part. IRC has hosted moderated chatrooms with thousands of participants in the past. The drawback is, that no single authority is in charge of conference control of an IRC channel, so a single rogue server can cause havoc.

PSYC uses a combination of both approaches, it has a central authority organizing a chatroom, yet uses multicast distribution to even out the load on the network. As early as 2001 PSYC events have been run hosting ten thousands of users in a single conference. Not all of them were permitted to speak freely, but all of them were able to write up submissions for the editorial team to choose from, in realtime.

Some MUC implementations use custom overlay networks to achieve a similar result as IRC channels. The drawback is, it is not a generic solution all XMPP servers can use to distribute the load - it only works for a set of servers run under the same administration (cloud-friendly thinking again).

It would be nice to see these kind of improvements in a generic way, so that all servers in the XMPP federation can take part (see the Smart Chat proposal above for example). This would address those frequently discussed scalability issues of MUC. And even better, if such a multicast improvement was generic enough to also encompass presence and pubsub. But, as we found out, this is not going to happen.

[edit]

2.5.1 Distributed MUC

Since the MUC strategy turns out to have several drawbacks and we found that this distributed MUC proposal (it should probably be called decentralized MUC as it isn't really distributed) submitted to council has several fundamental shortcomings we submitted our own version, which is mainly a straightforward mapping of the PSYC group join to XMPP. This was accepted by the council for publication on 2010-03-29 (see minutes).

Feedback was meagre so we decided not to update the specification. Some feedback given on obscure mailing lists and slides here and here.

In 2012 a XEP-0289: Federated MUC for Constrained Environments has been published which tries to handle netsplit situations similarely to IRC, even including so-called private message routing.

Are we afraid of trust issues in a distributed scenario? If a server X is trusted to bring in new users and servers it could just generate fake user and fake server names. The PSYC approach to this depends on the use case. If the use case is an anonymous broadcast, then PSYC doesn't want to know any names of users and servers behind your node. If the use case instead is such, that each node must be authenticated and authorized, then each node must make its admission request to a master node before getting assigned to a node in the tree. You may also have multiple masters, as in the case of "FMUC," but that is a mere question of conference control and no reason to provide less distribution capability.

So with all of these developments there is no convincing reason why XMPP shouldn't provide scalable distributed MUCs by default and add any authentication paranoia later. Of course this may harm some business models of some companies.

[edit]

2.5.2 /msg doesn't care for OTR

A minor privacy concern with MUCs is the /msg command which sends private messages through the MUC, not directly to the recipient. This is frequently not obvious to the people using this function. The MUC could therefore listen into private conversations if suitably modified. Also, if people usually do end-to-end encryption, they may fail to notice that while an IRC client would also encrypt /msg, a Jabber client won't, thus transmitting over the network in the clear.

[edit]

2.5.3 Potential identity confusion

Another minor concern is about identity. XMPP resources are case sensitive. Due to that, there can be 3 persons in a MUC with the identities foo@bar/AbC, foo@bar/aBC and foo@bar/ABc. This could cause confusion, as they may appear as the same person to other participants.

[edit]

2.5.4 Supporting MUCs from PSYC

PSYC users can either use a graphical interface, a Jabber client or a command like /c xmpp:someMUC@conference.example.org to take part in a MUC happening on an XMPP server.

PSYC will however not be able to apply multicast optimizations retroactively: If several PSYC users join the same MUC, each of them will receive its own copy of each message. For a cleaner set-up it's good to use PSYC chatrooms, and invite XMPP users in. In this constellation, all PSYC and IRC users will benefit from interserver multicast routing. Only XMPP users will be handed out redundant copies of messages. This is how it works for Jabber users:

[edit]

2.5.5 XMPP users entering PSYC places

If you want to enter psyc://example.org/@someplace, you need to join an XMPP JID like *someplace@example.org. The best way is to query the official addresses of the room by issuing the /status command in it.

This applies to both people using psyced as a Jabber server directly and regular Jabber users in the XMPP federation network, given the PSYC server runs interserver XMPP, too. As explained before, entering via XMPP defeats multicast routing and similar PSYC functions, but provides you with the extras of a native XMPP server in exchange.

[edit]

2.6 Profiles in Jabber

Jabber comes with two different profile formats. One (XEP 0154 aka User Profile), which is very elaborate, but rarely implemented and another one that is very limited, but available everywhere (XEP 0054 aka vcard-temp).

The latter is a mapping from the vCard format defined in RFC 2426 into XML. The way it keeps phone numbers is a bit impractical: 

 <TEL><VOICE/><HOME/><NUMBER>303-555-1212</NUMBER></TEL>

VOICE, HOME and NUMBER are adjacent children of TEL, defeating XPath-like XML access strategies. Even parsing this XML with a line mode parser doesn't work as the DTD doesn't enforce this order of children. HOME may come before VOICE and VOICE after NUMBER.

You can read about the PSYC way to encode this sort of data on the Profile page – it, of course, supports keyword inheritance, so you can extend any field with new semantics as you require.

[edit]

2.7 Reliability

[edit]

2.7.1 Closing Idle Connections

In 2006, four implementations of XMPP were closing connections in an unsafe manner, potentially losing messages that the other side may have started to send while this side is closing the TCP connection. XEP 0190 has been written to address the problem. The new behaviour has also been worked into the current XMPP RFCs. The issue is hopefully solved by now, although we haven't made another survey.

[edit]

2.8 Minor protocol optimizations

  • XMPP uses uptime as a technical value (JEP-0012). That's impractical, as the value gets old while it travels across the network. Using boottime is much better: You can keep it for as long as you know the other side is up (typically while the TCP line is up) and calculate the uptime from it by a simple time() - boottime subtraction. Needless to say, PSYC uses boottime. For end user messages it provides both values.
[edit]

2.9 JIDs aren't flexible enough

Because the world is round and not flat, I mean because there are other chat systems out there beyond Jabber, the user@host syntax, also known as Jabber Identity, is bound to create problems.

It could have all been so easy if the JID was simply an opaque string, where only the user's server needed to understand what's in it. Why does a client need to know what an icq:1234 is, or an INFITD0E at DM0TUI1S - as long as both the user and his server know what it means, that should be fine. Okay, forget the second example (which is a BITNET address, by the way). If JIDs in XMPP client-server were opaque, then we could have asked our users to drop the uniform into the add buddy wizard's address field, and poof connection is made. Instead people often have seperate username and hostname input fields, and the majority of chatsystems out there aren't compatible with it.

Therefore Jabber has to mingle all kinds of gateway addresses into the user@host syntax. This may work for icq:NUMBER as you can turn it around into NUMBER@icq, and that's kind of logical to the average user too. But once you have an address like msn:user@example.net, how are you going to go about it? 'msn:user' is not a legal user name in Jabber, and even if it was, it is just nothing the end user would guess - to put the protocol scheme into the username field. Oh, XMPP uses UUCP syntax for that: user%example.net@msn.transport.example.org.

All of this is just theory anyway, as according to Jabber philosophy your home server is not supposed to handle any gateways itself (psyced is the exception, as usual). You get friends with a transport service instead, then you encode all your friends as being at home in that transport - This means you have to shift all your friends to new addresses if that transport service goes out of business or you start doubting its disinterest in your private affairs. These are issues that are being worked upon, however. We haven't heard of any improvements in this area in the past >5 years.

The fact, that the protocol scheme is not supposed to be transmitted in an XMPP JID has a strength and a weakness:

  • CON: Jabber servers have no easy way to figure out if XMPP is the only possible protocol to reach a certain person.
  • PRO: All Jabber clients and servers have a scheme-free form of address of all people in their data structures. Thus one could replace the XMPP protocol at the backbone of Jabber and use PSYC (or whatever else) instead. So, if a sufficient number of Jabber installations also provided PSYC protocol, the network would slowly and transparently shift to a possibly more efficient interserver protocol. This idea is discussed in 'PSYC as Jabber S2S'. But this is only applicable to a structurally similar protocol such as PSYC in federation mode. So we did:

Users of Jabber clients connected to a psyced can communicate using user@host JIDs as usual, yet psyced will figure out which protocol is best to use to reach each recipient.

[edit]

2.10 The Odyssey of Dialback

Concerning dialback, one of the reasons why it was preferred over an IP address check is jabberd was not able to bind() the socket to a particular IP at the time. And why does it require two TCP connections? Because in Java it is nicer to have a reader and a writer thread, and having them use the same socket is impractical (as mentioned here). At times there were even more than two TCP links between two XMPP servers, because of multiple hostnames. PSYC uses just a single so-called circuit in all of these circumstances. There have been many attempts to deal with this and close the book on dialback, but it kept coming back as the only properly functioning method. Fifteen years later it looks like XMPP has finally caught up and solved this.

[edit]

3 Other Techie Details

[edit]

3.1 Full JID as a stumbling block

XMPP distinguishes between a 'bare JID' which points to your identity as in joe@example.org and a 'full JID' such as joe@example.org/coolChat. The 'resource' appended at the end distinguishes the client connection which is supposed to receive certain messages from your identity and potential other clients of yours.

It all kind of fits together somehow, although it is a bit strange, that a remote chatroom (a MUC) should know which client you are chatting on, rather having your own server know about that. It would be interesting, if your client was entering the MUC directly using its own IP address (using something like a UNL in PSYC-speak) – that would allow for high bandwidth applications if the data no longer needs to flow throw your server.

It would also be practical if any application or chatroom could send you things without having to keep track of the client apps you are currently using, just by sending to your 'bare JID.' Unfortunately, several servers will reject such a 'stateless' mode of operation.

All in all it's not dramatic, but a bit strange how XMPP introduces an in-between concept, that combines the disadvantages of going through a server and not having a person's full peer address with the disadvantages of having to keep track what resource a user is currently using networkwide. If you think it is good to hide your peer address for privacy or security (which is a question of trust architecture really, but in many cases that's a valid point), then you shouldn't have to let everyone keep track of your current client resource. The server should figure out where packets need to go. It knows which chatrooms a client entered, after all.

But that breaks with XMPP philosophy. XMPP servers are intentionally agnostic and don't look at packets they deliver on behalf of clients. So, end of story, you can't always send messages to the 'bare JID.' Sometimes they will be considered illegal and return an error. It's up to you to consider this an achievement or an unnecessary complication of things.

[edit]

3.2 Vocabulary

Broadcast stands for a single transmission operation to reach all recipients on a medium in a single go - like on a radio. Once you transmit, everyone can listen - there is no extra effort involved in getting the content to each recipient. In the XMPP world the word however actually means a quite inefficient fan-out of copies of a message to each member of a collection of recipients, which, in terms of bandwidth consumption, results in pretty much the opposite of what is expected from a broadcast.

Also the word multicast is used in a confusing way to actually mean what the XCAST RFC describes as multi-unicast on page 14. The naming of XEP 0033 itself has been corrected, but the contents still frequently mentions multicast with an incorrect semantic connotation while XEP-0253 implements multicast but makes no mention of it. It uses the word chaining instead. Twenty years of earlier research work do not matter, the world deserves new vocabulary.

[edit]

3.3 Inter-server (S2S) encryption

Quite often Jabber is being advertised as being more secure because of its interserver encryption using TLS. Some even propose to remove dialback from the XMPP standard, because Jabber is secure now. fippo's 2007 survey however shows that only 4% of publicly accessible XMPP servers actually provide proper valid certification, everything else is either untrustworthy or unencrypted.

With all the recent efforts to improve that, we expect the number to have risen consistently – still the conversation data is unencrypted within server memory, and your talking partner's server might be participating in the PRISM program, so you better employ some OTR. PSYC in the meantime has given up on federation and TLS along with it and is now moving on to more secure strategies based on public-key addressing.

[edit]

3.4 How to protect who is talking to whom?

From a privacy standpoint the fact that, even if you are employing OTR for end-to-end encryption, servers always see who is talking to whom, is a serious problem – especially in the light of the PRISM program and the largest XMPP deployments on earth being affected by that.

PSYC can deliver encrypted PSYC packets in an onion routing style in its native binary form, creating not much overhead besides the fact of doing onion routing. The secushare project is working on this. Doing so in XMPP however would mean to shuffle around big blobs of base64-encoded text. It can be done, but it sure is ugly. First of all you need to get your friends off of the PRISM servers.

[edit]

4 Other Opinions

[edit]

4.1 Lessons learned with OneSocialWeb

From the FSW2011 paper presentation by Diana Cheng, Vodafone Group R&D and Dan Appelquist, W3C.

  • not all developers like XML
  • developers prefer easy to use APIs
  • deployment of XMPP instances is not as easy as standard web infrastructure
  • many find it too geeky and won't be bothered
  • hosting XMPP is more expensive
  • problems with BOSH: webchat clients frequently breaking. should try web sockets instead?
  • we need better javascript libraries for XMPP
  • pointless having a big standard noone wants to implement. have a working implementation first, have the standard come later.

Unfortunately these points have not been published in the paper document but rather presented in voice at the FSW2011 (see the video for the W3C Codebase session on June 4th). They're in the OneSocialWeb-for-FSW slides however!

[edit]

4.2 Is XMPP likely to be superseded sooner or later?

... at least according to Adam Nemeth. His perceived weaknesses of XMPP are:

  • Jingle File Transfer being so complicated to implement, that XMPP effectively goes without file transfer feature.
  • XML being increasingly replaced by JSON in real-world applications.
  • BOSH being inefficient.

<lynX> 2014 adds HTTP/2.0 to the menu of efficient XMPP replacements...

[edit]

5 Questions and Answers

Für Deutschsprachige: FAQ und weitere Gedanken auf de:Jabber.

[edit]

5.1 Why don't you help XMPP getting better instead of spending so much time documenting its weaknesses?

By writing a Jabber server and gateway as a part of psyced we are just documenting the problems we keep encountering. This page is a by-product of that. At the same time we are helping Jabber. Our first XEP (XMPP Extension Protocol, formerly known as Jabber Enhancement Proposal aka JEP) on Dialback Key Generation and Validation has been accepted as number 0185. Also XEP 0190 on Best Practice for Closing Idle Streams has been adopted.

See above for our Smart Unicast XEPs, which have been rejected in a council meeting on 2006-06-14. The hard data had been given to them before. See in the Federation vs Scalability paragraph above. Also the test framework existed at the time. psyced implements this experimental extension (just #define XMPPERIMENTAL).

Here we are in 2013, the historic chance to fix XMPP has been missed – by now there are devices implementing design mistakes in hardware – new technology will be so radically different, that it will slowly lure people away from XMPP's cloud architecture into the darknets.

[edit]

5.2 So you are saying that XMPP is bad?

<fippo> No, otherwise we would not support it. We think it's an ugly protocol and many of the design decision are wrong, but since it enables us to communicate with more people than without it, we deal with it. After all, you do, too.

We did however experience situations that suggest that the XMPP Extension Protocols aren't always developed in an open fashion, contrary to the statement that the XSF be a "completely open and transparent standards development organization." See for example how the Jingle protocol was developed). Nor that the "developer-friendly standards process [...] places a premium on the values that built the Internet in the first place: rough consensus and running code." See this thread on standards-jig for that.

<lynX> Actually yes, I would say XMPP is bad as it has impeded better solutions from finding momentum and public interest. Everyone is so keen on having a standard, it doesn't matter as much how many bugs it has. Doing chat with XML as a wire syntax sounded like a grand idea in the year of the big XML hype, but it puts up a hard barrier on how far XMPP can go.

[edit]

5.3 Why do messages to XMPP users take so long to echo on the screen sometimes?

First of all, feedback that a message has successfully reached its destination, so-called echoes, is a PSYC feature. The Jabber gateway generates an echo as soon as it successfully delivered a message to the recipient Jabber server, which isn't exactly the same thing, but an acceptable approximation.

Jabber echoes should therefore actually appear quicker than PSYC echoes. They sometimes however don't, because most Jabber servers shutdown inactive network links after a short period. They are considered a cause for load rather than a chance for quick delivery - which is an architectural problem. By consequence after that short period a new connection has to be established between the servers. Using the XMPP protocol this means opening two TCP connections involving two TCP handshake processes plus the XMPP dialback handshakes. This totals in a typical delay of several seconds. It takes less if authentication based on TLS certificates is used instead of so-called dial-back.

This still is no cause for concern. Only when no echoes appear at all, you should presume that your message has not successfully reached the destination, as is the case with PSYC messages, too. Normally you will receive a detailed error notice as soon as your server gives up on delivery of your message, but this may take a while as psyced keeps a queue of outgoing messages and tries to recover the communication link a couple of times before returning the queue to its respective senders.

[edit]

5.4 By reading your lines I get the impression these Jabber people never get anything right

No no that is certainly not the case. The competence is there, in abundance, but for some reason it doesn't always end in the technology itself. Maybe it is simply because the technology started on several wrong feet, and you just can't keep tradition and compatibility at the same time with radical repairs. Also the new way of doing things has grown roots, and all the XML structures and acknowledgement stanzas may look and feel beautiful to those who like them. After so many years using the technology this way, tradition prevales over the disadvantage count. Only an by-stander still has the capacity to see all the problems in their full ugliness.

On the other hand, so many aspects of technology are falling apart as we look at them, especially in the light of 2013's revelations. Some folks broke the Internet. The task at hand is to fix it, rewrite a GNU Internet stack, and XMPP is among the technologies with a completely wrong architecture to be helpful. We have to restart from scratch anyhow.

[edit]

6 Links

[edit]

7 See also