News feeds
Publish-subscribe using Jabber
- , ,
Overview
Jabber, based on the IETF Extensible Messaging and Presence Protocol (XMPP), is a streaming XML technology. Publish-subscribe is a well-known design pattern to implement event driven information systems. This paper describes the basics of the publish-subscribe pattern, and its use in the set of Jabber protocols for efficiently distributing structured data in near real time. It uses Extended presence as an example for showing the actual message exchanges between concerned entities and describes how it can be used for publishing news.
Design Pattern
Publish-subscribe is another name for the Observer design pattern [Gamma95], also known as Callback, and is the basis of event driven information systems. In the most simple form, the pattern considers two interacting components: Subject and Observer. A subject holds a piece of data that the observer wants to keep track of. To this end, the subject publishes a change to the data when it occurs. As a result, the observer gets notified of the change. Usually, there can be more than one observer. Notification can be implemented in several ways, including message passing and function calls.
An extension of this pattern adds a third type of component: the Mediator. Where previously the subject needed to keep track of its observers, and handle notification itself, we now place the mediator in between the subject and the observers. The mediator keeps track of the subscriptions to the notifications by the observers, and sends out these notifications on behalf of the subject when it publishes to the mediator. The advantage of this extension is twofold. First, there is decoupling between the observers and the subject, lifting some responsibilities of the subject. Second, there can now be a dynamic relation: the observers can subscribe and unsubscribe at any time, without requiring interaction from the subject.
The observer pattern can be extended in several ways. In its most basic form, the notifications do not include the actual change, only that something has changed. An extension is to include the actual data, or delta, in the payload of the notification. Further, an mediator could keep track of more than one source of data. The most common way is to have different topics, that are an handle of the data to be observed. Other systems have content-based subscriptions, where a subscriber can define some query on published data and only get notifications that match the query.
Examples of the use of the observer pattern are: graphical user interfaces, mailing lists, chat rooms, presence and contact lists. Basically any system with event handlers, callbacks, signals & slots is an implementation of the observer pattern.
Pub-sub in Jabber
The basic building blocks of XMPP [RFC3920], [RFC3921], on which Jabber is based, are:
<message/>. A one time message.<iq/>. Info-query, a request-response type of interaction.<presence/>. A publish-subscribe type of interaction.
As Jabber started as a technology for Instant Messaging (IM), <presence/> is naturally used for conveying the online availability of an entity, while <message/> is typically used for transporting the text in chats. As Jabber uses an open protocol, people started to implement clients that sent along other information in these containers. For example, which song is currently playing on the desktop, was sent along with the normal presence.
Overloading presence in this way gives a few problems, though. First, this results in sending out new presence information every three minutes. Second, this gets sent out to everyone on your contact list, whether they actually want to or not. Third and last, the information was usually added to the normal status message instead of using Jabber's extensibility via XML namespaces, thereby losing semantics.
To remedy these issues, a protocol was developed for the publish-subscribe pattern in Jabber, specified in the Jabber Enhancement Proposal JEP-0060 [JEP-0060]. There, we have a publish-subscribe service that takes the role of mediator, keeping subscriptions to nodes (topics), receiving publishes and sending out notifications. Entities can create nodes, publish to nodes, or subscribe and unsubscribe to nodes.
Every node has a list of affiliations to keep track of the node's owner(s) and publishers for access control, along with a list of subscriptions. Nodes can also be persistent in that they store published items for later retrieval.
Extended Presence
<iq from='ralphm@example.com/home' id='c1'
to='pubsub.example.com' type='set'>
<pubsub xmlns='http://jabber.org/protocol/pubsub'>
<create/>
</pubsub>
</iq>
The server responds:
<iq from='pubsub.example.com' id='c1'
to='ralphm@example.com/home' type='result'>
<pubsub xmlns='http://jabber.org/protocol/pubsub'>
<create node='25e3d37dabbab9541f7523321421'/>
</pubsub>
</iq>
Now, Peter subscribes to Ralph's tunes:
<iq from='stpeter@example.org/work' id='s1'
to='pubsub.example.com' type='set'>
<pubsub xmlns='http://jabber.org/protocol/pubsub'>
<subscribe node='25e3d37dabbab9541f7523321421'
jid='stpeter@example.org/work'/>
</pubsub>
</iq>
<iq from='pubsub.example.com' id='s1'
to='stpeter@example.org/work' type='result'>
<pubsub xmlns='http://jabber.org/protocol/pubsub'>
<subscription node='25e3d37dabbab9541f7523321421'
jid='stpeter@example.org/work'
subscription='subscribed'/>
</pubsub>
</iq>
Now, I publish my tune to the node:
<iq from='ralphm@example.com/home' id='p1'
to='pubsub.example.com' type='set'>
<pubsub xmlns='http://jabber.org/protocol/pubsub'>
<publish node='25e3d37dabbab9541f7523321421'>
<item id='current'>
<tune xmlns='http://jabber.org/protocol/tune'>
<artist>Sister Sledge</artist>
<title>Lost in Music</title>
</tune>
</item>
</publish>
</pubsub>
</iq>
<iq from='ralphm@example.com/home' id='p1'
to='pubsub.example.com' type='set'>
<pubsub xmlns='http://jabber.org/protocol/pubsub'/>
</iq>
This publish action results in all subscribers receiving a notification:
<message from='pubsub.example.com'
to='stpeter@example.org/work'>
<event xmlns='http://jabber.org/protocol/pubsub#event'>
<items node='25e3d37dabbab9541f7523321421'>
<item id='current'>
<tune xmlns='http://jabber.org/protocol/tune'>
<artist>Sister Sledge</artist>
<title>Lost in Music</title>
</tune>
</item>
</items>
</event>
</message>
The advantages of publishing and receiving this way, as opposed to overloading presence, are:
Only subscribers get notifications, so no unneeded traffic to contacts that are not interested in your musical tastes.
Online availability did not change, so no need to send that information again.
Semantics of the information is preserved by using a custom namespace to contain it.
Other types of extended presence include User Mood, User Activity, Geolocation and Address. While some of these might seem frivolous, extended presence be very valuable, even in business settings. Surely, knowing where co-workers are or what they are currently doing (in a meeting, having lunch, on the plane) can be helpful when deciding how, when or if to contact that person.
Note that this should not have to be all manually provided. Geolocation information can be sent out automatically by a device that can determine its position using GPS. Activity information may be (partially) derived from the electronic agenda.
News delivery
Technologies like RSS and, more recently, Atom have traditionally used a pull-based mechanism for delivering news items to end-users, in the form of feeds transported using the HTTP protocol. A desktop application or web-based service, both referred to as feed readers, poll the news provider at a particular URI to retrieve a document (feed) containing news items. The act of polling sites for new news items is often called aggregating. For better efficiency, the HTTP protocol provides features for compression and only sending full feeds when it has actually changed.
Another way to go about news delivery is using a push mechanism where the news provider publishes a new item that will be actively delivered to interested parties. Atom is an unambigious XML language for describing news items. Jabber is all about streaming XML. Therefore the Jabber publish-subscribe protocol is an obvious candidate for being the transport to implement such a push-based news delivery system. This has been formalised in an Internet Draft [PSA05].
The idea is instead of publishing complete Atom documents, a news provider only publishes new or changed atom:entry elements to a certain publish-subscribe node. The advantages are that when nothing has changed, no traffic is being used, and interested parties are notified instantly of the news.
One of the parties that uses this technology is pubsub.com. It aggregates news from many different sources and allows users to subscribe to its system with queries on all aggregated items. As soon as a matching item has been discovered, the subscriber receives the mached item in a publish-subscribe notification. The company provides a browser plugin that shows these notifications as they are received. There are also Jabber clients that consume these notifications.
Another type of application is Mimír (http://mimir.ik.nu/), a service that is a web-based news reader that also sends out updates as Jabber chat messages. The service receives news items using publish-subscribe. Users can subscribe to channels they are interested in. Once a new item has been received by the system, it checks the online presence of the channel subscribers. For the users that are online, a text-only rendering of the news item is delivered as a Jabber chat message. The offline users have the item marked as unread. Unread items can then be read on a personalized web page, much like other web-based news readers.
Mimír allows the user to specify for which presences to receive notifications. For example, when the user sets its presence to Do Not Disturb, news items go the personlized web page.
Future ideas on enhancing news consumption includes handling enclosures and allowing news providers to include a link to a Jabber group chat room, to enable real-time discussions on news items. On the other hand, existing desktop readers can be fitted with a Jabber protocol implementation to receive notifications and display them as usual.
Other uses and future work
Of course, publish-subscribe can be used for many more application domains. Examples include sending out weather reports or emergency alerts (earthquakes, tsunamis), distributed document editing and whiteboarding, WebDav change notifications, calendaring and addressbook synchronisation. In all cases, this concerns structured data, usually already available in XML languages.
Some of the future work includes more efficient data distribution. Since the Jabber network is highly distributed in nature, this can be exploited by introducing repeater components that subscribe to publish-subscribe nodes and then publish this information again to its subscribers. Using this, a distribution graph is created that makes it even more efficient to push out data to large numbers of subscribers.
Conclusions
Many types of data have a structure that can easily be expressed, and often is already available, in XML languages. As Jabber is based on streaming XML, it appears to be a good transport for this type of data, delivered in near real time. The publish-subscribe pattern, and its implementation in Jabber, provides a good method for managing interested parties through subscription and decoupling by providing a mediating entity, while leveraging the near real time properties of Jabber.
Bibliography
[Gamma95] E. Gamma, R. Helm, R. Johnson, J. Vissides, Design patterns: elements of reusable object-oriented software, Addison Wesley, 1995.
[RFC3920] P. Saint-Andre, RFC 3920: Extensible Messaging and Presence Protocol (XMPP): Core, Internet Engineering Task Force (IETF).
[RFC3921] P. Saint-Andre, RFC 3920: Extensible Messaging and Presence Protocol (XMPP): Instant Messaging and Presence, Internet Engineering Task Force (IETF).
[JEP-0060] P. Millard, P. Saint-andre, R. Meijer, Publish-Subscribe, Jabber Software Foundation, http://www.jabber.org/jeps/jep-0060.html.
[PSA05] P. Saint-Andre, J. Hildebrand, B. Wyman, Transporting Atom Notifications over the Extensible Messaging and Presence Protocol (XMPP), Internet Engineering Task Force (IETF), http://ietfreport.isoc.org/all-ids/draft-saintandre-atompub-notify-04.txt.