Looking forward: XMPP at the core?

There are a few big problems with CIA that I've had my eye on recently:

1. The old web interface (The root page, and everything under /stats) sucks.

   I want to redo this so that the entire site is running on the new codebase with the new look, but I'd like to make some changes to the way project/author stats are displayed. You can see a prototype of the new stats interface, but there's still much work left to be done. New commits should appear in real-time as they happen. Historical commit data should automatically load as you collapse the more recent items. Each item should have contextual links in the righthand margin for related projects, authors, and version control systems. I'd also like to finish the graphing functionality that's been half-implemented for ages...

2. The bots are hard to maintain, and lack key features.

   I've been working on rewriting the CIA bot daemon itself in Erlang lately, to solve these problems.

3. The messaging hub is a closed system. Want to implement your own IRC bot or commit ticker, or syndicate the real-time data in a new way? Tough.

I've been thinking about replacing CIA's message hub with an XMPP server to solve problem #3 and help out a great deal with #1 and maybe #2.

Well, I just came across XEP-0124: Bidirectional Streams over Synchronous HTTP. Better yet, this was already implemented by Mabber in ejabberd. Hurray! This means that XMPP on its own might just provide most of the infrastructure necessary for CIA:

• Routing messages from projects to various syndication systems
• Providing an open API that anyone can use to get real-time messages
• Enabling real-time Comet-style updates
• The existing input mechanisms (XML-RPC, email, repository poller) should be able to simply deliver messages via an internal XMPP node
• The existing syndication mechanisms (IRC, Web stats) can also simply be XMPP nodes that subscribe to feeds
• Scalability across server clusters

There are still some problems that, as far as I know, will still be somewhat tough to solve even with XMPP:

• Message filtering.

  Traditionally, CIA's hub has been dealing with a single all-encompassing stream of commit messages that then get filtered using rulesets. With Jabber, each project's feed would probably become a discrete resource. If you want all the projects a particular person works on, those feeds would have to be explicitly aggregated. This has benefits and drawbacks.

• Message formatting.

  As far as I know, without a protocol extension there wouldn't be a way to directly ask the XMPP server to send out pre-formatted commit messages. Each client (including those written in Javascript, unfortunately) would need their own code to format the commit messages.

• Message storage and retrieval.

  The XMPP server would only broadcast events when new commits are delivered. There would still need to be a separate system which subscribes to all new messages, records them to disk, indexes them in various ways, then provides a service for querying that data.

I'm still quite an XMPP newbie, so I'd be very interested in hearing opinions from anyone who's more familiar with the technology.