The Pushbutton Web: Realtime Becomes Real

The realtime conversation is fun because it's living and breathing. Immediacy has become such a strong news value because of the rapid pace of the web.

Very good article, I will be looking forward to reading the commentary... in real time.

I don't mean to crassly TL;DR you, but what does this mean to the lay person?

What's in it for users?

Great post Anil, thanks for highlighting this growing technology trend. We've got 10+ years of failed efforts at event notification over the web, but the technology and use cases keep developing.

Here's what really interests me in all this.. Notifications between services imply applications that are a mesh of services. The traditional Web business is a single server talking to a bunch of clients. Typepad is a blog hosting service, and it has a lot of clients who write blogs (and more who read them). One to many. But all this web service / JSON stuff is enabling federated services. And event communication / notifications / PushButton is a natural extension of that federation. So sites like TypePad are no longer a service for users, but rather one piece of an ecosystem that also involves Facebook, Twitter, Google, Flickr, etc. All these services talking to each other.

That seems new to me, a new way to think about product and business development. We engineers have been able to cobble together event notification for a long time, but we seldom do it because it's a hassle and because we haven't had compelling product reasons to do it. Maybe that's changing now?

in terms of prior art and patents, my patent on this technology is dated 2001

http://tinyurl.com/647huw

Great read. I'm currently working on a project that would definitely work best if it were as real time as possible. Unfortunately, I'm just prototyping it and they still haven't found a developer. Hard to design without realistic constraints. I'm looking forward to reading your next installments on this for the sake of learning both technical and practical insight on this growing trend. Thanks.

So would this be like a Peer-to-Peer messaging thing that pushes the content out?

Anil, the fact that you don't have a use case for this should be a much bigger concern than any worries about standards or competition. And I say that as someone who spent a few years working at a company that tried to make this technology stack happen. Back then we really thought that if everybody signed up to an architecture based on messages and hubs rather than HTTP polling that it would enable a new class of applications. The problem is that nobody can quite describe what those applications are. And it's not a matter of "if you build it, they will come," since this technology has been around in various forms for 10 years now. If there were a class of applications to be built using this stack, it would have emerged by now, don't you think?

Actually, I think the Google Wave protocol will go a long way toward enabling these types of interactions. For what it's worth wave is over XMPP.

For past 2 years I talked about Mellow Clutter at design conferences and that was an analogy for "a bookcase of curios that you'd collect and share" online. In other words, a topic-based portal with community. Now the technologies are here to build it and cool. The hub on Bike Hugger is an example and we're ajaxifying that next. The Mellow Clutters are most interesting for product blogs -- here's everything online about X and let's talk about in use by Y. The pushbutton connects this all together in realtime.

Good stuff.

Anil,

That is a wonderful article.
- I disagree with the newspaper analogy. Right now, you don't have anyone that comes and tell you that there is a new edition. You, as a user have to got check if there is one and get it if it's appropriate.
- I think you're right in the threats that you expose. However, the biggest threat that I see is that publishers will not implement these push technologies. Of coure, the Twitters, the FriendFeeds (who are under extreme load due to that never ending polling) or the Anil Dashes (who just love implementing new cool stuff) will implement that, but what about the rest of the web, the small corporate blogs, the non-geek webservices? I think GNIP's experience here shows us that not everybody is encline to pushing its content as long as there are not enough subscribers. Mostly, they will probably not push to "everybody", but rather rely on middleware (hubs). Then, the question of choosing a hub is raised : you don't want to change every 2 weeks. the "Demo Hub" is an option, but several people don't want to rely too much on that...

I think that is why Superfeedr (http://superfeedr.com) -shameless self promotion- has a great role to play here. As I like to say, we poll the web so that nobody else has to do it (since we push the content when we find it). In the end, we expect that the services that we poll the most will slowly start to push toward us, but they will never represent more than a few percent of the whole web.

What do you think?

Anil, thank you for the big dose of clarity and high-level user-friendliness.

What about http://brdfdr.com/? "Prototype implementation of the FETHR distributed microblogging/publishing platform." What's its possible place in the pushbutton landscape?

how long before this will be available via a MovableType blog?

Anil:

Nice overview. Been using webhooks & pubsub to wire legacy systems (successfully). Companies want to leverage online apps, but getting data to and from legacy systems (CRM, Accounting, firewalled apps) is a major hurdle. Webhooks, pubsub and smart connectors make wiring easy(er).

Also, I'm certain Pushbutton could deliver on Kevin Kelly's Dream Machine.
http://kk.org/ct2/2008/01/my-dream-machine.php

Interest in event-driven, non-polling distribution has certainly picked up. I've been working on a very lightweight protocol called SLAP for announcing changes in feed contents. See this blog post.

Great piece Anil. Hopefully this will increase awareness of the fact that the Internet is changing (in a big way) and those who look to it to make a living need to be aware of those changes.

but what about the rest of the web, the small corporate blogs, the non-geek webservices?

I can see free services popping up to provide this. The interface for adding that support in your publishing platform could be as easy as adding XML-RPC update ping URLs is in WordPress or Movable Type is today.

great post! you capture the main points around "real-time" web interaction really well. being an HTTP head myself... leveraging the already prolific protocol for this stuff only makes sense (as you point out).

I don't think we need a new name for this though, as we're already littered w/ them; gnip, pubsubhubbub, webhooks, sup, etc. IMO, what you're capturing in the content primarily is pubsubhubub (distributed "push"/notification).

while the industry adopts more distributed models, Gnip's (my company; http://gnip.com) taking a more centralized approach at the moment. while we'll support pubsubhubub soon, we've found a need/market for a more collective "push" model. that doesn't by any means obviate the need for the broader, "standard"/"open", framework for "push"/"real-time", we're trying to build a business around the idea. when standards grow up around us, we'll adopt them to ensure maximum compatibility/satisfaction.

that said, Gnip has always been wide open to community feedback/suggestions/needs, and we still are. http://groups.google.com/group/gnip-community

Anil,

This sounds fantastic to the responsible among us, who just want to build cool applications and perhaps businesses based on those apps. But if that's the case, it must also sound fantastic to spammers, who are forever seeking new and faster ways to bombard users with their payload. How spammable is this architecture? Are the folks that are hacking this set of problems talking about how to prevent an inexpensive, decentralized system accessible at low cost, that will attract developers and businesses without the wherewithal to establish a central spam-prevention authority like Google (and increasingly Twitter) from getting overwhelmed by bad actors attracted by the very same low barriers to entry?

I didn't see that in your list of worries, so curious to hear your thoughts.

XMPP (Jabber), NNTP (Usenet), IRC

NNTP is 30 years old and IRC 20, and neither have fundamentally changed since the mid-90s, while XMPP is only 10 years old, has been under constant active development since then, and as you point out has active supporters like Wave, Google Talk, etc. XMPP certainly isn't perfect by any stretch, but I'm surprised to see you paint with such a wide, inaccurate brush.

I think PubSubHubBub and rssCloud differ in one approach (at least in the testbed instances).

PubSubHub acts as you describe by accepting "posting" events and letting subscribers to the "hub" know about the event.

The RssCloud also accepts posting events and lets subscribers know about the new post.

Where they differ in implementation is this:

PubHubSub also accepts the "posting" text (message) and can delver it to the subscribers as you show graphically.

The RssCloud just relays the event of the "posting" but the user would then go to the authortative source of the post text... the author's "rss feed".

It appears that rssClouds could then have less stress to implement since they won't have to accept and archive millions of text messages. Only the events are achived.

PubHubSub will thus create a massive archive of posted text and all that that implies... searchable, indexable, monetizeable text. Something you would expect (and maybe trust) Google to build for us all.

An rssCloud will just have events and archiving functions are pushed out to the content creators.

Of course, an rssCloud could easily be designed and implemented to archive and forward all posts as well... that's just an implementation detail and the events elements would simply tell the readers where to go to get a post (to the Cloud) or to the content creator's feed.

There's should be a "land grab" for start-ups to create creation and consumption tools that work with their Clouds/Hubs or with "standardized" Clouds/Hubs.

Let the games begin. I see settlers racing across the great plains again for a nice parcel... A PushButton Bubble?
One can only hope. These bubbles feed a lot of people before they pop.

Hello, I though this was an interesting article considering the project I am working on called CarrotSync which is very close to what you are talking about and has allowed me to build for example, a remote control Youtube video player as can be seen in this video

http://www.youtube.com/watch?v=wpnK3UoYd7g

You can check out some of the tests and results at http://www.carrotsync.com

Anil,

great article, I really enjoyed reading it. You discuss the idea of decentralizing, but the model you describe sounds awfully centralized to me. While senders and receivers don't have to rely on any one service, such as Twitter or Facebook, they do still have to rely on *a* service. This is in essence still highly centralized.

Sure, with OSS being so prevalent now, you can run your own blog site or any other set of services you choose. However, cloud computing is pushing us away from the idea of doing things yourself and relying on service providers to do it for you. (You can run your own mail server too, but do you?) This is precisely why Blogspot, Google Mail and countless other service providers are so popular.

As with any new technology, an ecosystem of service providers will come into existence. In that life cycle, there will likely be a period where many flourish, followed by many failures (no monetization model, poor execution, etc), then finally an equilibrium. When that equilibrium period occurs, we'll be left with a few key players owning this service market (think Gmail, Hotmail, Yahoo mail, etc). Is that decentralized? No, but it is competitive.

Fred

Well said and well thought out. The future of Internet usage is definitely not stateful http client-server requests, nor is it next-gen messaging queues. The always-on experience will definitely demand smart intermediaries . Envisioning how how soon that could be built out was a very satisfying read. Thx.

As a developer, the most interesting part about all of this is the potential for inter application communication. What's your thoughts on using this as a distributed event system?

As someone who remembers the heyday of Usenet, before the World Wide Web and subsequently the blogosphere and then social networks took centre stage, I had to smile when you mentioned NNTP, the technology that underpinned that earlier conversation network (now pretty much useless, alas, except for the archives over at Google Groups).

I'm hoping that enough time has passed since the debacle that was 'push technology' to allow this new PushButton concept to gain some traction and not so much skepticism. And yes, I remember that issue of Wired Magazine, hehe.

Anil, great post. I invite you to check out Adobe Wave at http://wave.adobe.com. We're focusing on the user experience piece. Would love to get your feedback.

If I understand this correctly, you are attempting to coin the term �pushbutton� (with capital), presumably so you can be viewed as a pioneer of the technology if it takes off like Ajax?

The difference here is that JJG didn�t go out of his way to make history. You�re coming off a tad inauthentic.

Hey Anil,

Good post.

I think the major obstacle here that you omit is the issue with message verification.

Delivery of "fat pings" has been discussed in the past but the issue of verification/validation that the content is actually from the author has complicated issues.

For example, I could forge a message as "Anil Dash" and say something embarrassing and post it into the cloud.

This has been avoided in the past by all existing systems by just avoiding the issue.

In the Six Apart Update Stream (we use most of these systems at Spinn3r) you can be assured that the content comes from the author since you're talking to the source directly.

Same thing with the Twitter stream, etc.

Pings avoided this by requiring the receiver to fetch the content directly from the source.

One solution here is to just use public/private key auth and sign all messages. This is similar to the way SSH exchanges keys between hosts. If a user were to upload their public key to Tweetdeck/Seesmic and host it on Twitter.com then all that is needed is to fetch the key once to verify all future messages.

One could use OpenID or authenticate messages against Twitter directly (by fetching them) but this defeats the whole point as this would allow the source to block/prevent access.

I'm very concerned by various industry efforts to block/censor content which they find objectionable.

One of the major reasons the web is such an amazing places is that for the most part you don't need permission to participate.

There goes our batteries!

I hope my iPhone is not a model for things to come with regards to push and power. I had to turn off most push features on the Apple in order to get some semblance of reasonable battery life.

What other ramifications to push? No more sleeping computers? P2P folks never sleep :)

I've seen estimates that cloud power requirements will quadruple over next 10 years (data centers currently estimated to utilize 1.5% of US electricity).

I wonder if this will drive the demand even higher.

Love the use of the old Wired "push" cover [never wrong, just early!]; also, that's nifty, the way the hashtag #pshb nicely represents both PubSubHubBub and Pushbutton.

It's great to see a vision for a more responsive Web, built on a more pervasive and sensitive "nervous system" (sorry Twitter -that's your line) laid out clearly like this.

burtonator: I think your security worries are unnecessary. There are four things to understand about authentication in PubSubHubbub.

1. The Hub will always go back to the original source feed to pull new content; this fetch can be over HTTPS, which authenticates the publisher.

2. The feed declares its hub (or hubs) authoritatively with an atom:link element. This proves to subscribers (possibly over HTTPS) the delegation of trust from the publisher to the hub.

3. Subscriber URLs are obfuscated URLs; when combined with HTTPS, these URLs act as passwords that provide a secure channel between hubs and subscribers. This prevents injecting fake feed data.

4. For more security, publishers may add XML digital signatures to their own feeds, giving them yet another level of provable authenticity.

bslatkin wrote: "The Hub will always go back to the original source feed..."
This may be true of hubs that *you* write, however, you can't guarantee that all hubs will be written that way. Also, given that entries from multiple feeds can be aggregated into a single feed, it would be necessary for the Hub to validate the atom.source for every entry found in a feed. I think it is unlikely that this level of validation is going to happen consistently.

Of course, for most applications, this uncertain authorship doesn't matter. However, we're likely to see applications that wish to use this pattern with more sensitive data. (For instance, an offer-to-sell or an offer-to-buy) This is, of course, why we ensured that Atom defined standard mechanisms for doing the kind of signing that would address burtonator's concerns.

bob wyman

insightful , thanks

Old wine, but the tubes are new, nice and shiny. Most of today's "wizz kiddies" or maybe we'll soon call them "push kiddies" may not even have been born then, but after IRC and early WWW chat systems and Pre-Internet providers like AOL, CompuServe or Minitel had this in a similar way. Usually you would only listen to a chat session, most other ways of listening, observing or "push" were not as far established.

Great stuff. Finally news I can use. Thanks for the primer.

a few years ago there was an idea in the air to try to rebuild email messaging on top of the architecture you describe...

http://mengwong.com/rssemail/

From: 'Capturables from Bad Hair Day #6'
(http://www.josschuurmans.com/2009/08/capturables-from-bad-hair-day-6.html)

"(...) The threesome talked about Anil's vision of the pushbutton web, on which he wrote an insightful post recently, how this relates to Dave's recent work on rssCloud, and to Google's PubSubHubBub. (...)

One application Anil dreams about in the show is of a spreadsheet which' cells are populated with formulas and live RSS feeds, so that streaming data can be analyzed in realtime.

Interesting stuff! (...)"

I noticed that no-one has mentioned the fact that PubSubHubbub is in fact a server-server protocol. It's not intended to used by clients (such as typical RSS readers), by virtue of the fact that hubs post notifications via HTTP POST's. That is, the old problem of true push notifications to clients that may be connecting through a firewall isn't addressed.

See the PubSubHubbub FAQ.

Wow, this is an old idea compensating for HTML's limitations. How is this a breakthrough? The blueprints for building the right solution are 20 years old and ready for duplication.

Great article Anil,
but I woul not rule out XMPP as a way to build P2P apps, even without central server state as GWave does. Look at LinkedProcess on another XMPP based protocol for executing more complex stuff over XMPP, turning any device into a cloud participant.

IMHO, this is one way to go, combining scalable infrastructure with P2P communication at application level.

Count me in when this is written in a way that makes the value propostion clear to the rest of us.

I'm all for distributed/simpler message delivery in the cloud that end users don't have to worry about, but how you explain it here fall short. Suggest taking another crack at it to clearly explain the benefits. Or maybe it's just plumbing stuff that end users won't care about, since just about everything is real-time these days, even email.

I definitely agree. We are moving to a web of loosely coupled objects and the concept of 'publishing' as it exists today is becoming archaic.

Anyone that doubts the value of real-time, take a look at the most-used real-time technology: SMS. They are more expensive to send than email, but people are willing to pay simply because they are real-time - that is, the sender expects the receiver to become immediately aware of the message.
In general, people spend an awful lot of their internet time repeatedly checking for things they could be notified about - an eagerly awaited email, an ebay auction, etc.
However, to introduce mainstream users to real-time, it needs to be built into browsers. Even better would be computers that could be activated remotely - when an important event happens, it switches itself on and displays it immediately.
Another use for real-time is search engine web crawling: notifiying search engines immediately that websites have been updated will make their indexes more up-to-date. Also, a lot of unnecessary crawling could be avoided.

@Jeffrey "If there were a class of applications to be built using this stack, it would have emerged by now, don't you think?"

HTML - 1990, CSS - 1996, Javascript - 1995, XML - 1996, XMLHttpRequest - 1999.
All these are the ingredients of AJAX which was relatively unknown until 2005. So it tool 6 years after the XMLHttpRequest for AJAX to popularize and be successful.

What I mean to say is that there is still time for people to try and build applications using this stack. So it should not be written off yet. It is just that like good scotch whiskey, the right blend takes a long time to be perfected.

Excellent extensive article on Real Time Web. (Dugg!) Although, IMHO, it misses a very important point about Real Time Web/Systems, which is, notification. Having your message delivered in real-time, for me, it's just part of the solution. We also need effective ways, and maybe protocols and technologies, to push (and control) notifications into our desktops, mobile phones, e-mail, xmpp, et al. And then again the problem of information overload. It's a tricky issue in my opininion, but it can be solved. I'm sure that in time we will have a set of technology and a pretty neat platform to power real time web/systems - with a gorgeous way to push notifications also.

Cheers. Congratulations.

Pedro Machado Santa