Results tagged “atom”

The Pushbutton Web: Realtime Becomes Real

July 24, 2009

Pushbutton is a name for what I believe will be an upgrade for the web, where any site or application can deliver realtime messages to a web-scale audience, using free and open technologies at low cost and without relying on any single company like Twitter or Facebook. The pieces of this platform have just come together to enable a whole set of new features and applications that would have been nearly impossible for an average web developer to build in the past.

Background

The most interesting area of new development on the web is the innovation happening around realtime messaging, the ability to deliver updates to a website or application in one or two seconds. While various systems like Yahoo News Alerts or feed readers like Google Reader have offered some simple ways of delivering fairly fast notifications, they are still built on an infrastructure that relies upon requesting a web page repeatedly. These systems do the equivalent of hitting the "reload" button in your web browser over and over.

Pushbutton FlowWhile those systems have been using these inefficient methods to deliver updates, newer platforms like Twitter, Facebook and FriendFeed have focused on building the infrastructure for efficient large-scale delivery of updates using their own proprietary networks. A lot of attention has been paid to Twitter's 140-character limit, or Facebook's News Feed, but the compelling technology that enables the user experience on these platforms is the immediacy with which updates are delivered. Earlier systems like instant messaging or chat allowed realtime messaging on a one-to-one or small group basis, but it's been harder to deliver those realtime messages to anyone in the world who wanted to receive them unless you had a lot of money, expertise and infrastructure.

Another barrier is that, while there are many different programs and clients that let you connect to Twitter or Facebook with your own applications, there haven't been any free and open options for delivering realtime messages to a large audience if you couldn't, or didn't want to, rely on those companies.

But recently, a few key pieces have fallen into place that make it inexpensive and relatively easy to add realtime messaging as an incremental upgrade to existing websites and web applications. This set of related technologies, which I'm calling the Pushbutton platform, will yield a broad new set of capabilities for users, publishers and developers on the web. Best of all, Pushbutton technologies are free, open and decentralized, meaning that the arrival of realtime on the web will not be owned or controlled by any single company.

Defining Pushbutton

The concept and potential of Pushbutton is a lot like Ajax — it's not a single technology or invention, it's a whole family of technologies, some of which have been in development or deployment for nearly a decade, that together enable this new realtime web. Pushbutton's foundation is built on these systems:

  • Atom and RSS: The most common feed formats, for syndication on the web
  • PubSubHubBub and RSSCloud: Powerful new "hubs" for distributing messages
  • Web Hooks: Simple web services for receiving messages, rather than sending them

Pushbutton systems rely on the web's fundamental HTTP protocol for communication between these component parts. The architecture of Pushbutton message delivery is also simple to understand. Before Pushbutton, in today's systems, when you create a message (a blog post, tweet or other update) that's published in your RSS or Atom feed, every application or site that wants updates from you has to repeatedly request your feed to know when it's updated. You can optionally notify ("ping") some applications to tell them it's time to come collect your new updates, but this is time-consuming and resource-intensive on both sides, especially if you want to notify a lot of people.

In the best case, the system we have now is analogous to a person coming by your house and saying "Hey, there's a new edition of your favorite newspaper today. You should go get it." And then you have to go to the newspaper's printing plant to pick it up. In a Pushbutton web, that person is delivering each story to your house the moment it's complete.

That's because Pushbutton-enabled applications will improve upon the current state of affairs by proactively delivering not just the notification that there's a new message, but the content of the message itself. And instead of requiring all those applications to come to your site to read the update, it uses a hub server in the cloud to pass along the message directly to all the receivers that are interested in it.

pushbutton delivery

  1. You, the Sender, create a message to be delivered via RSS or Atom
  2. Your application gives the messsage to one or more PubSubHubBub or RSSCloud hubs, which reside in the Cloud
  3. The PubSubHubBub or RSSCloud hubs deliver the message to any Receivers, the applications or sites that have requested updates from you

In this way, each time you create a new message, a large number of Receivers can consume that message in near realtime (usually less than a second) without a lot of complexity. This kind of messaging has been possible with custom-built or more obscure technologies in the past, but the Pushbutton ecosystem is a breakthrough for a few reasons:

  • Sending messages just requires a minor change to an RSS or Atom feed, and a simple, well-defined update notification, instead of major changes to the application where you create your messages.
  • Receiving messages is also very simple, only requiring a developer to handle incoming notifications of updates.
  • Most of the system's complexity is handled in the hub servers, which are well-documented, implementable in a variety of programming languages, and built around open code that will likely attract a large developer community.
  • Most of the scaling effort and expense happens at the hub level, and all current hubs are designed to run on inexpensive cloud systems like Google App Engine or Amazon's EC2.
  • The software for Sending, Receiving or running a hub is free, open source and available on almost any platform.
  • Messages sent on Pushbutton platforms are delivered via HTTP, which is familiar to any web developer and runs well on any hosting environment. All requests between the different layers of a Pushbutton system can be made as simple REST calls.
  • Pushbutton technologies can be adopted incrementally, so that features can be added piecemeal on either the sender or receiver side, without requiring a wholesale upgrade to infrastructure or application architecture.

Who's Behind Pushbutton?

Pushbutton technologies have been created and advocated by some of the most credible and experienced developers of social web technologies. Here's a brief overview of the impressive pedigree of these components:

  • PubSubHubBub was co-created by Brad Fitzpatrick and Brett Slatkin of Google. Brad was founder of LiveJournal, and created or co-created fundamental social web technologies like Memcached, OpenID and more.
  • XML-RPC update pings, RSS and the RSS Cloud ideas were pioneered by Dave Winer, who has been actively developing open implementations of each of these technologies.
  • Web Hooks have been evangelized by Jeff Lindsay, and have been deployed by a variety of different companies and platforms which all independently developed the technique.

In addition, Google has supported Brad and Brett's development of PubSubHubBub, and enabled it on the Google FeedBurner service. A number of smaller companies are deploying large parts of this infrastructure as well. In short, some of the best reputations in developing open web systems have made Pushbutton possible, from the biggest tech companies to the most steadfastly independent developers on the web.

Related Ideas and Prior Art

There are a lot of existing technologies that have influenced the creation and evolution of Pushbutton technologies; If you're familiar with any of these systems, you're probably already ahead of the curve in understanding part of what Pushbutton is trying to enable.

  • Twitter Firehose, FriendFeed SUP, TypePad Update Stream: These realtime delivery systems offer up the content of their respective platforms as an unending stream that developers can consume and use in their applications. At the present time, they all have varying licenses and degrees of openness, and slightly different formats for delivering updates, but have proven the utility of the "sending" part of Pushbutton's realtime functionality.
  • XMPP (Jabber), NNTP (Usenet), IRC: These older internet protocols all delivered various degrees of realtime messaging and distributed messaging capabilities, and can form a very useful base of experience for Pushbutton developers to learn from. In some cases, fundamental architectural choices about security, authentication or architecture were made when the Internet was less populated and less complex, making them inappropriate for today's applications. In all cases, these protocols are less-known by most contemporary web developers, and thus lack familiar toolkits and development resources, which make them quite challenging to deploy in common, inexpensive environments.
  • TrackBack and Pingback: These systems for delivering updates between blogging systems were very effective in enabling rich distributed conversations in the early days of the blogosphere. These have declined in usefulness due to poor or missing implementations of authentication, which led to spam problems, and a general lack of understanding of their utility by a lot of newer bloggers. Pushbutton may offer an opportunity to restore some of the value of the idea behind these systems.
  • Reverse HTTP may end up being a useful component of some Pushbutton deployments, as a complement or companion to Comet and related techniques.

What should we worry about?

  • A format war? If you're familiar with the communities around technologies like feeds, you may know they have a deserved reputation for being contentious and even breaking into heated disputes over arcane details. I don't think that's likely to happen this time, because there are only one or two viable formats for each layer of the platform, and the creators of each part have shown some consistent good-faith efforts to promote interoperability where possible and peaceful coexistence where necessary. In the Ajax community, for example, the "X" in Ajax often stands for JSON instead of XML, but this hasn't hindered its broad adoption at all. I'm also willing to personally commit to try to prevent any kind of interpersonal conflict that would inhibit the adoption of Pushbutton technologies. Worry? No.
  • Scaling issues? There will inevitably be some learning to do about how to scale the resource-intensive hub layer of a Pushbutton system. But because the hubs live on cloud systems that make enormous amounts of computing resources easily available, because the coders creating the reference implementations of the hub software have great experience making web-scale systems, and because it's relatively simple to introduce new hubs as needed, this will likely not be a gating factor for adoption of Pushbutton. Worry? No.
  • Intellectual Property Concerns? I'm not a lawyer, and this isn't legal advice. But there has already been a great deal of interest in these systems, and it's likely that any bad actors who were interested in throwing their patent lawyers at this sort of system would probably already be suing people left and right. And the main players who are already involved have shown a consistent desire to make truly open systems that don't have IP encumbrances. Put simply, I think anybody smart enough to invent these kinds of technologies is smart enough to not want to look like jerks by suing somebody for using them. Worry? Probably not.
  • Competition from centralized systems? Pushbutton technologies are not just free and open, they're decentralized, which is a serious threat to the "lobster trap" model of social software. We can expect serious competition from the centralized networks that are currently building these sorts of systems. If a threat arises to Pushbutton's adoption, this is the most likely source. Worry? Definitely.
  • Bad user experience? One of the worst things we can do in making use of new technologies is to ignore the social, personal or even political implications of their use. Messages that are immediately delivered can't, by their nature, be erased from all the places they appear. The idea of permanently archiving these types of messages is unfamiliar to a lot of less technically-savvy users. And whenever we see something shiny and new, we have the temptation to use technology for technology's sake, whether or not we're solving a real problem or providing a real value. If Pushbutton gets a bad rap early on despite having tremendous potential, this will be why. Worry? Hell, yes.

Conclusion

I have tremendous excitement about the new realtime era of web applications. While I'm fundamentally an optimistic person, I have great skepticism when it comes to mindless hype about new technologies, so it's with a bit of reluctance that I indulge in some hype myself. But I think the Pushbutton web has the opportunity to give individuals and organizations with distinct and passionate voices the ability to be even more immediate and expressive on the web, and after ten years of publishing on the web, that's the part I love the most.

wired-push-1997-sm.jpgI have no doubt that some skeptics will say "Pushbutton is just PubSubHubBub by another name", just like they said "Ajax is XMLHttpRequest by another name", and if that's what the super-geeky guys want to believe, I'm fine with that. And I'm sure there will still be some significant technical details to resolve. But I think by giving the overall concept an approachable, understandable name and (hopefully!) an explanation that can be understood by anyone with an interest, it can catalyze interest in a whole new area of innovation on the web. And to be honest, when I see folks like Brad Fitzpatrick and Dave Winer hacking on the same set of problems, I can't help but think something interesting will come of it.

Over the next few days, I'll be outlining some of the opportunities around Pushbutton, espousing more of the philosophy that has the potential to imbue Pushbutton with a bit more meaning than most new web tech, and providing some simple explanations of how you can get started both learning about and taking advantage of these technologies. Most of all, I hope you'll offer your pointed criticisms, thoughtful critiques, detailed corrections and even better ideas. I'll be following the conversation here in the comments, across the blogosphere, and on Twitter using the tag #pshb.

Atom Wins: The Unified Cloud Database API

April 4, 2008

Almost five years ago, I got involved in a project that would eventually become Atom, a pair of matching standards functioning as a syndication format and a publishing protocol. Though its contentious genesis was in the world of blogging and feeds, what's amazing is the improbable end result of the Atom community refining and debating the standard for half a decade.

AtomPub has become the standard for accessing cloud-based data storage. From Google's GData implementation to Microsoft's unified storage platform that includes SQL Server, Atom is now a consistent interface to build applications that can talk to remote databases, regardless of who provides them. And this milestone has come to pass with almost no notice from the press that covers web APIs and technology. A single common interface for keeping data in reliable cloud storage is as important a development as infrastructure-on-demand offerings like Amazon's EC2 and S3 web services, and can radically impact the creation of applications for distributed social platforms like OpenSocial or the Facebook API.

Of course, there's a lot more to the AtomPub story. There's open source support from infrastructure-level implementations like Apache Abdera to application implementations as in Movable Type and interoperability testing as made possible by The Ape. But those have been around for a while: Being able to build a program that doesn't care if the backend is Google Base or SQL Server is new.

The reason this is a true milestone, and that it's especially disappointing that the tech trade press hasn't been paying attention, is because we've seem some remarkable announcements.

From Microsoft's David Treadwell:

Microsoft is making a large investment in unifying our developer platform protocols for services on the open, standards-based Atom format (RFC 4287) and the Atom Publishing Protocol (RFC 5023). At MIX we are enabling several new Live services with AtomPub endpoints which enable any HTTP-aware application to easily consume Atom feeds of photos and for unstructured application storage (see below for more details). Or you can use any Atom-aware public tools or libraries, such as .NET WCF Syndication to read or write these cloud service-based feeds. ...

The intent for these early, experimental releases are to gather valuable feedback from the community around our idiomatic and freely licensed extensions to AtomPub which deal with important service scenarios, such as URL formats, nested directories, image streams, and service metadata. You can read more about this on the Project Astoria team blog.

And from Google's Joe Gregorio:

We've always encouraged other developers to adopt Atom, the Atom Publishing Protocol, and the extensions that Google has created on top of those standards, but we realized the issue of patents may have held back some adopters. Well, those concerns end today as we are giving a no-charge, royalty-free license to any patents we have that you would need to implement Atom, AtomPub, or any of those extensions.

There's clearly still work to be done, of course. I believe Microsoft will find Google's licensing terms open enough for them to feel comfortable implementing GData-compatible APIs. Amazon should find a way to map these same APIs onto the SimpleDB service, though I understand fundamental differences in architecture may make that a challenge. The multitude of smaller cloud-database vendors will find it easier to get adoption and customers if they can assert compatibility with these implementations.

But, put succinctly, Google + Microsoft = AtomPub wins. To paraphrase Dave Winer, the act of putting aside ego and saying a competitor's API is good enough, and that you're going to support it, is a brave and important act in the world of technology. That makes this convergence particularly exciting.

And it's a real act of technological courage because any reasonably competent technology company can find reasons to object to any standard or API that's ever presented. "We'd prefer it work this way." "Your implementation is too verbose!" "What about just using this other technology?"

In the end, it doesn't matter if a standard is perfect; All that matters is that it works. I am, by no means, the kind of person who thinks that anything adopted by two giant tech companies is always going to be acceptable, but the fact that there is also significant support for AtomPub from small, independent hackers is a pretty good sign that this victory is good for the web as a whole.

Now I just hope people make something really cool out of this. I want every program that thinks of itself today as a "blogging client" to reimagine their market as being a front-end to a database in the cloud. I want all the apps built on smart database abstractions to think about this new unified cloud API as an option they must support. And most of all, I want geeks to make something cool with this that we couldn't do before.

Some key links:

Subscribe! (A little housekeeping)

July 12, 2006

Just a little geek note, there's now a box that will appear the first time you visit my site (it disappears after the first visit) that will let you subscribe to my feed in your favorite feed reader, or to get posts delivered via email.

If you haven't already done so, or the box went away before you got a chance to subscribe, or you just needed a reminder, here's the links:

  • Add to Google
  • Add to My Yahoo!
  • Subscribe in NewsGator Online
  • Subscribe in Bloglines
  • Add to My AOL
  • Add to My MSN!
  • Add to LiveJournal Friends



And of course, there's the requisite XML Feed link:

Feed Icon Subscribe to this blog's feed (What's an XML Feed?)

Now back to our regularly scheduled idiocy.

1