Results tagged “rss”
October 30, 2009
Humans create the web, but we've largely abdicated the act of organizing web content to software. That could change.
- Twitter this week made its new Lists feature broadly available. As they've been described, Lists, allow you to enumerate a collection of some of the Twitter accounts that you follow, and then easily read updates from just those accounts. Others can view your lists, and choose to subscribe to them as well. But Lists are also available for other applications to use, modify and share. Looked at from a slightly different perspective, this means Lists are a way to tag an arbitrary set of realtime web feeds. You could look at the lists that I've been added to as a set of tags describing my Twitter feed.
- Much of the precedent for the idea of sharing (non-realtime) feeds comes from the world of outlining, and in particular Dave Winer's work here in creating OPML. Though it was designed to generically exchange outlines, OPML is the most popular format today for sharing arbitrary lists of feeds. (The computer science folks balk at some of the technical aspects of OPML but it's a bit like Churchill's comments on democracy — it's the worst format, except for all of the other alternatives.) What's interesting about having an established format for exchanging feeds is that there doesn't really need to be any changes in order for the format to accommodate realtime feeds like Twitter accounts. In fact, a few weeks ago, I moved about 150 the noisier, less pressing Twitter accounts I follow into Google Reader, by exporting them as an OPML file. Twitter became more pleasant to use, and I could still keep up with all of those folks by dipping into my feed reader whenever I want to.
- Lists have a few traits that make them more interesting than they seem; we can think of these as the Laws of Lists. First, you have to be signed in to Twitter with a valid account in order to create them. (This seems obvious, but it's important.) Second, by adding a Twitter accounts to a list that you create, you follow that user's updates, at least while viewing that list. This combination of authentication and requirement of relationship is a very good recipe for reducing spam.
- One of the earliest hopes for organizing web information was the human-edited directory. Efforts like the Open Directory Project still exist, but the model focused a lot on having defined editors for topics and a hierarchy of who could edit the site. That's a stark contrast to the default-open editing permissions of projects like Wikipedia, and is probably the most significant difference between the "human-edited" and "user-generated" eras of the web — we've always had people contributing content, the difference was in how much we trust them. Similarly, more outline-focused directories of content emerged, like Halley Suitt's Top Ten Sources, which is now defunct, but was based upon the idea of curated lists of feeds by topic. In each case, trying to scale a team of editors to keep up with the rate of growth in new sites on the web has been a losing cause. But we've seen sites like Delicious demonstrate the value of tagging individual pages or posts on a site — a new generation of directories could demonstrate the value of tagging entire streams of posts, or as we call them, feeds.
- Of course, you can't talk about directories and lists on the web without talking about Yahoo. Yahoo's original sin was in trying to create a human-edited directory of the web, and before they unfortunately achieved their goal of becoming the only successful web portal, the directory was Yahoo's signature element. (Until recently, Yahoo had maintained a page with the directory in a format resembling its original state, but even that is basically a blog now.) Instead of embracing authentication and relationships to prevent spam submissions from overwhelming the site, Yahoo leaned heavily towards requiring payment for inclusion of companies in the directory, limiting its utility. Human edited directories became mostly a footnote in both Yahoo's, and the web's history.
That fundamental history of being made by humans is some part of Yahoo is trying to evoke with its Y!ou and Yahoo campaign. But of course, it's a pretty good sign that a campaign isn't going to hit its mark when a completely unknown brand like HTC can launch virtually the same campaign as a household name like Yahoo, yet both companies think their message is going to resonate.
The truth is, if Yahoo wanted to help people reimagine the web stalwart at its best, they would do well to look to their roots in a human-edited or user-generated directory. Thinking of Yahoo at its peak of influence a decade ago, it becomes clear that instead of trying to insert their ubiquitous exclamation point into you, Yahoo should look at the story of The Matrix. I don't know if the brothers Warner or Wachowski would be inclined to license the property, but the only way to truly resonate with people in a narrative of Yahoo vs. Google is by adopting this theme: Man vs. Machine.
Just as in the Matrix the humans had originally created the machines that undermined them, to some large degree, Yahoo begat Google. And Yahoo would do well to suggest that the most human way for the web to evolve is if we all work together to organize it ourselves — a mission that happens to fit in well with Yahoo's largely-mishandled acquisitions of Flickr and Delicious. I'm not sure that the marketing folks at Yahoo are going to embrace that narrative, but an interesting opportunity definitely exists around the larger concept.
We all have the ability to create and exchange curated collections of feeds, using hubs like Twitter's Lists as connection points. We can extract the descriptions from those collections to form tag clouds about individual feeds. If we want to embrace hierarchy, we can organize the collections into a hierarchy by inheriting the category structure of sites like Wikipedia. If we're worried about spammers, we can now use widely-available systems of authentication and defined relationships to define who has the authority to create lists in a particular context. And of course, the ability to aggregate all of the distributed content from a defined set of feeds in realtime has now been commoditized, where i would have been exorbitantly expensive a decade ago.
In short, we can learn from Twitter's Lists to resurrect one of the web's original ways of organizing itself: Human-curated directories. We're used to exploring photographs or individual web pages by clicking on tags that were assigned by the creators or their community, and it will be just as valuable and useful to be able to explore entire feeds the same way. Open formats and APIs for exchanging this data already exist, so I can't wait to see a few enterprising hackers build the tools that let us revisit the idea of web directories. I love computers and robots, but I love humans even more, and I think we can do a pretty good job of guiding each other to the most interesting feeds around.
August 7, 2009
Google Wave is an impressive set of technologies, the kind of stunningly slick application that literally makes developers stand up and cheer. I've played with the Google Wave test sandbox a bit, and while it's definitely too complex to live up to the "this will replace email!" hype that greeted its launch, it certainly has some cool features. So the big question is whether Wave will succeed as overall in becoming a popular standard for communications on the web, because Google has made an admirable investment in documenting the underlying platform and making it open enough for others to build on and extend. I think the answer is no, and the reason is because the Wave way is not compatible with the Web way.
What do I mean by "the Web way"? Well, if we look at the history of new technologies being adopted to extend web sites and enhance communications, we see a few trends emerge:
- Upgrades to the web are incremental. Instead of requiring a complete overhaul of your technical infrastructure, or radical changes to existing behaviors, the web tech that wins is usually the sort of thing that can be adopted piecemeal, integrated as needed or as a normal part of updating one's websites or applications.
- Understanding new tech needs to be a weekend-sized problem. For a lot of web developers, long before they start integrating a new protocol or platform into their work, they hack together a rough demo over a long weekend to make sure they truly grasp how it works. And a weekend-scale implementation on a personal site usually translates roughly into a 90-day implementation cycle in a business context, which is a reasonably approachable project size. (In tech, three days in personal effort often translates to three months of corporate effort.)
- There has to be value before everybody has upgraded. This is basically a corollary to Metcalfe's Law. While we know networks increase in value as they add more nodes, the nature of web tech is that, in order to be worthwhile, it has to provide value even if the people on the other end haven't upgraded their software or web browsers or clients or servers. Otherwise you're shouting into an empty room.
- You have to be able to understand and explain it. Duh.
Now, if we take a look at some examples of what has worked, we can see how various successful technologies have displayed these traits. One great example is feeds. When RSS feeds were new, it was easy to understand their potential immediately, and since I was working at a newspaper at the time, I just spent an afternoon understanding the format and hacking together a quick feed of headlines that anybody could subscribe to. If nobody had adopted feedreaders yet, that was no problem, since there was no cost to just having the feed sit there with no subscribers — the "nobody's upgraded" problem would only result in me having wasted a few hours.
Ajax had a similar adoption pattern. It took a little bit more time to comprehend, but not much more than an afternoon, and the development effort required for adding Ajax enhancements to an application started as a weekend-scale project and has only gone down over time. Following the principles of progressive enhancement, well-designed implementations performed just fine on older browsers or systems that couldn't handle the new features. And most sites that have added Ajax features have done so by adding the requirements as a checklist item in the course of normal ongoing updates, not as standalone efforts to migrate to a new technology.
This brings us to Wave. Wave offers excellent opportunities to extend its core features and to add richness to its "wavelets", and I have no criticisms over its utility as a developer platform that third parties can build upon. But the fundamental Wave protocols are, I fear, a bit too complex to ever be fully and correctly implemented by anyone other than Google. Interoperability is likely to be a challenge that plagues the platform for its entire existence. In short: It's likely that nobody will ever build a fully-compatible clone of Wave that competes with Google's own implementation.
Why is that true? Let's look at what's built in to Wave:
- Powerful realtime collaboration features
- Unlimited versioning of content
- Built around robust XMPP protocol
- Combines chat, document editing, and message threading — wikis + blogs + comments + IM
- Delivered as a very polished rich user interface
Each of these is a very compelling experience. But a lot of developers' reactions to seeing them was not just "I can't wait to use that!" but also "I want to add that one feature to my own existing application!". And that's where it gets tough. Let's take a look at Joe Gregorio's list of the protocols that power Wave. (Joe works at Google, but made this list before he was working on Wave. I appreciate his research and openness on this topic, and presenting his work here is a tribute to what makes Wave great, not a criticism of his effort.)
- Federation (XMPP)
- The robot protocol (JSONRPC)
- The gadget API (OpenSocial)
- The client-server protocol (As defined by GWT)
That's a lotta stuff! XMPP alone is a bear to implement, let alone to deploy at large scale. (I can't think of anyone outside of Google, Earthlink and LiveJournal who have deployed XMPP to millions of users.) But if you wanted to make another application that truly interoperates with all that Wave can do, combining all of these pieces would just be the starting point.
And people aren't looking for a replacement for email, or instant messaging, or blogs, or wikis. Those tools all work great for their intended purposes, and whatever technology augments them will likely offer a different combination of persistence and immediacy than those systems. Right now, Wave evokes all of them without being its own distinctive thing. Which means it's most useful in providing reference implementations of particular new features.
If a developer wants one of the compelling individual features of Wave, like near-realtime collaboration, they're more likely to use something like (wait for it...) Pushbutton technologies. The infrastructure afforded by the components of the Pushbutton Platform comes nowhere near the richness and polish displayed by Google Wave. Pushbutton isn't even designed to offer the benefits demonstrated by Wave. But to its credit, Pushbutton displays nowhere near the complexity of Wave in its interoperability requirements. More importantly, integrating Pushbutton features into a website or application isn't a monolithic process of building dozens of cutting-edge features, but rather can be deployed incrementally by even non-expert webmasters.
In this context, it might help to think of Pushbutton tech as a "micro-Wave". As Gina Trapani said in mentioning Google Reader's support for PubSubHubBub:
Huh-wha? you ask. Yeah, I know. It's no Google Wave. But that's what makes this exciting. This kind of small Pushbutton implementation is how real web pages will easily use existing technology to notify one another of new updates. The Google Reader/FriendFeed integration is just the first tiny step in what will be a broad deployment of realtime-enabled sites. These sites and services will let one another know when they have new data to share without the sucky inefficiencies of polling. Check out how fast FriendFeed updates when you share an item in Google Reader in the video above.
In short, it's almost zero latency.
Why is this clearly "inferior" technology going to win? Well, as just one example, XMPP is way too complicated for any normal human to deploy. Whereas if you're reading this, you probably already have access to a regular HTTP web server that could talk to a Pushbutton hub. In fact, the only two backers I know who have worked extensively with XMPP are Brad Fitzpatrick and Artur Bergman, who co-created Djabberd. And they are both excited about PubSubHubBub. Realistically, someone like Yahoo might try to do all of this, and inevitably one or two open source projects will try to lash together open implementations of each of these pieces to make a kind of FrankenWave application. There are probably already one or two teams working on the inevitable "Enterprise Wave Server" platforms as well, though I haven't heard about them myself. These efforts may succeed, but that doesn't mean they'll ever be robust enough that people will trust them for communicating on the web.
More to the point, I'm a regular blogger who knows a little bit about scripting on a normal web server. I can poke around the documentation and add a few tweaks to my RSS feed (or, in my case, do nothing and have Feedburner automatically handle it for me), and all of a sudden my blog's feed is part of the Pushbutton web, ready for others to build on. I literally wouldn't even know where to start with the Wave developer documentation if I wanted to integrate it with my site or any of the little apps I like to hack on during a long weekend. What seems more realistic — that someone will figure out a way to incrementally build on top of realtime feeds to enable Wave-like experiences, or that all this talk of Waves, wavelets and blips is going to suddenly become easy to understand.
In short, web-way tech like feeds, Ajax and Pushbutton win because people who make good sites and applications have a place to start with it. Does this mean we get fancy realtime simultaneous editing right away, now that Pushbutton exists? Nope. In fact, Wave might even get the early jump on those kinds of features for web apps, simply because it's pioneered that part of the user experience. But Wave only runs to its full potential on the most cutting-edge web browsers. And there may only be a dozen companies in the world with the in-house expertise to clone the entire complement of technologies underlying Wave in order to make a full-fledged competitor. Worse, the monolithic nature of the Wave experience means it will even be a challenge to make a full-fledged open source competitor to the official Google service.
I hope that Wave succeeds, because I love to see ambition and innovation rewarded. But I think it's mostly likely that Wave's success will be in inspiring people to create similarly compelling experiences by adding incremental enhancements to their existing sites. That's how the web's always advanced in the past.
July 24, 2009
Pushbutton is a name for what I believe will be an upgrade for the web, where any site or application can deliver realtime messages to a web-scale audience, using free and open technologies at low cost and without relying on any single company like Twitter or Facebook. The pieces of this platform have just come together to enable a whole set of new features and applications that would have been nearly impossible for an average web developer to build in the past.
The most interesting area of new development on the web is the innovation happening around realtime messaging, the ability to deliver updates to a website or application in one or two seconds. While various systems like Yahoo News Alerts or feed readers like Google Reader have offered some simple ways of delivering fairly fast notifications, they are still built on an infrastructure that relies upon requesting a web page repeatedly. These systems do the equivalent of hitting the "reload" button in your web browser over and over.
While those systems have been using these inefficient methods to deliver updates, newer platforms like Twitter, Facebook and FriendFeed have focused on building the infrastructure for efficient large-scale delivery of updates using their own proprietary networks. A lot of attention has been paid to Twitter's 140-character limit, or Facebook's News Feed, but the compelling technology that enables the user experience on these platforms is the immediacy with which updates are delivered. Earlier systems like instant messaging or chat allowed realtime messaging on a one-to-one or small group basis, but it's been harder to deliver those realtime messages to anyone in the world who wanted to receive them unless you had a lot of money, expertise and infrastructure.
Another barrier is that, while there are many different programs and clients that let you connect to Twitter or Facebook with your own applications, there haven't been any free and open options for delivering realtime messages to a large audience if you couldn't, or didn't want to, rely on those companies.
But recently, a few key pieces have fallen into place that make it inexpensive and relatively easy to add realtime messaging as an incremental upgrade to existing websites and web applications. This set of related technologies, which I'm calling the Pushbutton platform, will yield a broad new set of capabilities for users, publishers and developers on the web. Best of all, Pushbutton technologies are free, open and decentralized, meaning that the arrival of realtime on the web will not be owned or controlled by any single company.
The concept and potential of Pushbutton is a lot like Ajax — it's not a single technology or invention, it's a whole family of technologies, some of which have been in development or deployment for nearly a decade, that together enable this new realtime web. Pushbutton's foundation is built on these systems:
- Atom and RSS: The most common feed formats, for syndication on the web
- PubSubHubBub and RSSCloud: Powerful new "hubs" for distributing messages
- Web Hooks: Simple web services for receiving messages, rather than sending them
Pushbutton systems rely on the web's fundamental HTTP protocol for communication between these component parts. The architecture of Pushbutton message delivery is also simple to understand. Before Pushbutton, in today's systems, when you create a message (a blog post, tweet or other update) that's published in your RSS or Atom feed, every application or site that wants updates from you has to repeatedly request your feed to know when it's updated. You can optionally notify ("ping") some applications to tell them it's time to come collect your new updates, but this is time-consuming and resource-intensive on both sides, especially if you want to notify a lot of people.
In the best case, the system we have now is analogous to a person coming by your house and saying "Hey, there's a new edition of your favorite newspaper today. You should go get it." And then you have to go to the newspaper's printing plant to pick it up. In a Pushbutton web, that person is delivering each story to your house the moment it's complete.
That's because Pushbutton-enabled applications will improve upon the current state of affairs by proactively delivering not just the notification that there's a new message, but the content of the message itself. And instead of requiring all those applications to come to your site to read the update, it uses a hub server in the cloud to pass along the message directly to all the receivers that are interested in it.
- You, the Sender, create a message to be delivered via RSS or Atom
- Your application gives the messsage to one or more PubSubHubBub or RSSCloud hubs, which reside in the Cloud
- The PubSubHubBub or RSSCloud hubs deliver the message to any Receivers, the applications or sites that have requested updates from you
In this way, each time you create a new message, a large number of Receivers can consume that message in near realtime (usually less than a second) without a lot of complexity. This kind of messaging has been possible with custom-built or more obscure technologies in the past, but the Pushbutton ecosystem is a breakthrough for a few reasons:
- Sending messages just requires a minor change to an RSS or Atom feed, and a simple, well-defined update notification, instead of major changes to the application where you create your messages.
- Receiving messages is also very simple, only requiring a developer to handle incoming notifications of updates.
- Most of the system's complexity is handled in the hub servers, which are well-documented, implementable in a variety of programming languages, and built around open code that will likely attract a large developer community.
- Most of the scaling effort and expense happens at the hub level, and all current hubs are designed to run on inexpensive cloud systems like Google App Engine or Amazon's EC2.
- The software for Sending, Receiving or running a hub is free, open source and available on almost any platform.
- Messages sent on Pushbutton platforms are delivered via HTTP, which is familiar to any web developer and runs well on any hosting environment. All requests between the different layers of a Pushbutton system can be made as simple REST calls.
- Pushbutton technologies can be adopted incrementally, so that features can be added piecemeal on either the sender or receiver side, without requiring a wholesale upgrade to infrastructure or application architecture.
Who's Behind Pushbutton?
Pushbutton technologies have been created and advocated by some of the most credible and experienced developers of social web technologies. Here's a brief overview of the impressive pedigree of these components:
- PubSubHubBub was co-created by Brad Fitzpatrick and Brett Slatkin of Google. Brad was founder of LiveJournal, and created or co-created fundamental social web technologies like Memcached, OpenID and more.
- XML-RPC update pings, RSS and the RSS Cloud ideas were pioneered by Dave Winer, who has been actively developing open implementations of each of these technologies.
- Web Hooks have been evangelized by Jeff Lindsay, and have been deployed by a variety of different companies and platforms which all independently developed the technique.
In addition, Google has supported Brad and Brett's development of PubSubHubBub, and enabled it on the Google FeedBurner service. A number of smaller companies are deploying large parts of this infrastructure as well. In short, some of the best reputations in developing open web systems have made Pushbutton possible, from the biggest tech companies to the most steadfastly independent developers on the web.
Related Ideas and Prior Art
There are a lot of existing technologies that have influenced the creation and evolution of Pushbutton technologies; If you're familiar with any of these systems, you're probably already ahead of the curve in understanding part of what Pushbutton is trying to enable.
- Twitter Firehose, FriendFeed SUP, TypePad Update Stream: These realtime delivery systems offer up the content of their respective platforms as an unending stream that developers can consume and use in their applications. At the present time, they all have varying licenses and degrees of openness, and slightly different formats for delivering updates, but have proven the utility of the "sending" part of Pushbutton's realtime functionality.
- XMPP (Jabber), NNTP (Usenet), IRC: These older internet protocols all delivered various degrees of realtime messaging and distributed messaging capabilities, and can form a very useful base of experience for Pushbutton developers to learn from. In some cases, fundamental architectural choices about security, authentication or architecture were made when the Internet was less populated and less complex, making them inappropriate for today's applications. In all cases, these protocols are less-known by most contemporary web developers, and thus lack familiar toolkits and development resources, which make them quite challenging to deploy in common, inexpensive environments.
- TrackBack and Pingback: These systems for delivering updates between blogging systems were very effective in enabling rich distributed conversations in the early days of the blogosphere. These have declined in usefulness due to poor or missing implementations of authentication, which led to spam problems, and a general lack of understanding of their utility by a lot of newer bloggers. Pushbutton may offer an opportunity to restore some of the value of the idea behind these systems.
- Reverse HTTP may end up being a useful component of some Pushbutton deployments, as a complement or companion to Comet and related techniques.
What should we worry about?
- A format war? If you're familiar with the communities around technologies like feeds, you may know they have a deserved reputation for being contentious and even breaking into heated disputes over arcane details. I don't think that's likely to happen this time, because there are only one or two viable formats for each layer of the platform, and the creators of each part have shown some consistent good-faith efforts to promote interoperability where possible and peaceful coexistence where necessary. In the Ajax community, for example, the "X" in Ajax often stands for JSON instead of XML, but this hasn't hindered its broad adoption at all. I'm also willing to personally commit to try to prevent any kind of interpersonal conflict that would inhibit the adoption of Pushbutton technologies. Worry? No.
- Scaling issues? There will inevitably be some learning to do about how to scale the resource-intensive hub layer of a Pushbutton system. But because the hubs live on cloud systems that make enormous amounts of computing resources easily available, because the coders creating the reference implementations of the hub software have great experience making web-scale systems, and because it's relatively simple to introduce new hubs as needed, this will likely not be a gating factor for adoption of Pushbutton. Worry? No.
- Intellectual Property Concerns? I'm not a lawyer, and this isn't legal advice. But there has already been a great deal of interest in these systems, and it's likely that any bad actors who were interested in throwing their patent lawyers at this sort of system would probably already be suing people left and right. And the main players who are already involved have shown a consistent desire to make truly open systems that don't have IP encumbrances. Put simply, I think anybody smart enough to invent these kinds of technologies is smart enough to not want to look like jerks by suing somebody for using them. Worry? Probably not.
- Competition from centralized systems? Pushbutton technologies are not just free and open, they're decentralized, which is a serious threat to the "lobster trap" model of social software. We can expect serious competition from the centralized networks that are currently building these sorts of systems. If a threat arises to Pushbutton's adoption, this is the most likely source. Worry? Definitely.
- Bad user experience? One of the worst things we can do in making use of new technologies is to ignore the social, personal or even political implications of their use. Messages that are immediately delivered can't, by their nature, be erased from all the places they appear. The idea of permanently archiving these types of messages is unfamiliar to a lot of less technically-savvy users. And whenever we see something shiny and new, we have the temptation to use technology for technology's sake, whether or not we're solving a real problem or providing a real value. If Pushbutton gets a bad rap early on despite having tremendous potential, this will be why. Worry? Hell, yes.
I have tremendous excitement about the new realtime era of web applications. While I'm fundamentally an optimistic person, I have great skepticism when it comes to mindless hype about new technologies, so it's with a bit of reluctance that I indulge in some hype myself. But I think the Pushbutton web has the opportunity to give individuals and organizations with distinct and passionate voices the ability to be even more immediate and expressive on the web, and after ten years of publishing on the web, that's the part I love the most.
I have no doubt that some skeptics will say "Pushbutton is just PubSubHubBub by another name", just like they said "Ajax is XMLHttpRequest by another name", and if that's what the super-geeky guys want to believe, I'm fine with that. And I'm sure there will still be some significant technical details to resolve. But I think by giving the overall concept an approachable, understandable name and (hopefully!) an explanation that can be understood by anyone with an interest, it can catalyze interest in a whole new area of innovation on the web. And to be honest, when I see folks like Brad Fitzpatrick and Dave Winer hacking on the same set of problems, I can't help but think something interesting will come of it.
Over the next few days, I'll be outlining some of the opportunities around Pushbutton, espousing more of the philosophy that has the potential to imbue Pushbutton with a bit more meaning than most new web tech, and providing some simple explanations of how you can get started both learning about and taking advantage of these technologies. Most of all, I hope you'll offer your pointed criticisms, thoughtful critiques, detailed corrections and even better ideas. I'll be following the conversation here in the comments, across the blogosphere, and on Twitter using the tag #pshb.
July 12, 2006
Just a little geek note, there's now a box that will appear the first time you visit my site (it disappears after the first visit) that will let you subscribe to my feed in your favorite feed reader, or to get posts delivered via email.
If you haven't already done so, or the box went away before you got a chance to subscribe, or you just needed a reminder, here's the links:
And of course, there's the requisite XML Feed link:
Now back to our regularly scheduled idiocy.