Results tagged “feeds”

Humans create the web, but we've largely abdicated the act of organizing web content to software. That could change.

• Twitter this week made its new Lists feature broadly available. As they've been described, Lists, allow you to enumerate a collection of some of the Twitter accounts that you follow, and then easily read updates from just those accounts. Others can view your lists, and choose to subscribe to them as well. But Lists are also available for other applications to use, modify and share. Looked at from a slightly different perspective, this means Lists are a way to tag an arbitrary set of realtime web feeds. You could look at the lists that I've been added to as a set of tags describing my Twitter feed.

• Much of the precedent for the idea of sharing (non-realtime) feeds comes from the world of outlining, and in particular Dave Winer's work here in creating OPML. Though it was designed to generically exchange outlines, OPML is the most popular format today for sharing arbitrary lists of feeds. (The computer science folks balk at some of the technical aspects of OPML but it's a bit like Churchill's comments on democracy — it's the worst format, except for all of the other alternatives.) What's interesting about having an established format for exchanging feeds is that there doesn't really need to be any changes in order for the format to accommodate realtime feeds like Twitter accounts. In fact, a few weeks ago, I moved about 150 the noisier, less pressing Twitter accounts I follow into Google Reader, by exporting them as an OPML file. Twitter became more pleasant to use, and I could still keep up with all of those folks by dipping into my feed reader whenever I want to.

• Lists have a few traits that make them more interesting than they seem; we can think of these as the Laws of Lists. First, you have to be signed in to Twitter with a valid account in order to create them. (This seems obvious, but it's important.) Second, by adding a Twitter accounts to a list that you create, you follow that user's updates, at least while viewing that list. This combination of authentication and requirement of relationship is a very good recipe for reducing spam.

• One of the earliest hopes for organizing web information was the human-edited directory. Efforts like the Open Directory Project still exist, but the model focused a lot on having defined editors for topics and a hierarchy of who could edit the site. That's a stark contrast to the default-open editing permissions of projects like Wikipedia, and is probably the most significant difference between the "human-edited" and "user-generated" eras of the web — we've always had people contributing content, the difference was in how much we trust them. Similarly, more outline-focused directories of content emerged, like Halley Suitt's Top Ten Sources, which is now defunct, but was based upon the idea of curated lists of feeds by topic. In each case, trying to scale a team of editors to keep up with the rate of growth in new sites on the web has been a losing cause. But we've seen sites like Delicious demonstrate the value of tagging individual pages or posts on a site — a new generation of directories could demonstrate the value of tagging entire streams of posts, or as we call them, feeds.

• Of course, you can't talk about directories and lists on the web without talking about Yahoo. Yahoo's original sin was in trying to create a human-edited directory of the web, and before they unfortunately achieved their goal of becoming the only successful web portal, the directory was Yahoo's signature element. (Until recently, Yahoo had maintained a page with the directory in a format resembling its original state, but even that is basically a blog now.) Instead of embracing authentication and relationships to prevent spam submissions from overwhelming the site, Yahoo leaned heavily towards requiring payment for inclusion of companies in the directory, limiting its utility. Human edited directories became mostly a footnote in both Yahoo's, and the web's history.

That fundamental history of being made by humans is some part of Yahoo is trying to evoke with its Y!ou and Yahoo campaign. But of course, it's a pretty good sign that a campaign isn't going to hit its mark when a completely unknown brand like HTC can launch virtually the same campaign as a household name like Yahoo, yet both companies think their message is going to resonate.

The truth is, if Yahoo wanted to help people reimagine the web stalwart at its best, they would do well to look to their roots in a human-edited or user-generated directory. Thinking of Yahoo at its peak of influence a decade ago, it becomes clear that instead of trying to insert their ubiquitous exclamation point into you, Yahoo should look at the story of The Matrix. I don't know if the brothers Warner or Wachowski would be inclined to license the property, but the only way to truly resonate with people in a narrative of Yahoo vs. Google is by adopting this theme: Man vs. Machine.

Just as in the Matrix the humans had originally created the machines that undermined them, to some large degree, Yahoo begat Google. And Yahoo would do well to suggest that the most human way for the web to evolve is if we all work together to organize it ourselves — a mission that happens to fit in well with Yahoo's largely-mishandled acquisitions of Flickr and Delicious. I'm not sure that the marketing folks at Yahoo are going to embrace that narrative, but an interesting opportunity definitely exists around the larger concept.

We all have the ability to create and exchange curated collections of feeds, using hubs like Twitter's Lists as connection points. We can extract the descriptions from those collections to form tag clouds about individual feeds. If we want to embrace hierarchy, we can organize the collections into a hierarchy by inheriting the category structure of sites like Wikipedia. If we're worried about spammers, we can now use widely-available systems of authentication and defined relationships to define who has the authority to create lists in a particular context. And of course, the ability to aggregate all of the distributed content from a defined set of feeds in realtime has now been commoditized, where i would have been exorbitantly expensive a decade ago.

In short, we can learn from Twitter's Lists to resurrect one of the web's original ways of organizing itself: Human-curated directories. We're used to exploring photographs or individual web pages by clicking on tags that were assigned by the creators or their community, and it will be just as valuable and useful to be able to explore entire feeds the same way. Open formats and APIs for exchanging this data already exist, so I can't wait to see a few enterprising hackers build the tools that let us revisit the idea of web directories. I love computers and robots, but I love humans even more, and I think we can do a pretty good job of guiding each other to the most interesting feeds around.

Background: Yahoo's launched an interesting and innovative new service, Pipes, which lets users with a relatively low degree of technological expertise combine structured sources of web data such as feeds. In this way, it's possible for non-experts to create new web services for their own use or for public consumption. Pipes combines a remarkably sophisticated development environment with some core social features such as the ability to clone or share the web services you produce. The service is fairly approachable, but somewhat complex once you get just under the surface, and should be moderately successful while radically raising the bar for other tools in its category.

Some quick links if you're interested in this topic:

• Plagger: an open-source, installable feed routing system created by Tatsuhiko Miyagawa which performs much of the core functionality of Pipes and is customizable, but lacks the user interface and integrated development environment (IDE) which distinguish Pipes. (Disclaimer, for what it's worth: I work with Tatsuhiko. But Plagger's free and not a Six Apart product, so... shrug.)

• Ning: Perhaps the archetypal social application platform for the web. Headed by Gina Bianchini, Ning has thus far defined the feature set for end-user creation of web applications, though the focus has not been on creating web services.
• An introduction to Unix pipes. Not merely the inspiration for the name of pipes, the powerful idea of routing data through a series of loosely-connected applications is one of the core concepts that powers most lightweight automation and data processing by non-programmers.
• The Mario-inspired image you see here is available on a Threadless T-shirt. My wife has one and it is very cute.

So, what is Pipes?

Okay, with all the introduction out of the way -- what the heck is Pipes? Yahoo's overview page offers the following explanation:

Pipes is a free online service that lets you remix popular feed types and create data mashups using a visual editor. You can use Pipes to run your own web projects, or publish and share your own web services without ever having to write a line of code.

In practice, this means you can use Pipes to pick a few feeds or APIs to retrieve data from, set up rules for processing the data which is provided by those feeds, prompt for user input in your processing, and then output the processed results as another feed or object format for use in your own applications. In the simplest case, you can apply some straightforward rules to a feed and then subscribe to the end result.

Pipes also has a full web-based IDE. Even more amazingly, the IDE is incredibly rich and powerful, with the usual complement of editing tools you'd expect from a visual editor in a desktop application. I'd first seen a browser-based IDE for editing LiveJournal's S2 style templates, but even that impressive effort pales next to the Pipes IDE. I'll never learn all of the Pipes IDE's features, but I will always admire its gee-whizzery.

Passing the Pipe

Most importantly, and perhaps most key to the success or failure of Pipes, are the social functions that underpin the application. With Pipes, it's easy to make your own web services public, to clone web services that others have made, or to offer your own services for others to clone. That element of social sharing of code, first pioneered by platforms like Ning, makes the open source ethos much simpler to participate in. Instead of setting up complex version control systems and submitting patches to a central repository, application cloning works on a principal of infinite forking, taking the idea of embracing failure and building it into the platform. Code 'em all, and let blogs sort 'em out.

There's also another key accomodation of social functionality: Pipes is pretty. As I mentioned in the introduction, much of this type of functionality is technologically possible with tools like Plagger. But, much as I love Plagger, I just don't have the patience to install half of CPAN to get it running just so I can hand-code an application on top of it. And that's even though I think it's a cool idea -- imagine if I weren't already familiar with the concept of routing feeds around.

Pipes is attractive without being overly pretty; There are the requisite nods to Web 2.0 design (it's blue!), but overall the site is refreshingly straightforward. The IDE is, frankly, a little cluttered unless you're running on a gigantic monitor, but that's been true of IDEs since Visual Studio was still Visual C++.

The Bottom Line

Is Pipes going to be a success? In many ways it already is. It lets Yahoo unequivocally be first at something, and if you count the broader market of web-based application development tools, it lets Yahoo be best at something, too. It's innovative, exciting, and well-done. There are still rough edges and inexplicable nods to the Big Purple Monster. (Whose idea was it to have the Yahoo Messenger anime avatars next to developer names on the site?) But the web needs a way to rip, mix and burn feeds, and Yahoo has stepped up to provide an essential platform in a way that seems open and approachable.

So, take a look at the docs, browse some pipes, and let me know if any of you can get it hooked up to the firehose that is the Six Apart Update Stream (that's an endless Atom feed of blog posts, flowing into your pipes at 30 posts a second). Because routing all these streams to the right place is exactly what pipes are for.

From the comments on one of my recent posts, here's a perfect example of the challenge of explaining technology. I'd provided a set of links for subscribing to my site.

From "AbC":

I hate something about this whole feed business. I see techies falling over themselves trying to use the simplest possible language to help the rest of us understand what feeds really are and how they work. It's like a lot of people dumb down the definitions and try to use terms they think we'll understand and be able to internalize.

Hell no.

Tell me exactly how these things work ... tell me a feed is like an HTML file that has the address/URL of where a site's latest content is posted and that you can specify exactly how you want to see the new content - snippets or full blow-by-blow.

I maybe wrong ... because techies are really muffling the real definitions with their softened, purpotedly easy to understand definitions.

I think AbC's wrong. Here's why, courtesy of Joanne:

Thanks for posting this. I've heard of RSS before, but it all seemed like black magic until I tried out bloglines. Your post inspired me to get started!

No offense, but somehow I find the argument (and potential readership) of the Joannes of the world more compelling.

Just a little geek note, there's now a box that will appear the first time you visit my site (it disappears after the first visit) that will let you subscribe to my feed in your favorite feed reader, or to get posts delivered via email.

If you haven't already done so, or the box went away before you got a chance to subscribe, or you just needed a reminder, here's the links:

And of course, there's the requisite XML Feed link:

Subscribe to this blog's feed (What's an XML Feed?)

Now back to our regularly scheduled idiocy.