Results tagged “pipes”

A couple Clay Shirky links for you today, one of which I just linked to, one of which I should have linked to last week, and all worth reading.

• In Defense of Ready, Fire Aim, a piece Clay wrote for the Harvard Business Review's list of breakthrough ideas for 2007. I really am a sucker for the Annual Ideas Lists that more and more publications are putting out, but a few of the items in this list seem particularly valuable. I linked to Clay's essay in yesterday's post about Yahoo Pipes, referring to his concept of open source being based on a principal of embracing failure:

In open systems, by contrast, the cost of failure is reduced, partly because less coordination is required among the various players and partly because each player is willing to accept some of the risks of failure directly. This means that worrying about whether a new idea will succeed is unnecessary; you simply try it out. The institutional barrier between thought and action—the need to convince someone that your idea is worth giving a whirl—doesn’t exist. The low cost of trying means that participants can fail like crazy as they continue to build on their successes.

In systems where anyone can try anything, the good has to be filtered from the bad after the fact. The cost of trying to prevent bloggers from saying stupid or silly things, for example, would be high, whereas the cost of allowing anyone to publish anything is low.

• A Clay Classic: A Group Is Its Own Worst Enemy. I was actually at the Etech conference where Clay delivered this speech; Stewart and Ben had just debuted Flickr a short while earlier, and the application was (at that point) a weird Flash community application that could do some image stuff in addition to working as an IM gateway. Clay was talking about group behaviors as exhibited on LiveJournal and The WELL and MetaFilter, with his points being especially relevant after the recent Yahoo logins hubbub on Flickr.

This pattern has happened over and over and over again. Someone built the system, they assumed certain user behaviors. The users came on and exhibited different behaviors. And the people running the system discovered to their horror that the technological and social issues could not in fact be decoupled.

This passage is especially resonant after the launch of Pipes:

We've gotten weblogs and wikis, and I think, even more importantly, we're getting platform stuff. We're getting RSS. We're getting shared Flash objects. We're getting ways to quickly build on top of some infrastructure we can take for granted, that lets us try new things very rapidly.

I was talking to Stewart Butterfield about the chat application they're trying here. [Speaking of Flickr.] I said "Hey, how's that going?" He said: "Well, we only had the idea for it two weeks ago. So this is the launch." When you can go from "Hey, I've got an idea" to "Let's launch this in front of a few hundred serious geeks and see how it works," that suggests that there's a platform there that is letting people do some really interesting things really quickly. It's not that you couldn't have built a similar application a couple of years ago, but the cost would have been much higher. And when you lower costs, interesting new kinds of things happen.

• A response to Henry Jenkins about Second Life. Clay has been taking a much-needed hard look at Second Life for some time, but I really am just linking to this because, as a student of social communication technology, Clay is uniquely qualified to create howlingly funny and yet still somehow polite and refined jabs at his debate partners. It's like the world's most dignified flame war. Witness:

You compare Second Life with the Renaissance and the Age of Reason. This is approximately insane, and your disclaimer that Second Life may not reach this rarefied plateau doesn’t do much to make it less insane. Using the Renaissance as a reference point links the two in the reader’s mind, even in the face of subsequent denial.

• And then finally, as Andre started experimenting with Arduino (film at 11!), who should pop up as the author of the source code for the Dimming LEDs tutorial but one Mr. Clay Shirky:

/*
* Code for cross-fading 3 LEDs, red, green and blue, or one tri-color LED, using PWM
* The program cross-fades slowly from red to green, green to blue, and blue to red
* The debugging code assumes Arduino 0004, as it uses the new Serial.begin()-style functions
* Clay Shirky <clay.shirky@nyu.edu> 
*/

// Output
int redPin   = 9;   // Red LED,   connected to digital pin 9
int greenPin = 10;  // Green LED, connected to digital pin 10
int bluePin  = 11;  // Blue LED,  connected to digital pin 11

// Program variables
int redVal   = 255; // Variables to store the values to send to the pins
int greenVal = 1;   // Initial values are Red full, Green and Blue off
int blueVal  = 1;

int i = 0;     // Loop counter    
int wait = 50; // 50ms (.05 second) delay; shorten for faster fades
int DEBUG = 0; // DEBUG counter; if set to 1, will write values back via serial

(Thanks to Cory Doctorow for the image.)

Background: Yahoo's launched an interesting and innovative new service, Pipes, which lets users with a relatively low degree of technological expertise combine structured sources of web data such as feeds. In this way, it's possible for non-experts to create new web services for their own use or for public consumption. Pipes combines a remarkably sophisticated development environment with some core social features such as the ability to clone or share the web services you produce. The service is fairly approachable, but somewhat complex once you get just under the surface, and should be moderately successful while radically raising the bar for other tools in its category.

Some quick links if you're interested in this topic:

• Plagger: an open-source, installable feed routing system created by Tatsuhiko Miyagawa which performs much of the core functionality of Pipes and is customizable, but lacks the user interface and integrated development environment (IDE) which distinguish Pipes. (Disclaimer, for what it's worth: I work with Tatsuhiko. But Plagger's free and not a Six Apart product, so... shrug.)

• Ning: Perhaps the archetypal social application platform for the web. Headed by Gina Bianchini, Ning has thus far defined the feature set for end-user creation of web applications, though the focus has not been on creating web services.
• An introduction to Unix pipes. Not merely the inspiration for the name of pipes, the powerful idea of routing data through a series of loosely-connected applications is one of the core concepts that powers most lightweight automation and data processing by non-programmers.
• The Mario-inspired image you see here is available on a Threadless T-shirt. My wife has one and it is very cute.

So, what is Pipes?

Okay, with all the introduction out of the way -- what the heck is Pipes? Yahoo's overview page offers the following explanation:

Pipes is a free online service that lets you remix popular feed types and create data mashups using a visual editor. You can use Pipes to run your own web projects, or publish and share your own web services without ever having to write a line of code.

In practice, this means you can use Pipes to pick a few feeds or APIs to retrieve data from, set up rules for processing the data which is provided by those feeds, prompt for user input in your processing, and then output the processed results as another feed or object format for use in your own applications. In the simplest case, you can apply some straightforward rules to a feed and then subscribe to the end result.

Pipes also has a full web-based IDE. Even more amazingly, the IDE is incredibly rich and powerful, with the usual complement of editing tools you'd expect from a visual editor in a desktop application. I'd first seen a browser-based IDE for editing LiveJournal's S2 style templates, but even that impressive effort pales next to the Pipes IDE. I'll never learn all of the Pipes IDE's features, but I will always admire its gee-whizzery.

Passing the Pipe

Most importantly, and perhaps most key to the success or failure of Pipes, are the social functions that underpin the application. With Pipes, it's easy to make your own web services public, to clone web services that others have made, or to offer your own services for others to clone. That element of social sharing of code, first pioneered by platforms like Ning, makes the open source ethos much simpler to participate in. Instead of setting up complex version control systems and submitting patches to a central repository, application cloning works on a principal of infinite forking, taking the idea of embracing failure and building it into the platform. Code 'em all, and let blogs sort 'em out.

There's also another key accomodation of social functionality: Pipes is pretty. As I mentioned in the introduction, much of this type of functionality is technologically possible with tools like Plagger. But, much as I love Plagger, I just don't have the patience to install half of CPAN to get it running just so I can hand-code an application on top of it. And that's even though I think it's a cool idea -- imagine if I weren't already familiar with the concept of routing feeds around.

Pipes is attractive without being overly pretty; There are the requisite nods to Web 2.0 design (it's blue!), but overall the site is refreshingly straightforward. The IDE is, frankly, a little cluttered unless you're running on a gigantic monitor, but that's been true of IDEs since Visual Studio was still Visual C++.

The Bottom Line

Is Pipes going to be a success? In many ways it already is. It lets Yahoo unequivocally be first at something, and if you count the broader market of web-based application development tools, it lets Yahoo be best at something, too. It's innovative, exciting, and well-done. There are still rough edges and inexplicable nods to the Big Purple Monster. (Whose idea was it to have the Yahoo Messenger anime avatars next to developer names on the site?) But the web needs a way to rip, mix and burn feeds, and Yahoo has stepped up to provide an essential platform in a way that seems open and approachable.

So, take a look at the docs, browse some pipes, and let me know if any of you can get it hooked up to the firehose that is the Six Apart Update Stream (that's an endless Atom feed of blog posts, flowing into your pipes at 30 posts a second). Because routing all these streams to the right place is exactly what pipes are for.