Imagine if half a million people marched on Washington, collectively broke federal law, did it in plain sight of the world's leaders and traditional media, and yet we all barely noticed? What if political leaders didn't even see it as a political act, but instead as some sort of funny stunt?
Over the last half-decade, it's become obvious that hundreds of thousands of people around the world have chosen to ignore copyright law and to upload copyrighted material to sites like YouTube without getting permission to do so. Technically, it's illegal. Practically, it doesn't matter. Politically, it's fascinating.
In the past, when an enormous number of people chose to willfully and blatantly disobey laws that they considered unjust, we called it an act of civil disobedience. We understood the social significant of their collective demonstration, and as a society started to reckon with the implications of their actions.Today, we instead see it as an odd quirk of online culture, and outside of some eggheaded discussions about the future of intellectual property law, we largely see it as unremarkable. And that's true despite the fact that traditional political demonstrations in the context of political activism are increasingly ineffective and anachronistic.
Putting the "Party" Back in Political Party
The open culture movement that's expressed through uploading content and remixes crosses conventional political lines and eludes identification with any traditional political affiliation. The sheer number of participants dwarfs movements (or perceived movements) that have attracted much more attention, such as the tea party efforts. Any given march on Washington these days ends not in policy reform or change in any enacted laws, but in pointless and contentious debate over how many people showed up and whether they represent an actual movement. But part of the reason this new online form of political demonstration is so effective in recruiting active participants is because it's made participation as easy as taking part in the existing social networks that so many of us contribute to every day.
For generations, political activists have said that the prerequisite to getting significant participation in a movement is to make the political personal. And nothing is more personal than the entertainment and media we consume and create on our social networks every day. Remixing is an increasingly political act.
So what happens when vast numbers of social networking citizens find another law that they consider irrelevant? What if it's something more contentious or fundamental than intellectual property law? What are the implications of the increasing disconnect between the letter of the law and its practice? Sure, we've had people disregarding marijuana usage laws for decades, but that kind of disobedience was practiced behind closed doors, not in an environment that's inherently public and social.
More importantly, what are the political efforts we can catalyze if we specifically design them to be as easy to participate in as social networking is today, and if we make sure they're not aligned to the traditional structure of political parties but instead are defined by communities of interest?
I don't know the answer, but it seems increasingly likely that even the most technophobic, regressive policy makers are going to start to understand the implications of large numbers of people in loosely-defined online communities choosing to remix and reform laws on the fly without any granted authority to do so. I can't pretend to know what this development implies. What I do know is that we've seen it as a sort of odd aberration for half a decade now, but soon we'll be obligated to see it as a new political tactic to be reckoned with.
Even more fun news! Today, I'm thrilled to announce the other big project I've been working on: Activate. It's a new consultancy, founded by Michael Wolf (you know him not just as one of the most established names in tech/media consulting, but also as former President of MTV), where I'll be joining as partner.
The compelling thing to me is that we'll be advising firms at the intersection of technology, media and entertainment. You don't have to have been reading this blog very long to know that's an area which is near and dear to my heart, and getting to work with the folks who are merging those disciplines at the highest levels of business and culture is pretty exciting. It took me the better part of a decade to figure out that I'm obsessed with how culture is made, but once I had that realization, it very quickly became clear that those of us in the technology world were the ones driving the transformation of these businesses.
But right now, most of the companies that do this kind of consulting for big media or entertainment firms operate from what I'd consider a fear-based standpoint: Oh no! Technology is happening! It's going to be scary and destroy you! The old-school consultants are good at slashing costs (and jobs), but I think it's a lot more interesting to figure out where new growth and opportunities are going to come from.
Which is important, because some of what we think of as traditional media or entertainment companies will figure out how to take their past strengths and turn them into huge new businesses that work in the modern world, and that respect the way people use technology today. Those businesses are the ones we're working with at Activate.
And yep, I'm still totally committed to my work as director of Expert Labs as well. (As I write this, I'm in Washington, D.C. trying to refine our next project.) I was very fortunate that the Expert Labs project was set up from day one to give me the flexibility to put my skills to use in all of the areas that I find interesting. More importantly, I'm learning in general how to help huge institutions evolve. Whether it's media companies, government agencies, or entertainment businesses, I am truly optimistic that many can transform themselves for the technology world that we live in. In fact, I think those are the only ones that will survive.
Honestly, I feel like the community of innovators and tech minds that I've been part of has earned the right to help determine the future of everything from governance to culture. We've made tools and platforms that have made some fundamentally new things possible. So I'm gonna do everything I can to help make sure our community has a voice at the highest level of the institutions that shape our lives. And Activate is an exciting new part of that work.
For more details:
Activate.com: Not much here yet, but don't we have a kick-ass URL?
Expert Labs, ThinkTank, Gina Trapani and our Grand Challenges
A few months ago, I started as director of Expert Labs, a new independent non-profit effort with the goal of improving government by letting policy makers tap into the collective wisdom of the public. We're part of the American Association for the Advancement of Science, and because our goal could have seemed a bit nebulous I've held off on explaining the full vision of the effort until today, when we're announcing our first project, platform and project director. Here's the highlights:
We'll be collaborating with the White House in support of the Grand Challenges initiative. The President has defined a list of the biggest scientific and technological challenges facing America, as part of his Strategy for American Innovation. But they need our help, especially from those of us in the scientific and technological community: What should our highest priorities be for the biggest technological challenges of our time? What items have been omitted from the President's list of priorities? In short: If you had to pick the next project on the scale of the moon landing, or the human genome sequencing, what would you suggest? And how would you find the leadership and community that would achieve that goal? These are the questions we want to help answer.
To help get answers for these questions, Expert Labs will be sponsoring the development of a technology platform that allows policy makers and community members to ask questions across the existing social networks that exist on the web. My guideline for the technology platform was that it be free and open source, make smart use of existing technologies and APIs, have a thriving developer community, and be appropriate for use in cloud environments for easy deployment by government agencies, private industry, and even individuals. So I'm excited to announce that we've selected the ThinkTank application as our first official technology platform project at Expert Labs.
And, as you might expect since we've agreed to sponsor her application, I'm ecstatic to announce that Gina Trapani is joining Expert Labs as our Project Director for the Grand Challenges project, overseeing our technology efforts around ThinkTank and making sure that the platform is a good fit for the community of policy makers, scientists, technologists and the general public that it's designed to serve. Gina is of course the founding editor of Lifehacker and publisher of Smarterware, a best-selling author, and a co-host of This Week in Google, one of the most popular podcasts on the Internet. She's also an incredible talent and a woman of remarkable character and I couldn't be more excited to have her on the team.
Phew! That's a lot of great news. Since I announced my role at Expert Labs two and a half months ago, we've been hard at work meeting with folks across the Federal Government to find out how we could be of the most value. The truth is, when I started this project, I really only had a hunch that there was something amazing happening at the confluence of technology and government. But the months since have shown that my optimism there is well-founded, even if it is still just early days for this kind of effort.
The Startup Mindset
You see, Expert Labs sits at an interesting intersection. We are not part of the government, don't take any money from the government or any tax dollars, and don't take orders from anyone in the White House or any other part of the administration. In the early days of refining Expert Labs, I saw us as something like a "gCombinator", creating technology that serves government needs, but with a model that looks a lot more like an entrepreneurial technology incubator.
And while we're proudly independent, we've also been given a remarkable amount of access. The federal government as a whole is making an incredibly rapid evolution towards becoming more open and accessible, particularly to technologists. You can look at something like the OpenGov Tracker and see the results of this in real time. That's not to say things are ideal; Only 611 ideas for improving government have been submitted in total thus far. But I think that we can get orders of magnitude more Americans to participate in, and suggest ideas for, better governance if we make it as easy as just using Twitter or Facebook. And I think we can provide great motivations for them to do so if we show that their ideas and inspiration have direct impact on the policy decisions that are made.
This is a time of remarkable opportunity for the tech industry that I have spent my career working in. I'm just a regular guy, who was working just a few years ago as a PHP coder building content management systems. Today, I've been able to go to the White House and help make the case that a better technology platform, connected to the social networks we already use, could have the same transformative effect on policy making that it did on the world of media or business. And they were ready to listen, not just to me, but to our entire community. (I'm not saying that to name drop; In the new world of open government, things like visitor records for the White House are actually easily accessible.) I mean, hell, I got excited just knowing that my project's website got linked to from the White House blog — imagine when that's a two-way conversation for all of us!
And if you're a web programmer today, you can have a huge impact, even if you don't know the first thing about government or policy. You don't have to work for the government to work for your country. All you have to do is follow the ThinkTank project and make submissions of any code fixes or improvements that you have. Or join the mailing list and become part of the community. Or simply run the app for your own business and submit your feature requests about how it could be better suited to answering large-scale questions on various social networks. Simply by playing with new technology, participating in an open source project, and sharing what you've learned about what works in crowdsourcing ideas online, you can make a huge impact in our government's ability to listen to our ideas.
Just Getting Started
I'm incredibly excited to get started with our first official project at Expert Labs, and there are more to come in the future. Today, I hope you'll read over the Grand Challenges Request for Information from the White House and understand a bit more about what this project is about. Then you can visit the Expert Labs site (or follow @expertlabs on Twitter) and keep up to date with us as both the technology platform and the overall Grand Challenges effort progress.
I'm doing a number of presentations and public appearances over the next several weeks, here's a quick chronological overview if you'd like to meet up.
The AAAS Annual Meeting in San Diego, this week from Feb 18-20. The American Association for the Advancement of Science is the parent organization for my new project Expert Labs, so I'm extraordinarily excited to get to attend and participate in this event for the first time. Even if you're not a scientist, you can check out Family Science Days for some very cool events that are open to the public.
Sweets and Treats on March 10 in Washington, DC. Debbie Weil's great event for bringing together DC's tech and government communities looks really promising, and that was even before I knew there was gonna be free cupcakes.
South by Southwest in Austin, TX on March 12-16. I'm really looking forward to this — Break Bread for Brad in memory of Brad Graham on the night of the 12th, the KICK kickball game is coming back on the morning of Saturday the 13th (more about that soon!), I'll be returning to Battledecks (this time as a judge), and of course all the usual Austin festivities.
Fast Company's Innovation Uncensored conference, April 21. I'd be excited just to attend this one — it's even more thrilling to get to present. Take a look at the lineup of speakers and I bet you'll agree.
Gov 2.0 Expo, May 25-27. I can already tell there's been a huge shift in the conversation about how government and technology relate, and if last fall's Gov 2.0 Summit was a watershed moment, this might be even more of a milestone.
And, if you can't make it to any of those events, I'll be doing a few things online, such as this career event about looking at your job skills in the context of Last Year's Model. I'll likely be adding in a few additional events, including Chicago in early March, which I'm hoping will let me meet even more of you. Oh, and of course I'll be blogging here as well, if you really don't feel like going anywhere.
The Power of the Audience
As I write this, the Twittersphere is going through its annual love/hate paroxysms about TED. Every year, the conference seems even more an event perfectly calibrated to inflame the bloggerati: Inarguably great presentations combined with unrepentantly exclusionary structure. A humanitarian aesthetic that shrouds a blatantly classist participation model. But what most vexes outsiders (including myself; I've never attended, though I've been invited to pay 2/3 the cost to attend one of their remote TV viewings of the event) is the sense of collective delusion that seems to overcome attendees, who fawn over even the least of the talks. It's sort of like Steve Jobs' famous Reality Distortion Field, but on more of a peer-to-peer basis. I like the idea of TED, and the people who run it, and the presentations it generates — but what's up with the groupthink?
The New York Times reveals the teenage founder of random video chat phenomenonChatroulette. Though the site's still new (and controversial) enough that Wikipedia's editors have thus-far decided it's not worthy of note, it's inarguably struck a chord, with a noticeably larger impact in media-fixated NYC than tech-focused San Francisco. Though the site can definitely be seen as creepy, people also seem mesmerized and intoxicated by the idea of connecting with strangers in such a visceral, immediate way.
One of my favorite startups (naturally, part of our burgeoning NYC tech scene) is Chartbeat, which recently explained the crucial distinction that realtime analytics are not just faster analytics, they're a difference in kind from traditional static stats.
So what do these data points have in common? They're actually essentially about audience and shared experience. In the realtime web, we've focused a great deal on the latest noise. But as I said when I first wrote about the pushbutton web, what may matter most about realtime capaibilties is the user experience that's enabled. And the best use for realtime communications on the web is not to simply bring in the most recent information on a topic, but rather to make clear that others are experiencing or interacting with the same content at the same time.
There's Others Here With Us
Audiences matter. Being in an audience isn't simply about being at the same place at the same time. We act differently when we're in the presence of an audience. As audience members, we're more susceptible to being connected emotionally, expressing ourselves in uncharacteristically free ways, and forming lasting connections with the presenter, performer or content that we're experiencing.
Shy people start to sing along at concerts. Tea Party meetups start with group prayers that encourage participation from people who might otherwise be uncomfortable talking politics with strangers. Ostensibly unbiased journalists applaud at Apple keynote presentations. We are transformed when we're part of a shared experience.
Just as importantly, performers are elevated by the presence of the audience. I do a fair bit of public speaking, and I have obsessively watched a lot of the best performers of the pop culture world for my entire life. In studying their work, especially for artists who are significantly different between the studio work they do on a recording versus the live performances they do on stage, you can see a remarkable elevation of expressiveness and personality when they're facing an audience.
I've even seen it with bloggers and writers; Though blogging was often described (not inaccurately) as "theater for introverts" in its early days, a lot of bloggers and writers have matured into formidable public speakers as well. The first time I saw Malcolm Gladwell speak, almost seven years ago, he was an awkward and quiet presence, the very picture of an introverted writer. By just two years later, when he keynoted SXSW in support of the publication of Blink, he was effortlessly charming and spellbinding. Sure, he'd had a lot of practice. But it was clearly the interaction and attention of the audience that were drawing him out and raising his game.
Hello, Cleveland
When I create on the web today, I'm still completely isolated from the sense of having an audience. I have a couple of different (largely inaccurate or worthless) metrics about subscriber numbers, follower counts, or page views that I can look at to estimate the impact of something I've created. For a decade, though, I created posts on this blog with only a vague sense of people actually having a shared experience of reading these words. If I'm really lucky, a few friends might send me an instant message after publishing, and I'll know there was really someone on the other end of the line. Even long comment threads have the feel of the occasional straggler walking into a mostly-empty coffee shop for a few minutes: Cumulatively significant, but sparse and unpopulated at any particular moment.
That's a huge disconnect, and a huge opportunity. When I wrote my Facebook usernames post several months back, I had one of those rare moments where something I write resonates outside of the core tech community, and I could watch links to or mentions of the post roll in from Twitter and Facebook, in realtime. That sense that (what would eventually become) hundreds of people were all on my site at the same time was gratifying and rewarding in a way that felt, for the first time, just like the satisfaction I get when I know I've killed it onstage with a good presentation.
And we've misunderstood that motivation online for a long time. We deride searches for mentions of one's own work as "ego searches", implying there should be some sort of shame in looking for responses to our creativity. Services from YouTube to Twitter make it effortless to see what you've favorited from other people, but nearly impossible to measure or monitor who's marked your work as one of their favorites. Even if you can see that data, it's in an asynchronous, disconnected manner, instead of making clear which of those people were responding at the same time. Chartbeat mitigates this somewhat for me as a creator, but that doesn't help you as a reader. MyBlogLog, as ungainly and awkward as it's always been in its short life, would show some avatars for site visitors when they were on the same page. People embed live chat windows on their site so that visitors can talk to each other. But the essential experience of being in an audience isn't actually of audience members talking to one another. And while I've certainly been at movies where an excited and responsive audience improved the experience, I've absolutely never wondered if I could see a list of everyone in the audience with me, sorted by the order in which they found their seats.
Standing Room
Today's rough approximations of the right experience still fundamentally deny us the opportunity to be part of an audience together when we see something we love. Sure, we've all sent a YouTube link to a friend over instant messenger so that it could be enjoyed simultaneously, perhaps even to someone in the same room. But we're never allowed to just "look around" and see who else is there at the same time. It's part of the reason that very, very few web experiences can grab us and truly move us the way that media like movies and songs and television do.
There's a big opportunity here. I'm a better writer, blogger and thinker when I know there's an audience. (If I could see your eyes glazing over, maybe I would have edited this into a shorter post!) It could be transformative to our experience as creators if we could actually have the feeling of a real audience when we're sharing our thoughts to the world, instead of the arbitrary counts that the people selling advertising on the web have been referring to as an "audience" all of these years.
Most importantly, those of us who've had our lives transformed by the web, or who have had emotional and meaningful experiences of common connection through the Internet could have a way of sharing those experiences with a far broader audience that's familiar with the traditional behavior of audiences. I can't wait to see what becomes the equivalent of a standing ovation.
I'm going to be offline for a little while (some would say that last rant of mine was a sign I should have gone offline a bit sooner) so I thought I'd leave you with some good sites to check out that you may not have been enjoying.
Dan C's Lost Garden. Though nominally about gaming (particularly Flash gaming), it's among the most consistently thought-provoking tech-oriented blogs that I read. Every idea of his is one I want to steal, and nothing exemplifies that pattern more than his recent work on Ribbon Hero.
Sleevage. Album covers, one at a time. Single-topic blogs run by passionate individuals (instead of paid blog barfers) are still among the best sites on the web. This one is a perfect example.
Modcult. Though I am Jeb's number one fanboy, I will begrudgingly concede that all of the authors of this venerable group blog are awesome curators.
Mixtape Maestro. Probably the single music blog that comes closest to my own fixations on the production end of pop; I miss its erstwhile spinoff 90s R&B Junkie (the archive is still online), but this is one of those few sites where I try to read every single post and feel let down if I miss one.
RC3. Rafe Colburn is living proof that some folks really hone their craft at blogging after being at it for a decade.
And then, two newcomers, from a genre I'm dubbing "Under a Rock" blogs:
Hobbited, where my friend Natalie is mirthfully blogging her way through her first-ever reading of Tolkien's classic The Hobbit.
Tellywonk, where Anna Pickard is documenting her first viewing of Lost, by trudging through every episode.
Both of those last two blogs touch on a recurrent fixation of mine, the myth of the cultural canon. No matter how ostensibly ubiquitous or universal a particular work of art is, no matter how frequently it's referenced or alluded to in culture, the majority of people have probably never seen it.
My friend Meg told me the other night that, as an early-to-bed morning person, she's never really seen an episode of a late night talk show. I would love to read a blog of her watching an episode of each of the major shows, documenting the things that seem remarkable or bizarre. I've toyed with the idea of blogging my way through playing Beatles Rock Band, since I've never actually listened to any Beatles album all the way through and only know their work from its pop culture ubiquity. This, despite my love of pop music in general. (I first heard "Eleanor Rigby" from Aretha Franklin, "Norwegian Wood" from P.M. Dawn, "We Can Work It Out" from Stevie Wonder, and probably have more examples like that than I can count.)
Inevitably, people react to that revelation from me with something between shock and dismay, often evolving into disgust or revulsion. But it doesn't much bother me; There's lots of culture that I haven't gotten around to participating in. I've never been to an opera, either.
What I'm curious about, though, is how people who are fairly culturally literate and very well-educated respond to works that pervade culture. Under a Rock blogs are great for showing how ideas percolate through the media world, and how those ideas are imperfectly absorbed.
So, confess: What have you never seen, heard, or read?
All Over The Web
Just a quick roundup of some recent conversations I've been having around the web:
Fast Company interviewed me about applying the lessons of Web 2.0 to government. I'm always happy when I can mention my love of New York City and pop music while also talking about the importance of using the web for civic purposes. They also published this Rennio Maifredi photo of me, which my Twitter friends agree is very creepy.
Reddit did an "Ask Me Anything" thread where people could ask me whatever they want. I answered a bunch of the questions in text, and a video of me answering the most popular ones will be up shortly.
And CNN just published an editorial of mine where I talk about the importance of the decentralized web. What if we had all decided to rely on AOL Keywords for 911 emergency services?
Hopefully you're not all too sick of me after that; I'll try to share some of the recent presentations I've made at events I've been speaking at recently as well — I'm very excited about a lot of the conversations I've gotten to participate in lately.
As my follower count crept past 300,000 a number of you responded with suggestions of what information you were curious about, submitting your ideas by using the #sulidea hashtag.
Before I reveal who's won an Amazon certificate, here's a list of all of the suggestions that I found, sorted by Twitter user name.
casey: I wonder what the rate of increase of followers for those on the SUL is—i.e., is there a spike on day 1 and then dropping each day?
Alex Guest: How many followers from pre-list days are still following and what is the decay curve?
Carol Hagen: Interested in stats on new followers & avg number of tweets from lists vs those of engaged followers
Nate: How many of your SUL followers have less than 10 total tweets after their first three months on twitter?
Chris F. Nicholson: How many of your followers follow a given number of your other followers (how deep is the network between them, and thus, you)
K Kishbaugh: How many new followers from suggested list R active on Twitter beyond a couple weeks? Do many never find the value in Twitter?
Michael Kubler: Are new followers not clicking/retweeting cause they are new? Do old followers used to Twitter do better?
Leonard: analysis of followers quantifying activity percentiles / current activity also would be pretty useful
Leonard: clickthroughs need to be corrected against users' avg tweeetstream rate/following and maybe age/activity
Mark J. Hulme: Improve twttr's suggested user list - make retweets a part of the calculus 4 inclusion - how so w/out encouraging gaming?
Matthew Glidden: Are you tempted to prune away non-engaged followers?
mizminh: How many people who don't follow arrived at via Seth Godin's blog about Anil Dash's blog?
natasha: What are the sort of followers you get. any specific country wins? many bots? young/old/new people?
Nathaniel McNamara: I would like to know which people are clicking on my own links (who are they?) -
Nathaniel McNamara: I would love to know the number of clicks generated from links posted by accounts on the SUL in the past week/month/year
Laura Conaway: Did the account's # of tweets go up after it got added to Suggested User List?
Peter J. Hester: Anil, Of your total followers, what other SUL accts are they following? Is there a trend?
Peter J. Hester: does the frequency of interaction from new followers increase/decrease over time?
Peter J. Hester: Of the available SUL accts., how many does the avg new user elect to follow when signing up. What criteria is used to decide?
Sharon Henry: Breakdown of your followers: Those following fewer than 50,100...being 1 of 50 greater influencer than being 1 of 10,000
Greg: I wonder what the distribution is for the amount of time people follow twitter celebrities before they they stop following them
esteban contreras: wonder what the real / lessthanreal follwr ratio is for companies vs. individuals & celebs on twtr suggester user list
Jason Staten: A ratio of tweets with a link to RTs of them might be an interesting statistic. Most linked tweets want to be shared.
Tyler Crowley: how much has the average CTR of your bit.ly links or twitpics changed since joining the list?
Carol Doane: What I want to know (it's not about users) when THE LIST goes away & followers drop, what is psychological affect on you?
George Eapen: Check out how many of your followers actually use the hashtag to figure out how many real users you have.
Alex Rose: Let's see a graph showing # of RTs over time by user. Anyone regularly RT on a weekend at least 5 days later?
Vojt?ch Turek: Idea, pt1#2: It might be interesting to have a chart of your most active followers (highest RT/reply/mention count) [...]
Vojt?ch Turek: Idea, pt2#2: [...] from back then & whether it's changed significantly after you'd been added to the Suggested User List.
David Wertheimer: Idea 1: I wonder how many silent followers are newbie abandonments. Good read: http://bit.ly/sulidea
David Wertheimer: Idea 2 re : maybe isn't seeing responses because his tweet volume is neither as offensive or as obvious as celebrities'
David Wertheimer: And idea 3 (best) re : I suspect the average Suggested User user is simply more likely to be a lurker than 's usual coterie
J. D. J.: I'd be interested in knowing what percentage of your followers have ever @mentioned you.
Since there were lots of good ideas, I've decided to give out two awards, one for the most universal, and one for the most thought-provoking.
Nate Chenenko asked, "How many of your SUL followers have less than 10 total tweets after their first three months on twitter?" I think this is the fundamental question. Are people who follow someone on the suggested user list interested in posting to Twitter at all? Is it just a passive experience for them? Ricardo Guerrero formulated this in terms of time period of activity instead of tweet count, which is similar but slightly less indicative, when he asked, "I'd be keen to know how many of your followers haven't updated at all in the last 1-3 months. Also % who've replied/RTed." And Jay Neff phrased it as, "What % of follows gained are actively tweeting? Would love to see a breakdown of active to inactive over x amount of time" So Ricardo and Jay get Honorable Mentions, along with a few others who asked similar questions, while Nate gets a prize.
And Sharon Henry gets a prize for articulating another common theme in an interesting way: "Breakdown of your followers: Those following fewer than 50,100...being 1 of 50 greater influencer than being 1 of 10,000 ". That seems eminently doable, so I really found it appealing. In short, what I'm hoping for is two core bits of data from which we can extrapolate a lot of meaning:
How many followers do each of my followers have?
How many tweets do each of my followers have, and when was the last time they were active?
Those are pretty straightforward requests to make with the Twitter API. So, there's still a chance to win another prize. If you're a coder, commit either of those queries as a feature built onto ThinkTank and I'll send you a 500 GB portable hard drive.
Thanks to everybody who participated! I'll try to make the data from these requests available as soon as possible, and the few questions above that I have answers to will be replied to shortly.
Remembering Brad L. Graham
I don't believe in life insurance. When I die, I want it to be a bad day for everybody. - Brad L. Graham, February 2002
My friend Brad L. Graham died unexpectedly last week, at only 41. It's hard to sum up someone so loved in a few words, but I wanted to say a bit about him because he had a profound effect on my life and on the lives of many of my closest friends. In short, Brad showed us that when we do something creative, we're not just making art, we're making connections to a real community.
Truth is, Brad and I didn't spend all that much time together in person. Over the past decade, we'd catch up once or twice a year, grabbing coffee one afternoon in San Francisco, stumbling into a diner in Austin at three in the morning, going out to a nice dinner in Chicago on a blustery evening. But he was a constant presence in my life online, and influenced the way that so many of us fundamentally view the online world.
If you look at Brad's venerable blog The Bradlands, it looks pretty ordinary. Sure, it's been around longer than most (Brad started blogging in 1998). And his April Fool's Day jokes were actually funny, unlike everybody else's (Brad inspired one of my most popular blog posts ever, where I ranted that nobody else was up to his standard), but it's otherwise unremarkable. Why, then, have dozens of people professed their love and respect for this man's work online in threads like this one on MetaFilter?
I think the explanation finds its roots in Brad's work in promoting theater. Brad was an eloquent and passionate advocate for the Rep in St. Louis, and a fixture in his corner of the city's gay community. In both cases, he was able to be a powerful voice because he was so charming and persuasive in his demeanor. I already miss his ability to call almost anybody "Darlin'" while being simultaneously complete sincere and totally cheeky.
But we're used to treating the folks who work at a theater as a community. What was surprising was that Brad brought this same sensibility to the early days of the personal web. Before the term "blog" was even coined, the distinguishing feature of the sites that a few of us were publishing was that these were made by real people, individuals with voices who had something to express. Yet the conventional wisdom was that the medium we were working in, the world we were living in, was somehow not real.
This manifested itself in a lot of different ways. Early reports on blogs would say "why are these people wasting their time shouting into the void? Who reads these things?" Even though I was participating in it myself, I would still be combative and antagonistic on my site at times, because I didn't always see the readers or other bloggers I interacted with as "real" people.
In that era, before meetups and tweetups and mass political movements organized by bloggers, Brad recognized that not only were there real humans interacting on these sites, but that all of us who shared our thoughts online were part of a creative community every bit as legitimate and unifying as his work in theater.
And the evidence of that belief is everywhere. Over the days since Brad's passing, amidst the heartbreak, I've seen literally dozens of people say, in their own words, "Brad was the first online person I ever met in real life." In cities all over the world, in one-on-one meetups to cities he'd never visited, or in his legendarily inclusive hundreds-strong Break Bread with Brad annual drinkfest in Austin, Brad brought together people who hadn't yet realized how they had made real, significant relationships online. A while back, I'd written about communities of creators, showing a small group of folks who were enormously and disproportionately influential in making the web more personal and social. Brad was there that night and always, more than holding his own as a peer to some of the most successful entrepreneurs on the web, even though he wasn't even really a geek. He just liked the medium because it let him connect with other people.
With Brad, it was a matter of course that you were going to catch up if you were close by. While the rest of us always intend to get around to inviting some people to go out, Brad was disciplined about it, actually cajoling and prodding you until you showed up. He invested in connection, and for me it never showed more than just a few weeks after 9/11. He had known I was feeling really alone, with few friends here in New York to comfort me after the attacks, so he flew up while others were still too terrified to get on an airplane, while the rubble pile was still burning, and spent half a day just hanging out and being there for me. And this was essentially only the second time we'd met in person, after South by Southwest where he'd invited me to my first Break Bread with Brad six months earlier. I was no less of a real friend in need of support, simply because 99% of our conversations happened online over instant messenger or in a private web community we belonged to.
By the next spring, when a few of us got caught out in the rain during an unusually powerful sudden thunderstorm in Austin, Brad's powers to bring people together were in full flower. The handful of us who'd been soggily sprinting through the Texas night bonded over our plight when we finally found refuge at a diner . I had noted it at the time as having learned the lesson that rain could make brothers out of strangers, and I didn't realize quite how true that was. I'm still in touch from time to time with the guys who were at the table. One of them even stood with me at my wedding just a few years later, as I did later at his. At that wedding, the bride and groom had first met in person at Break Bread with Brad.
[Update: James McNally, who was there in the rain with us that night, and also here in the comments, pointed out that I had incorrectly placed that evening in 2002, instead of 2001 when it actually happened. Despite my memory getting foggy over the years, the error just shows that I'd already felt like Brad was an old friend just a few days after we'd met in person for the first time.]
Best of all, Brad used these formidable powers for good. At at time when the Day Without Art was at its peak of influence about the AIDS pandemic, Brad organized an online counterpart in Day Without Weblogs (the term "blogs" wasn't yet in popular usage) beginning in 1999, eventually evolving it into Link & Think. While many of us believed in the cause, there was a lot of skepticism about comparing blogs to art, and even more about whether a community of bloggers could raise awareness about anything. The fear, essentially, was that a day without weblogs would be just like every other day.
Instead, many media outlets' very first time ever mentioning the new medium was in their coverage of the campaign. We'd demonstrated an ability to collectively get a message out, and I'm still proud that one of the seminal demonstrations of activism in the medium was one rooted in compassion. Brad grounded our first demonstration of our collective power with his expression of conscience.
And Brad's example changed my life as well. These days, I very rarely get into pissing contests with other bloggers or butt heads with commenters on other sites. Sure, some of it is having grown up and become a bit more of an adult. But most of it is due to the example of Brad (and those whom I met through him) showing me that there were real people on the other end of the line. It's one of a million little ways in which he made my life better. Some understandably want to assign a neat headline to his work, by saying "Brad coined the word blogosphere". Though it's technically true, the coinage was an act he thought was a bit silly, more worthy of blame than credit. He generously felt the recognition could easily go instead to those who wanted it.
Rather, Brad's work was altogether more messy and funny and human and passionate and complicated, just like the man himself. I can offer no more succinct summation of the man than that he was a good man and a good friend, profoundly funny and profoundly kind. I cared about him and he introduced me to many more whom I care about. Brad made my work more meaningful and was there for me when I needed him. I can't believe he's gone.
You can donate to Brad's beloved Repertory Theater in his memory, thanks to Judith's efforts. A number of us are working to preserve his work online, and Matt Haughey has taken care of keeping his web hosting going. There will be a Break Bread for Brad at South by Southwest this year. While details of venue and timing are still being settled (I'll update this post when they're decided), I hope all of us who had our lives better by knowing him can gather together to raise a drink in his honor.
There are many, many tributes to Brad across the web, most linked in that MetaFilter thread. A few of note to me:
Recently, it has come to our attention here at Dashes.com that the lyrics to Britney Spears' single "3" contain some inaccuracies and poor decisions, and as a result, we'd like to offer some corrections, as a public service to Britney and her staff.
First, the chorus:
1, 2, 3
Not only you and me
Got one eighty degrees
And I'm caught in between
Countin'
1, 2, 3
Peter, Paul & Mary
Gettin' down with 3P
Everybody loves [labored moan]
Countin'
The "180 degrees" reference here works, and we congratulate you for avoiding any unfortunate "69" references in a such a number-heavy song. However, the "Peter, Paul & Mary" reference here is inexplicable. First of all, we're pretty sure one of them is dead, and the other two are close, and while the "threesome" concept tests well with focus groups (and is great for ranking in Google!), "necrophiliac threesome" is considerably less popular. Also, this reference is to a group that peaked roughly 40 years ago, putting the target demographic somewhere in their late 50s or early 60s — not the image you should be shooting for. Finally, while information about Woodstock is hard to find on wikipedia due to it having happened about half a century ago, we're pretty sure that two of these folks are dudes and that is, again, something that doesn't test as well.
In lieu of the "Peter, Paul & Mary" lyric, here are some suggested replacements featuring younger protagonists and the preferred MFF arrangement suggested by public polling and a series of very systematic Twitter searches:
"1, 2, 3 / Jack, Janet, Chrissy" - While a "Three's Company" mention also predates your lifespan, the target audience could be as young as 35, and therefore constitutes an acceptable target demographic for a pop culture reference.
"1, 2, 3, / Velma, Scoob, Daphne" - Everybody loves Scooby-Doo. Some might balk at the inclusion of a dog in the lineup here, but this is still clearly better than an old person, let alone an old dead folk singer.
The "3P" is a nice touch, and a video game reference feels a lot more contemporary than the rest of this Summer of Love stuff. "1, 2, 3 / Princess, Toad, Luigi", perhaps? Continuing on, we have the pre-chorus after the initial verses:
Are - you in
Livin' in sin is the new thing (yeah)
Are - you in
I am countin'!
Here we have another factual error. "Livin' in sin" is not the new thing, unless this is supposed to be a period piece taking place around the same Woodstock timeframe in which Peter, Paul & Mary is a relevant reference. More importantly, the concept of "living in sin" seems to have been completely abandoned by our culture at some point around the turn of the millennium -- is this another one of those Louisiana things? Let's get some folks to tight this part up. It's sort of nonsensical for a twice-divorced single mother of two to be using this line as seduction, at any rate.
Finally, the bridge:
What we do is innocent
Just for fun and nothin' meant
If you don't like the company
Let's just do it you and me
You and me...
Or three....
Or four....
- On the floor!
We have a conceptual issue here. While the beat to this song is certainly insistent, and the bridge has a profoundly conventional boom-chik backing track, the outro/vamp that follows this section actually don't use a traditional four-on-the-floor drum pattern. Clearly, this is just sloppy songwriting (or this was written to a different track, while the final one was still rendering in ProTools), so it's not your fault this wasn't caught in pre-production. I suggest letting this slide, as it'll be a useful snippet of "vocals" for producers to use on the club remixes. No harm, no foul!
Thanks again for your time, and we hope you've found this bit of fact-checking useful. We appreciate the effort you've made to simplify your lyrics to ease our task (although we do miss the good old days, where "Womanizer" only used 6 different words and no complete sentences!) and hope all's well with you.
I'm here at the Web 2.0 Expo in NYC today, my first big tech industry conference in a long time, where I'm also excitedly getting ready for my keynote tomorrow.
But one of the things I'm most proud of is that has something of a valedictory feel to it, as we note that many of the best, most interesting, most subversive and disruptive startups in the world are based here. From Foursquare to Hunch, Kickstarter to Square, Etsy to the newly-funded 20×200 (they're hiring!). That's not counting the dozens of tech-based media businesses that have spring up in the wake of Gawker and Huffington Post. And best of all, I think many of them have been influenced by the seminal NYC Web 2.0 startup, Meetup, which not only helps knit our startup community together, but introduced many of the elements of social responsibility and an old-fashioned We Make Money business model that distinguish New York startups from those in Silicon Valley and elsewhere.
(Update: To my chagrin, I forgot Outside.in, another great NYC startup that I've found inspiring. I'm sure there are more omissions, too, but I'll add 'em as they come to me.)
New York City startups are as likely to be focused on the arts and crafts as on the bits and bytes, to be influenced by our unparalleled culture as by the latest browser features, and informed by the dynamic interaction of different social groups and classes that's unavoidable in our city, but uncommon in Silicon Valley. Best of all, the support for these efforts can come from investors and supporters that are outside of the groupthink that many West Coast VC firms suffer from. When I lived in San Francisco, it was easy to spend days at a time only interacting with other web geeks; In New York, fortunately, that's impossible.
Am I biased? Sure. But are there half a dozen startups anywhere in the world as interesting and full of potential as these new NYC efforts? Isn't it exciting that these are all built around the full potential of the open web, instead of merely trying to be land grabs within the walled gardens of closed platforms? I'm more optimistic about the environment and opportunity for starting new ventures than I've been in ages, and for me the fundamental reasons why are demonstrated best by startups that could only happen in New York City.
Plus, we have bagels. Delicious bagels.
The Web in Danger
I love the Internet. I love lots of things that are on the Internet. I have less love for things that want to undermine the Internet.
If you've followed my thinking about Web 2.0 from the beginning, you know that I believe we are engaged in a long term project to build an internet operating system. In my talks over the years, I've argued that there are two models of operating system, which I have characterized as "One Ring to Rule Them All" and "Small Pieces Loosely Joined," with the latter represented by a routing map of the Internet.
The first is the winner-takes-all world that we saw with Microsoft Windows on the PC, a world that promises simplicity and ease of use, but ends up diminishing user and developer choice as the operating system provider.
The second is an operating system that works like the Internet itself, like the web, and like open source operating systems like Linux: a world that is admittedly less polished, less controlled, but one that is profoundly generative of new innovations because anyone can bring new ideas to the market without having to ask permission of anyone.
I've outlined a few of the ways that big players like Facebook, Apple, and News Corp are potentially breaking the "small pieces loosely joined" model of the Internet. But perhaps most threatening of all are the natural monopolies created by Web 2.0 network effects.
One of the points I've made repeatedly about Web 2.0 is that it is the design of systems that get better the more people use them, and that over time, such systems have a natural tendency towards monopoly.
And so we've grown used to a world with one dominant search engine, one dominant online encyclopedia, one dominant online retailer, one dominant auction site, one dominant online classified site, and we've been readying ourselves for one dominant social network.
Missing in action is credit to what goes below private platforms like Twitter, MySpace and Facebook — namely the Net, the Web, and the growing portfolio of standards that comprise the deep infrastructure, the geology, that makes social media (and everything else they support) possible.
Look at four other social things you can do on the Net (along with the standards and protocols that support them): email (SMTP, POP3, IMAP, MIME); blogging (HTTP, XML, RSS, Atom); podcasting (RSS); and instant messaging (IRC, XMPP, SIP/SIMPLE). Unlike private social media platforms, these are NEA: Nobody owns them, Everybody can use them and Anybody can improve them. That’s what makes them infrastructural and generative. (Even in cases where protocols were owned, such as by Dave Winer with RSS, efforts were made to remove ownership as an issue.)
Tweeting today is in many ways like instant messaging was when the only way you could do it was with AOL, Microsoft, Yahoo, Apple and ICQ. All were silos, with little if any interoperabiity. Some still are.
The rise of the “app store mentality” is a direct attack on the web, and on the very nature of free discovery and choice built upon URL-based hyperlinks. By depriving us the ability to pick and choose which “stores” we shop from on these devices — we’re empowering a new breed of middle men and ceding to them monopoly control over our digital experience. The architecture of the web was intended to withstand such threats — but that all changes when the hardware makers get into the content business! Even though developers are beginning to see the dark side of this faustian bargain, the momentum is huge — and big business smells money.
By removing our ability to navigate, choose, and share freely — these app stores are exchanging our freedom for a promise that they’ll keep us safe, give us everything we need, and do all the choosing of what’s “good enough” for us — all starting at ninety-nine cents a hit.
We cannot say we were not warned. We will not be able to say "nobody saw this coming". It's clear that, even those who are privileged by access and wealth and the ability to amplify their own voices have anticipated that we'll all be disenfranchised by the private companies that own and control our networks of communication. And yet, most of our effort and ambition in the technology industry are not going towards building for the open web. Most communities that are disadvantaged are still trying to win on networks that they don't own and will never control. Most of us are still cheering when the most powerful voices in culture and society embrace closed networks, instead of properly criticizing them for doing so.
I am still optimistic; Apple's control over smartphone usage with the iPhone today is but a sliver compared to AOL's enormous control over Internet access a decade ago, and AOL still eventually crumbled in the face of open standards. But the web's victory over the proprietary networks that have been built on top of it is not inevitable — it's going to take lots of hard work. And right now, it's not just the attention that's disproportionately lavished on proprietary platforms that want to undermine the open web, it's the money too. We'll have to turn those strengths into weaknesses if we're going to undo the trend towards disempowerment and centralization that's going on right now.
This, for me, is a social issue, a cultural issue, and a political issue, not just a technological issue. Perhaps we need to speak of it that way more often, to make the stakes clear.
Twitter, Outlines, Lists, Directories, Y!ou
Humans create the web, but we've largely abdicated the act of organizing web content to software. That could change.
Twitter this week made its new Lists feature broadly available. As they've been described, Lists, allow you to enumerate a collection of some of the Twitter accounts that you follow, and then easily read updates from just those accounts. Others can view your lists, and choose to subscribe to them as well. But Lists are also available for other applications to use, modify and share. Looked at from a slightly different perspective, this means Lists are a way to tag an arbitrary set of realtime web feeds. You could look at the lists that I've been added to as a set of tags describing my Twitter feed.
Much of the precedent for the idea of sharing (non-realtime) feeds comes from the world of outlining, and in particular Dave Winer's work here in creating OPML. Though it was designed to generically exchange outlines, OPML is the most popular format today for sharing arbitrary lists of feeds. (The computer science folks balk at some of the technical aspects of OPML but it's a bit like Churchill's comments on democracy — it's the worst format, except for all of the other alternatives.) What's interesting about having an established format for exchanging feeds is that there doesn't really need to be any changes in order for the format to accommodate realtime feeds like Twitter accounts. In fact, a few weeks ago, I moved about 150 the noisier, less pressing Twitter accounts I follow into Google Reader, by exporting them as an OPML file. Twitter became more pleasant to use, and I could still keep up with all of those folks by dipping into my feed reader whenever I want to.
Lists have a few traits that make them more interesting than they seem; we can think of these as the Laws of Lists. First, you have to be signed in to Twitter with a valid account in order to create them. (This seems obvious, but it's important.) Second, by adding a Twitter accounts to a list that you create, you follow that user's updates, at least while viewing that list. This combination of authentication and requirement of relationship is a very good recipe for reducing spam.
One of the earliest hopes for organizing web information was the human-edited directory. Efforts like the Open Directory Project still exist, but the model focused a lot on having defined editors for topics and a hierarchy of who could edit the site. That's a stark contrast to the default-open editing permissions of projects like Wikipedia, and is probably the most significant difference between the "human-edited" and "user-generated" eras of the web — we've always had people contributing content, the difference was in how much we trust them. Similarly, more outline-focused directories of content emerged, like Halley Suitt's Top Ten Sources, which is now defunct, but was based upon the idea of curated lists of feeds by topic. In each case, trying to scale a team of editors to keep up with the rate of growth in new sites on the web has been a losing cause. But we've seen sites like Delicious demonstrate the value of tagging individual pages or posts on a site — a new generation of directories could demonstrate the value of tagging entire streams of posts, or as we call them, feeds.
Of course, you can't talk about directories and lists on the web without talking about Yahoo. Yahoo's original sin was in trying to create a human-edited directory of the web, and before they unfortunately achieved their goal of becoming the only successful web portal, the directory was Yahoo's signature element. (Until recently, Yahoo had maintained a page with the directory in a format resembling its original state, but even that is basically a blog now.) Instead of embracing authentication and relationships to prevent spam submissions from overwhelming the site, Yahoo leaned heavily towards requiring payment for inclusion of companies in the directory, limiting its utility. Human edited directories became mostly a footnote in both Yahoo's, and the web's history.
That fundamental history of being made by humans is some part of Yahoo is trying to evoke with its Y!ou and Yahoo campaign. But of course, it's a pretty good sign that a campaign isn't going to hit its mark when a completely unknown brand like HTC can launch virtually the same campaign as a household name like Yahoo, yet both companies think their message is going to resonate.
The truth is, if Yahoo wanted to help people reimagine the web stalwart at its best, they would do well to look to their roots in a human-edited or user-generated directory. Thinking of Yahoo at its peak of influence a decade ago, it becomes clear that instead of trying to insert their ubiquitous exclamation point into you, Yahoo should look at the story of The Matrix. I don't know if the brothers Warner or Wachowski would be inclined to license the property, but the only way to truly resonate with people in a narrative of Yahoo vs. Google is by adopting this theme: Man vs. Machine.
Just as in the Matrix the humans had originally created the machines that undermined them, to some large degree, Yahoo begat Google. And Yahoo would do well to suggest that the most human way for the web to evolve is if we all work together to organize it ourselves — a mission that happens to fit in well with Yahoo's largely-mishandled acquisitions of Flickr and Delicious. I'm not sure that the marketing folks at Yahoo are going to embrace that narrative, but an interesting opportunity definitely exists around the larger concept.
We all have the ability to create and exchange curated collections of feeds, using hubs like Twitter's Lists as connection points. We can extract the descriptions from those collections to form tag clouds about individual feeds. If we want to embrace hierarchy, we can organize the collections into a hierarchy by inheriting the category structure of sites like Wikipedia. If we're worried about spammers, we can now use widely-available systems of authentication and defined relationships to define who has the authority to create lists in a particular context. And of course, the ability to aggregate all of the distributed content from a defined set of feeds in realtime has now been commoditized, where i would have been exorbitantly expensive a decade ago.
In short, we can learn from Twitter's Lists to resurrect one of the web's original ways of organizing itself: Human-curated directories. We're used to exploring photographs or individual web pages by clicking on tags that were assigned by the creators or their community, and it will be just as valuable and useful to be able to explore entire feeds the same way. Open formats and APIs for exchanging this data already exist, so I can't wait to see a few enterprising hackers build the tools that let us revisit the idea of web directories. I love computers and robots, but I love humans even more, and I think we can do a pretty good job of guiding each other to the most interesting feeds around.
How to run Windows 7 under Mac OS X 10.6 for free
Update: Since this post got a lot more readers than I expected, it's become clear to me that the title was unintentionally vague. I thought it's amazing that a technology I still think of as fairly advanced, virtualizing operating systems on the desktop, has become commoditized enough that free, open source tools are very mature. When I said "for free" here, I meant that virtualization is available at no cost, not that Microsoft's giving Windows licenses away for free. Sorry for assuming that was obvious!
Pardon the uncharacteristically nerdy post, but I thought I'd write up a handy way I'd found to run Windows 7 in a seamlessly-integrated virtual machine under Mac OS X 10.6. I started with these basic components:
If you're like a lot of geeks that I know, you have a Mac as your main machine, but often need to drop into Windows to check things like browser compatibility or to use some particular Windows applications. I happen to just really like Windows 7 (it's on par with Mac OS overall for me, with some parts being better, such as the Windows Taskbar being much better than the Mac's Dock, and of course some parts being worse.) Some of these instructions may be obvious, but I hadn't seen a writeup anywhere, so here goes.
Here's what you'll need to do:
Install Windows 7 under Boot Camp, following the normal instructions. All of the Vista drivers for Boot Camp worked fine for me, and the install was actually pretty quick.
Download and install VirtualBox. This is an open source virtualization system that runs on Mac OS, a lot like Parallels Desktop or VMWare Fusion, but available for free.
The tricky part: You'll need to do a little bit of geeky stuff. First, eject the Windows boot camp disk in Finder. (It's usually called "Untitled".) Then, launch Terminal so you can enter two commands.
Start up VirtualBox, make a new Windows 7 machine, and browse to win7raw.vmdk in your home directory to choose the virtual hard drive for the machine. Your Windows install should boot up. It'll fuss for a little while as it installs new drivers.
Once that's done, you can optionally install the VirtualBox Guest Additions software to let your Windows install completely integrate with your Mac OS X environment.
While it's not quite as seamless as some of the paid alternatives out there, I've found it was very easy to do (under an hour total, and only 15 minutes or so if you already have Windows installed), works very well, and is speedy enough to use regularly.
As always, your mileage may vary, and comments or corrections or feedback are welcome. I was too lazy to do screenshots of the whole process, but if you want to turn this into a complete gadget blog-worthy writeup, I'll be happy to link to it. If you really liked this how-to, you can buy WIndows 7 from Amazon and I'll make a few bucks.
Communities of Creators
Last week, I found this picture of a group dinner at Guero's restaurant in Austin, TX, taken during South by Southwest in 2002.
At the time, most of us at the table knew each other primarily through the web and through the then-nascent blogging community. But in the seven and a half years since then, many of us have gone on to become entrepreneurs or creators, launching dozens of companies and products. I'm still collecting names and companies in the comments on Flickr, but just a cursory glance shows founders from Blogger, Six Apart, Adaptive Path, Flickr, Gawker, Twitter and more.
I point this out not (just) to name drop — you can click through to the Flickr image to see notes about who was there, read what they've done, or add your own annotations. But I also wanted to highlight one of the most important resources that creative people need to truly succeed: A community of peers.
In the business world, and especially in the technology industry, we focus a lot on the functional requirements of raising money, or on the technical requirements of having certain features or technological capabilities. What I've found, though, is that being part of an active, ambitious, supportive and diverse community of peers is just as valuable, if not more so, than any of the more prosaic prerequisites for success. That's even true in this photo — some of the people whom I met in person for the first time that night or that weekend have gone on to become among my closest friends, the biggest supporters of my work, and have ventured their formidable social capital to support my career. An even more diverse community of others whom I met at similar dinners or other events have played a similar role as well. Yet, at the time this photo was taken, I don't think any of these people had ever taken venture capital money for any project they'd ever done — everyone here had bootstrapped their way to the table.
So, it's easy to focus on the money or the little technological accomplishments, but I am glad I found these old pictures as a nice reminder that we should set aside time for a great meal with smart friends every once in a while. If it's not enough enticement that you're just having a good time, you can also justify it as one of the most worthwhile investments you can make in your future success.
DRM and Friends
This one's been kicking around in my head for a while, and maybe you can all help me understand it. With any contemporary social networking site, I can control who has access to the things I share, and I can update or change or revoke the relationships that enable that access at any time.
For example, I can share a photo on Flickr with just my friends, or a post on Vox with just my family, or display my profile on Facebook to just my contacts. And then, if somebody ceases to be my friend, I can change their status and they no longer have access to that information. It's a unliateral, technologically enforced restriction, and circumventing the restriction would be tantamount to hacking and likely to get you banned from any of these services.
So, with all of that being said, how are privacy settings on social networks different than DRM restrictions placed on media content files from companies? Is it because I'm not a corporation? Is it because the DRM technology is provided by Flickr or Facebook instead of by Apple's iTunes or Microsoft's WIndows Media? Is it because I only (theoretically) grant permissions to dozens or hundreds of people, instead of millions?
This is a genuine question, because it's something I'm not sure I know how to articulate. I can certainly identify the difference in intent, but I am not sure I can explain the difference in definition. Feel free to comment here, or post a link or reply to @anildash on Twitter and I'll collect the best explanations I get.
Created by Carsten Höller, the room is a remarkable art installation that also happens to be a complete room suite that you can stay in for a night, letting us live the dream of camping out in the museum and sneaking out among the exhibits while it's closed.
I had no inkling of the plan, just being told by my wife when to be ready to go out. Adding to the surreality, the BBC was there to greet us, filming our entrance and initial encounter with the exhibit for their video segment.
I had been inclined to write a Yelp-style review of the stay ("The continental breakfast served in the morning was serviceable, but our room didn't even have a television!"), but since the Revolving Hotel Room is sold out, it seemed as if that would be unnecessary. As it turns out, the signature revolving motions of the platforms that hold the furniture in the room are barely noticeable once you're asleep, though when you're awake it's very easy to observe how quickly you're moving. In fact, that only thing that might have kept the night from being restful was the noise generated by the other exhibit pieces, echoing through the giant open rotunda of the building. But we had a friendly attendant/guide/security guard who, after escorting us through a personal tour of all the exhibits, graciously turned off all the artworks that used bright lights or loud sounds.
Right when we returned from our stay in the room, Alaina posted a brief writeup as well as a photo set on Flickr including some images and video from our vantage point staying in the room. Since our stay was only the third night the room was open, not many reviews or images of the exhibit had filtered out, so we inspired quite a few follow-up stories, from Gothamist's salacious take to Art21's more analytical look. Art21 also hints at the part of the experience that perhaps lingers with me most: The other exhibits we took in.
Being able to see the museum uncrowded and unhurried by the usual crush of competing patrons was the most memorable and distinctive part of the experience. We could take our time, really appreciate the works (as well as the incredible architecture of one of NYC's signature buildings), and form our opinions without the awareness of thousands of people around us. The fact that, to me, many of the works seemed informed by the short, text-heavy world I live in, all a blur of Twitter updates and SMS messages, made the exhibit in its entirety particularly resonant.
The truth is, the Guggenheim as a space makes a terrible hotel. The room was hardly secluded, the amenities were perfunctory, and while the bed and chairs were comfortable enough, the gracious staff was the only part of the experience that compares to the quality of other fine hotels. That being said, I'd stay there again in a second.
D'Angelo and the Demons of the New Minstrel Movement
Spin Magazine's piece covering the rise and fall, and perhaps second rise of D'Angelo has been lingering in my mind for weeks. As you might expect, I was a fan of D'Angelo's from the start.
And that's true even though I was clowning him when he got arrested. To tell the truth, I hadn't quite realized just how far the man had fallen. If you look at the comments on my post from three years ago, you can see that even then people were saying they just wanted the man to get well so they could hear more of his work. ?uestlove articulates the challenge here better than anyone, though: "The new minstrel movement in hip-hop doesn't allow the audience to believe the artist is smart."
It's a particularly striking observation given that Spin's look at D'Angelo mentions in passing how that tension between art and commerce has affected so many of the acts I love. The world of R&B success demands either heaven or hell -- you either become a preacher and lose all of the sexiness and swagger that made you compelling in the first place. Or worse, you succumb to the demons.
While D'Angelo grew increasingly isolated, the rocky path he was traveling was, ironically enough, quite crowded with like-minded compatriots. At least three of neo-soul's other late'90s leading lights — Maxwell, [Erykah] Badu, and [Lauryn] Hill — have spent much of the new millennium on the sidelines.
Hill's struggles have been well documented: She followed her 1998 breakthrough, The Miseducation of Lauryn Hill, with an MTV Unplugged set four years later that felt like the soundtrack to a real-time nervous breakdown. She's yet to offer a second studio album and, apart from some aborted Fugees reunions, occassional shows, and involvement with a shady guru, much of her time has apparently been devoted to her family.
Badu released her triple-platinum debut, Baduizm, in 1997 and a successful follow-up, Mama's Gun, three years later, and then said she had writer's block and went on what she dubbed "The Frustrated Artist Tour" in search of inspiration. She eked out a slight EP in 2003 but then was largely silent, until the well-received release of New AmErkykah (Pt. 1 4th World War) last February.
Maxwell's journey probably parallels D'Angelo's most closely. The Brooklyn-born singer released three platinum albums between 1996 and 2001, earning frequent comparisons to D'Angelo, then seemed to disappear entirely. A new album, Black Summers' Night, was originally slated for spring 2004 but has been delayed repeatedly. Some close to him suggest that, like D'Angelo, he's been wrestling with a rather ill-fitting public image as a sex god.
There's much, much more in the story, but it's almost impossible to overstate how much a lot of us had put our faith for the future of soul music in a small group of talented artists. A decade later, it almost seems as if no one's even trying to carry the torch anymore.
We'll see how it goes; I've got tickets to see Maxwell in concert next month, and I'm still holding out for that new D'Angelo record.
?uestlove, the heart of The Roots and the standing leader of the neo-soul movement, is a born blogger.
And Erykah's "Honey" is one of the best videos of the year:
Me and Your Bicycle
My friend Mat Honan amused and beguiled you a few months ago with Barack Obama is Your New Bicycle. As is the course of such things, he got a book deal for his efforts, despite having been responsible for the onslaught of unfunny ripoffs of the site which followed his success.
But, I take some very small satisfaction in this whole thing because Mat very graciously credits me (both in the book and in conversation) with having helped spread the word about his site. It's just another in the long string of goofy web memes for which I have become an unofficial ambassador. It's a good thing there's no Hell, or surely I'd rot in it for all that I've done.
At any rate, Mat's quite an entertaining and engaging interviewee, as evidenced by his recent stint on Internet Superstar, and as there's a totally gratuitous and flattering mention of me at about the four-minute mark, I felt obliged to link to it here.
You can buy Mat's book at Amazon and other reputable booksellers near you.
Details of Execution
Sometimes if you do something very difficult, and you do it really well, the end result is that your achievement becomes completely invisible.
I mentioned a year and a half ago that I like Twitter. That was a little bit less common a position to take back then, but in the months since, tons of people have taken to the little messaging service, so clearly this was no great insight on my part -- it's just a useful, fun service.
But of course, that popularity has not been without its problems. Twitter's gotten a reputation for being unreliable, as a result of its rapid growth. In fact, in many ways, the Fail Whale and its related frustrations has come to define Twitter's brand more than almost anything else.
I'm no expert at these things, but there are a lot of reasons startups fail, and the reasons almost never include the fact that thousands of users clamoring for a service. Indeed, it seems to me that most companies (whether they're tech startups or anything else) fail because of being poorly managed. Put another way, execution is everything.
With that in mind, it's worth pointing out how particularly well-executed Twitter's recent acquisition of Summize has been. I don't know any of the deals of the financial or business arrangements, except that I'm a little disappointed that Twitter isn't maintaining a presence in New York City, instead moving all of the employees to San Francisco. That nitpick aside, the public face of this transition was extremely well executed.
Ev Williams, co-founder and the most public face of Twitter, speaks about the deal at some length in this excellent, candid interview with Techcrunch. (Which site, by the way, may rank as my "most improved" blog of 2008.)
Rumors of the Summize acquisition leaked a few weeks ago, but both companies kept discipline around communications and didn't acknowledge or respond to the conversation. And then, when it came time to announce the deal, the sites had been fully integrated, a lengthy and personable blog post complete with a sketch of some future ideas for integration was posted, consistent branding was in place on the acquired site, and the roadmap for what was going on with employees affected by the acquisition was clearly communicated.
In all, that's a formidable amount of coordination to happen across the country, while business deals are being worked out, and while maintaining secrecy about the fact that it's taking place. And, all of that was done with an eye towards providing a good user experience to their shared customer base.
There are a lot of things to criticize in such deals most of the time, though it seems likely that this will be a successful acquisition, from an outsider's point of view. But what's striking to me is that, as quick as so many are to criticize Twitter (fairly) for technological problems, people haven't been as eager to acknowledge a remarkable discipline and execution on the business side of the company. Frankly, all of those who'd suggested that Twitter should be sold to a larger company seem to have forgotten that almost none of the big companies suggested as acquirers have a history of consistently pulling off this kind of execution. And that's even more true for the smaller innovative companies that they've acquired.
I work at the new Six Apart (in New York!)
Five years ago, I said I work for Six Apart. At the time, that sort of thing was a big deal, not because of me, but because so few of us who loved blogging could get a job doing what we loved.
Since then, amazingly, it's become downright common to work in the blogging business. I have literally dozens of friends who work on creating tools and technology for blogs, and dozens more who blog for a living as part or all of their job. I even get to work with the best of them, from San Francisco to Paris to Tokyo. And now I can celebrate the company and industry I support in the city that I love, since we have an office in New York City.
As always, I'm immensely proud of working at Six Apart, even more proud to count such amazing coworkers as peers and friends, and proudest of all of what our community of bloggers has accomplished. When I started working at this company, my hopes were that we'd be able to teach more people about blogs, and that we'd be able to build a sustainable, ethical company that gave a bunch of talented people a great place to work. But in retrospect, I find it almost impossible to believe the role we've played in helping blogs become so common that they're taken for granted.
That's not to say it's been easy. At Six Apart, we've made a number of mistakes, and learned from them. We've all been through a lot of stress, both personal and professional. But even after all we've been through, Mena wrote a beautiful post in my honor, and last Friday offered one of the kindest compliments to me that I've ever gotten, recognition in front of all of my coworkers, a group of people whom I hold in the highest esteem.
But one point that she highlighted last week was that all acts of entrepreneurship are really acts of faith. My title these days (though I often cringe when I say it), is "Chief Evangelist". I've always been uncomfortable with the religious implications of it, but I've become comfortable with the fact that it reflects a bit of faith. This goes back to why I started doing this work in the beginning:
So I make tools that help people communicate. Mostly because I love technology, mostly because I love to try and build things and to get other people to think these things are cool, too. And certainly because I'm hoping to impress my friends and family with the end results. But some small, central part of the effort is because I know I'm privileged to be able to talk to anyone in my family at any time. In the span of a few decades, my father went from not being able to even send a letter to his father for a few years to being able to instant message me frequently enough to pester me.
This is more true than ever. I am glad to have stuck with a company, and with blogging, through both points of ceaseless hype and endless criticism. Well past any point of blogging being "cool" to the insular world of tech geeks, blogs have become enough of the fundamental infrastructure of communication to actually become interesting to the world at large.
And of course, I had some personal goals, too. I wanted to work with good friends, with people I know and trust. I wanted to show people that New York City is, and will be, one of the centers for real, hardcore technology innovation and invention. (We're hiring!) I wanted to bring together the worlds of the two things I have always been passionate about, technology and media.
As is likely obvious from our announcements this week, we're close to being all of the things I'd hoped a company like Six Apart might become. In just the past year, we've damn near reinvented the company, with Ben and Mena and our CEO Chris Alden have been leading some brave efforts to do what few have the courage to do: Reimagine a company that's already successful and growing, and picture it honoring its innovative roots in a way that's actually new. We've invented, launched, and promoted more things that make the web better in the past year than at any time since the beginning of the company.
That kind of creative destruction, the willingness to take apart something that's working in order to make it something truly inspiring, is actually even more ambitious than I'd imagined Six Apart being when I'd joined. And it's the reason that, after five years, the milestone for me is that it feels much more like I'm starting a new job than that I've been at one for half a decade. I can't ask for much more than that.
Jeff Bezos, Ray Ozzie and Pierre Omidyar on Workspace
Continuing from yesterday's look at the soundtrack to the creation of Lotus Notes, we can look more at the physical space where it was created. For contrast, I also throught I'd start looking at some of the responses I'd gotten from Jeff Bezos about the same questions.
Interestingly, when it came to the music or movies that were playing while he was first creatiang Amazon.com, Jeff's answer was succinct: "I don't remember." Maybe I might have done better to focus on what books he was reading. But when it came to describing the actual workspace, Jeff remembered a lot more details:
A garage enclosed so it was converted into a room. Whiteboard with long list of priorities -- didn't change much. Door desks. Costco swivel chairs. Big orange extension cords draped across the floor just about everywhere.
That sense of a chaotic but comfortable space is echoed in Ray Ozzie's description of the early offices at Lotus:
it began in a small office (actually an old home converted to an office) we rented in 12/84 in Littleton, MA.  The office was mainly just one big room for the three of us.  I founded it in December, and my co-founders Tim Halvorsen and Len Kawell joined me from DEC in January.
We used IBM PC AT's as our dev systems, which were released just as we were starting to work.  Even though our office was Spartan, we bought the best hardware available and tricked it out as best we could:
a "massive" second monitor ("Genius" I think) - 1024-by-something monochrome portrait mode
a removable iomega Bernoulli disk drive, so we could do builds, archive things, bring them to Cambridge where our partner lotus was located, etc
we replaced the crystals on the motherboards to get 8mhz out of the computers, rather than the stock 6mhz
sytek 2mbps (I think) LAN card
a state-of-the-art newfangled "laser printer" - an apple laserwriter - that we all shared
You get the idea.
We went to a used furniture store and bought the CHEAPEST crappiest (but strong) fold-out tables, with strong/comfortable chairs.
We spared no expense on massive whiteboards that covered the walls.
Pierre Omidyar's description of the workspace where eBay was created is no less evocative:
Definite clutter. I worked primarily out of our spare bedroom that I used as an office. I had some sort of computer desk that had multiple Macs in various states of use or disrepair. I also used a Mac laptop, a Powerbook Duo among other models I think. Later I very reluctantly switched to a Toshiba laptop and Windows, because the Mac OS wasn't keeping up with the cutting edge back then. (A non-Mac hiatus that lasted until 2001 I think.) I had a wireless internet radio thing hooked up to it so I could access the Internet mobile. I used post-it notes on the monitor of my desktop Mac or in the laptop, but no whiteboards. It wasn't until I got an office that I started using a whiteboard. I like whiteboards, but the markers smell funny.
In each case, it's gratifying how familiar this combination of clutter and creativity feels to any of us who've ever pulled an all-nighter to get a product launched.
Atom Wins: The Unified Cloud Database API
Almost five years ago, I got involved in a project that would eventually become Atom, a pair of matching standards functioning as a syndication format and a publishing protocol. Though its contentious genesis was in the world of blogging and feeds, what's amazing is the improbable end result of the Atom community refining and debating the standard for half a decade.
AtomPub has become the standard for accessing cloud-based data storage. From Google's GData implementation to Microsoft's unified storage platform that includes SQL Server, Atom is now a consistent interface to build applications that can talk to remote databases, regardless of who provides them. And this milestone has come to pass with almost no notice from the press that covers web APIs and technology. A single common interface for keeping data in reliable cloud storage is as important a development as infrastructure-on-demand offerings like Amazon's EC2 and S3 web services, and can radically impact the creation of applications for distributed social platforms like OpenSocial or the Facebook API.
Of course, there's a lot more to the AtomPub story. There's open source support from infrastructure-level implementations like Apache Abdera to application implementations as in Movable Type and interoperability testing as made possible by The Ape. But those have been around for a while: Being able to build a program that doesn't care if the backend is Google Base or SQL Server is new.
The reason this is a true milestone, and that it's especially disappointing that the tech trade press hasn't been paying attention, is because we've seem some remarkable announcements.
Microsoft is making a large investment in unifying our developer platform protocols for services on the open, standards-based Atom format (RFC 4287) and the Atom Publishing Protocol (RFC 5023). At MIX we are enabling several new Live services with AtomPub endpoints which enable any HTTP-aware application to easily consume Atom feeds of photos and for unstructured application storage (see below for more details). Or you can use any Atom-aware public tools or libraries, such as .NET WCF Syndication to read or write these cloud service-based feeds. ...
The intent for these early, experimental releases are to gather valuable feedback from the community around our idiomatic and freely licensed extensions to AtomPub which deal with important service scenarios, such as URL formats, nested directories, image streams, and service metadata. You can read more about this on the Project Astoria team blog.
We've always encouraged other developers to adopt Atom, the Atom Publishing Protocol, and the extensions that Google has created on top of those standards, but we realized the issue of patents may have held back some adopters. Well, those concerns end today as we are giving a no-charge, royalty-free license to any patents we have that you would need to implement Atom, AtomPub, or any of those extensions.
There's clearly still work to be done, of course. I believe Microsoft will find Google's licensing terms open enough for them to feel comfortable implementing GData-compatible APIs. Amazon should find a way to map these same APIs onto the SimpleDB service, though I understand fundamental differences in architecture may make that a challenge. The multitude of smaller cloud-database vendors will find it easier to get adoption and customers if they can assert compatibility with these implementations.
But, put succinctly, Google + Microsoft = AtomPub wins. To paraphrase Dave Winer, the act of putting aside ego and saying a competitor's API is good enough, and that you're going to support it, is a brave and important act in the world of technology. That makes this convergence particularly exciting.
And it's a real act of technological courage because any reasonably competent technology company can find reasons to object to any standard or API that's ever presented. "We'd prefer it work this way." "Your implementation is too verbose!" "What about just using this other technology?"
In the end, it doesn't matter if a standard is perfect; All that matters is that it works. I am, by no means, the kind of person who thinks that anything adopted by two giant tech companies is always going to be acceptable, but the fact that there is also significant support for AtomPub from small, independent hackers is a pretty good sign that this victory is good for the web as a whole.
Now I just hope people make something really cool out of this. I want every program that thinks of itself today as a "blogging client" to reimagine their market as being a front-end to a database in the cloud. I want all the apps built on smart database abstractions to think about this new unified cloud API as an option they must support. And most of all, I want geeks to make something cool with this that we couldn't do before.
Now that his single Sensual Seduction (a.k.a. Sexual Eruption) is in the Top 10, it's official: Snoop Dogg can see the future. First, let's pause to watch the best video in the history of anything, ever. (The song is about four minutes long, so you'll want to allot approximately half an hour so that you can watch it in its entirety at least six times.)
Though the song has been out a few months, I had somehow missed the video when it was first released, and now I can barely express how profound my regret is that I have lived a few more weeks of my life without this video in it. That being said, we have a lot to learn from, here. I'll be exploring that in my next few posts.
(Special note to the Universal Music Group: I've been trying mightily to promote your product, and yet you are so intent on sucking ass that I can't even embed your video on my blog. N.B. I'd have been happy to do so with your ads wrapped around it. Meanwhile, there's about 1,000 other copies of it on both YouTube and Brightcove. Weak! Also, given that this is a Snoop video, I think that while the video is buffering, it should say "Loading, Player" instead of "Loading Player". Thanks!)
Free Publicity: Who do we help?
I'm not a Democrat; I don't much care about the scorekeeping of who has more seats in any given chamber of Congress. But I do think there are things that need fixing in this country, and one of the most important is acknowledging when things are going the right way. More to the point, we need to find a way to use our collective powers of amplification for something that helps us, instead of as a reward for distracting us.
Tonight will be the President's State of the Union address. I'm very interested in what he covers, not least because the address will be the start of a two-way dialogue, as I outlined on the Expert Labs site. I think that's a pretty big improvement over simply addressing our elected officials.
But the world I inhabit, at the intersection of tech and media, is far more obsessed with what Apple's going to announce about its tablet. People who write about gadgets for a living gotta pay the bills, and I love cool stuff as much as the next guy. What leaves me at a loss, though, is how many otherwise sane and sensible people give their time and energy freely to help support a company like Apple that, despite its elegant designs and generally excellent products (I use many of them), certainly doesn't need free PR from some of the most talented people on the web.
Though Apple is a reasonably progressive company, they explicitly don't give a shit about poor people. (Let's pretend I found a nicer way to say that.)
Who does need your help? I'd say the current administration does. Because the biggest difference between now and 18 months ago is not that President Obama has gotten elected; It's that those who support his agenda have gotten lazy about helping in the effort. Remember "We're the ones we've been waiting for?" Well, it seems like a lot of people got tired and gave up on themselves. What if all the energy that went into free promotion for the Apple tablet went into free promotion for what's been achieved so far, in the hopes of encouraging more achievements in the future?
The Feature List
I know, I know. the conventional wisdom is "Obama ain't done nothin'!" But that's clearly bullshit. Obviously, political opponents are going to parrot that idea, but I'm surprised that even supporters are lazy enough to believe it without fact-checking. Perhaps everybody's attention spans have been a little too shortened by chasing the next Apple rumor, because the facts are obvious. In one year, here's what I caught (you might have your own list):
The last U.S. Marines are leaving Iraq.
Credit card companies can no longer charge interest on fees, and can't retroactively raise your interest rate on existing balances.
We know who visits the White House, and who they're affiliated with.
There's a quarter billion dollars more funding for National Parks, and $50 million more for the National Endowment for the Arts.
We responded, imperfectly but with heart and sincere effort, to the disaster in Haiti. Just as we wish we had after Katrina. Leadership matters most in emergencies.
Our current President readily admits when he's made mistakes, respects the validity of arguments that he disagrees with, and has members of the opposing party in his cabinet.
The Department of Homeland Security now allocates its security spending according to threats, not by spending the same amount of money on Montana as it does on New York.
My 401k is up 30% since the current President took office.
Our President asked both corporations and individuals to reduce their electricity consumption. He asked politely.
Trains. There's a plan to build more rails and more trains for transporting actual humans around the country.
The Matthew Shepard hate crime bill was passed.
Now, that's just my list. These matter to me. Maybe you have your own list. Or maybe there's only have a wishlist of features for an Apple tablet. The difference is this: Our current President is listening to what your requests are, and wants to hear them. Steve Jobs doesn't give a fuck about you. I promise. I'm typing this on an Apple keyboard hooked up to a MacBook, and I don't use Windows anymore, but I guarantee you that Steve Jobs is not going to get those last Marines out of Iraq.
And I know, I know, people will piss and moan about the stuff this administration hasn't gotten done yet. So my question is this: What did you do to help? Did you do 1/10 as much as you did to get these folks elected? Did you do as much, today, as you did to help Apple sell billions of dollars of products that you get no stake in, that don't help make life better for you and your friends and neighbors? What are you waiting for, somebody to ask nicely? I'm asking nicely: Please find a cause you care about, and beat the drum to stir up public sentiment to support it. Make it your wallpaper on your new tablet.
I'm not scolding you; I'm scolding me.
I had to ask myself these questions. Sure, I've got a bunch of tweets about Apple features that I want to request, and of course I'll watch the Stevenote as rapt as when I watch the State of the Union. But we all have a choice to make about how we invest our time, attention, and passion. And I'll bet in eight years, today's tablet is gonna look an awful lot like a first-generation iPod looks today. Some efforts age better than others.
My goal here isn't to browbeat anybody, or to lecture. I'm in the same boat as everybody else who loves technology. But my personal reckoning has just shown me that a bunch of libertarian-leaning geeks in Silicon Valley who refuse to engage with government and civic society at all are never going to make an impact on most of the things that actually make a difference in our lives. Everybody in Silicon Valley will tell you they have a gay friend, but they couldn't stop Prop 8 or get the hate crimes bill passed. Probably everybody at Apple thinks "We should do more to support the arts!" but they weren't funding the NEA. There will be no iTrain.
Right now there are a lot of hopeful, and possibly deluded, people in the old-line media businesses who hope that an Apple tablet will prop up their failing magazine, newspaper or television businesses. Those of us who are digitally savvy are probably having a chuckle at their expense, snickering at their wishful thinking. But Apple will invest a lot more in saving any given book publisher than they ever will in saving civic society, in protecting individuals' rights, or in engaging in diplomacy to neutralize the threat of violent extremists.
I'm gonna try to spend at least as much time advocating for issues I care about as I do for the purchase of new gadgets. I hope that even those who disagree with me on those issues do the same. Maybe there'll be an app for that.
Last week, I wrote a bit about what it's like to be on Twitter's suggested user list. The response to that post has been really gratifying, and I wanted to share a bit of what I've learned, as well some of the more interesting responses.
First, to recap: I had about 18,000 followers of my own back in October, when I got added to the suggested user list. (Let's call these "organic" followers.) If I'd have continued my normal rate of growth, i'd have about 25,000 followers today, but thanks to being on the list, I've got close to 300,000 followers. Surprisingly though, I only get as many retweets and replies as I'd get with my organic number of followers.
I thought at first that maybe the list wasn't valuable to me because I'm not a celebrity; maybe I'm just noise, but could bigger brands find some value by having a large number of followers?
The Results Are In
As I hoped, my initial post about my experiences inspired others on the list to chime in with their findings.
Creative Commons, despite being a stalwart organization at the intersection of technology and intellectual property, saw no increase in responses after being added to the suggested user list.
NBC's Today Show is one of the signature brands of broadcast media. But being on Twitter's list? Didn't do anything.
What about Starbucks, one of the definitive examples of a powerful worldwide brand? Nothing.
I mentioned in my earlier post, that Kim Kardashian is being paid $10,000 a tweet to promote sponsors on her Twitter account. But what are those sponsors paying for? Because, while she clearly has influence over a certain community, and her Twitter page says she has about 2.7 million followers, I think the reality is obvious: Nobody has a million followers on Twitter.
Does that mean Twitter's follower counts are lying? No. Instead, Twitter accounts that have over half a million followers listed actually represent (at most) a few hundred thousand people who've chosen to become organic followers of someone, along with millions who are passively along for the ride. Some of them are inactive users, some are spammers, some just ignore the noise of the accounts that don't interest them, like spam in an email inbox. But they can't count as "followers" in any meaningful sense.
A few people have asked what my goal is in writing about the experience of being on the list, and why I am offering up prizes to encourage asking questions about it. Well, perhaps the best way to articulate it is that I think the list is being used as a useful fiction for distorting the value and promise of this new medium.
The Million Dollar Gift
There are incentives to promoting the fiction of the suggested user list, of course. If I were the brand manager or Chief Marketing Officer for some big company that got on the list, I bet I'd be proudly trumpeting to senior management that "our social media efforts are bringing us thousands of new followers a day on Twitter". Somebody's gonna get a huge bonus for being the beneficiary of an act of random benevolence. Hell, I'm a pretty persuasive guy — if I found the right (i.e. sufficiently desperate) media outlet, I could probably have sold my Twitter account to somebody for half a million dollars. Well, at least I could have until last week.
And the list preserves a certain amount of power and influence for Twitter itself. (Twitter the company, not twitter the medium.) Because, for every one of the organizations i quoted above mentioning how the suggested user list provided them no value, I got a private message from another list member confirming these findings but not wanting to be quoted on the record.
People being afraid to publicly state their opinion about something of little value for fear of antagonizing a particular company is a clear sign of a completely unhealthy dynamic. I don't think the folks at Twitter would retaliate for public criticism by removing people from the list, because Twitter execs are both extremely busy and fairly thick-skinned, but it shows how insecure people feel about having won the follower lottery. (And how pageview-obsessed publishers are: Every entity that was afraid of being removed from the suggested user list is in the business of publishing content online.)
Fact Check
CNN famously reported on Ashton Kutcher beating them to be the first to get a million followers on Twitter; Today's celebrity reporting often includes a mention of a celeb's follower count as a matter of course. But I'm hoping to encourage some skepticism, to provide a basis for fact-checking that demonstrates these pronouncements are inherently suspect. It's a bit like when I worked at a newspaper: Every reporter thought "Well, our circulation is a million copies, that must mean a million people read my column." Facing the reality that only 10,000 of those people read the column, or that perhaps only 1,000 of them were reading the advertisement on the opposite page, forced a useful and important reckoning into some false assumptions that were underpinning that industry's workings.
The truth: Nobody has been able to point me to a single Twitter account that's earned over 250,000 followers on its own. Nobody's been able to point me to a Twitter account on the suggested user list that's gotten favorites, replies, retweets or responses from a larger number. And nobody's been able to demonstrate why the inflated follower count numbers should be used as a measure of anything but the growth in signups to the core Twitter service itself. [Update: I had suspected some popular artist like Nicki Minaj, the Lil Wayne protege who has famously rapped about her Twitter following, might exceed these numbers. As it turns out, the highest organic follower count I've found is from teen pop heartthrob Justin Bieber with over 800,000.]
That leaves an inescapable conclusion. Nobody has a million followers on Twitter. And being on the suggested user list doesn't add value to a Twitter account, regardless of whether you're a regular guy like me, or one of the biggest brands in the world.
Reminder: I'm running a contest for ideas about how to get more data from my being on the suggested user list. I've been running Gina Trapani's smart little Twitter application ThinkTank since before I was added to the suggested user list. As a result, I have an archive of all my followers, tweets and replies going back for months.I'll provide a prize to one random person who suggests an idea of what information we should query from that data set, as well as one random programmer who contributes code to help.
Here's the prizes and how to participate:
Have a question or specific bit of data that you'd like to know about an account on the Suggested User List? Submit it to Twitter with the hashtag #sulidea and one random person who makes a suggestion will get a $25 Amazon gift certificate.
If you're a programmer, watch ThinkTank on GitHub, commit any updates you have to the project, and one random person who commits code to the project will win a 500 GB portable hard drive.
I'll be picking winners for both prizes on January 15th.
Life on the List
In the time it takes you to read this sentence, I'll have gained another follower or two on Twitter. Within an hour, I'll have added more followers than 99% of Twitter users ever have. On a typical day, I'll have averaged 100 new followers every hour. It's not that I'm great at writing tweets or because of any effort or merit on my part; It's because I'm part of Twitter's list of suggested users.
The Suggested User List has been one of the most controversial and misunderstood parts of the explosive growth of everybody's favorite cerulean social service, though the company has loudly hinted that its life is limited. So I thought I'd explain a little bit about what Twitter is like when you're on the list. I'll explain the surprising impact that being added to the list has on replies and retweets. And at the bottom of this post, I'm even offering up a chance for people who are curious about being on the list to win some prizes, too.
What is the list?
Twitter's Suggested User List works in a fairly simple way. When a new user signs up for Twitter, they're presented with a list of about 20 "default" accounts to follow. These recommendations are a random subset of a full list of over 400 suggested users. In addition, the full list appears on the Twitter site itself, so if any user clicks on "Find People" at the top of their Twitter page, they're only one click away from choosing to follow some suggested users.
It's obvious why the team created these suggestions; If you just signed up for Twitter and weren't following anyone, it'd be a pretty boring service. Social applications have provided plenty of precedent for the practice of suggesting content or connections, but Twitter's exceptional success and the fact that tweets are seen more as a new medium rather than merely a feature of the Twitter service have made the suggested user list into a polarizing reminder of the company's power over the service.
What's not obvious is why I was picked as a suggestion. I have a number of friends at Twitter, including about half a dozen let's-grab-dinner-when-you're-in-town level of friends. As Biz noted, I was an early an enthusiastic fan of the service. And I'd like to think I'm not a terrible tweeter — my updates are a mix of interesting links that I find, random thoughts, brief reviews/mentions of music and media that I like, and promotion for the projects I'm working on. But I'm obviously not a better tweeter than 99 million other Twitter users, I never asked to be on the list, and it's never been explained to me why I was chosen. Ultimately it's clear that the decision of whom to feature is essentially an arbitrary choice by Twitter , and that at best, I represent something they'd want to show new users.
A list of suggested contacts makes perfect sense when a service has about 10,000 users, to help them get started in an unfamiliar space. But it's a system that starts to strain a bit once a service reaches 10,000,000 members. (Or even, as it appears, nearly 100 millon members.) Of course, the folks at Twitter had no way of knowing they'd leap from a five-digit user count to a nine-digit one faster than anybody else on the web ever has. Combine Twitter's support for user-defined lists on the service and the criticisms of the list that have surfaced, and it's easy to see why Twitter's announced that the list's days are numbered. I'd be shocked if it doesn't disappear entirely in 2010.
So, I don't have any real issue with the fact the list was made in the first place; If I were a Twitter shareholder, I'd fully expect the team to design the best possible experience for new users. (If I were a substantial Twitter shareholder, I'd buy a round bed and fly it through space like Snoop Dogg. But I digress.)
I do have some misgivings about the effect of the list, though. In addition to showing how much control Twitter has over the medium they've created, the list also causes some pretty uncomfortable and awkward distortions. It conveys remarkable privileges to the few hundred of us who are members. A lot of celebrities, some past their prime, have pointed to their enormous numbers of followers on Twitter as evidence that they still command some sort of passionate following online. Other nascent talents have had their profiles raised by becoming "Twitter stars", with their thousands or even millions of followers held up as proof of strong demand for their ideas.
A Dutch kid sold his Breaking News account to MSNBC, and Kim Kardashian is famously selling her tweets for $10,000 a pop. But I've been able to determine that having hundreds of thousands of Twitter followers is basically only a measure of having been on the suggested user list, and doesn't consistently indicate any intent from Twitter users at all. So, not to take away from Breaking News or Kim Kardashian, but there are people making a significant amount of money simply by virtue of having been on the suggested user list.
And it turns out, those suggestion-heeding followers might not actually be paying any attention at all.
The Power of Suggestion
I had no advance notice I was going to be added to the list. I went out for coffee with a friend, and returned to find a few hundred emails in my inbox, all of them notifications from Twitter that someone had followed me.
To my surprise, and to the disbelief of nearly everyone who's asked me about it since, I wasn't immediately excited or thrilled to have won the Twitter jackpot. For the first weekend, I wasn't sure what to do with all these new followers, and I didn't update my status at all for 2 or 3 days after I first got added to the list.
Now, that's pretty unusual behavior for me — I've been blogging for ten years, and I'm fairly public within the tech industry. I don't get nervous standing in front of thousands of people when speaking, and over the years my blog's gotten a pretty significant number of subscribers as well, yet I never had any similar concerns here. So what changed? Well, I tend to use social services in a more personal way than my public blog post. And, honestly, the sheer rate at which people follow a suggested user on Twitter's list is overwhelming. Let's look at the velocity with which a suggested account accrues new followers.
Here's a chart of my new followers, courtesy of TwitterCounter;:
The small flat area at the extreme left of the graph is what my growth rate looked like before I was on the list. It doesn't seem like it, but that was actually an uncommonly high rate of new followers. For contrast, I did a comparison with Chris Messina, who accrues new followers at about the same rate I had been, writes about similarly geeky topics as I do, and actually started wtih more followers than I did:
Yes, compared to being on the suggested user list, a very popular normal Twitter user's growth looks pretty much flat. That's how different it is. Nevertheless, after a few days of being on the list, I decided I was going to just tweet the same way I always had, and not overthink things too much.
Finding Meaning
People who accept the suggestions of the list are almost all new Twitter users, and have barely formed a model of how Twitter works. In some cases, due to the extraordinary amount of hype around Twitter, they've barely formed an idea of how the web itself works before signing up for Twitter and becoming one of my ostensible followers.
There's precedent for this sort of "bundled content", of course. The crappy "shovelware"; programs that come with most Windows PCs are a perfect example — they often nag users, are frequently of little value, and often detract from the experience. I often update with non-sequitirs about stuff like peanut butter jelly time, so I have to imagine that a regular Twitter user seeing my updates must see me like a notice that their new Windows computer has cleaned up the icons on their desktop.
Of course, services like Amazon and iTunes feature content as well, but these are usually pretty straightforwardly analogous to endcap displays in retail spaces like a grocery store or Walmart; The stores sell placement and brands that want exposure pay for the real estate.
After just a few days of being on the list, though, I made an interesting discovery that offers a dramatic distinction from buying featured position in an online store: Being on Twitter's suggested user list makes no appreciable difference in the amount of retweets, replies, or clicks that I get.
Once in a while, I get confused replies from people asking who the hell I am, but for the most part they don't interact with me at all. The replies, retweets and conversations that happen for me on Twitter have the same frequency and volume that they would have had if I'd never been added to the list. I'm sure celebrities (whether on the suggested user list or not) get a disproportionately high number of people trying to catch their attention, but for a normal person, being on the list just adds followers, not real connections.
Twitter followers who come from the suggested user list don't form real relationships or respond to the suggested users like "normal" followers do. If I'd have continued gaining followers at the rate I had been before being on the list, I'd have about 10% as many followers, but I suspect I'd have exactly the same number of replies and retweets. Before being on the list, a typical link that I tweeted would get between 250 and 500 clicks; After being on the list that hasn't changed at all.
And for me, that's a little off-putting. I feel very much like I've earned the readers who subscribe to this blog. When I meet someone at an event and they tell me they've read a post of mine, or that they regularly read my blog, it's still a thrill, even after a decade, because there is some core sincerity to the exchange, a real basis to the relationship. With Twitter, it's hard for me to tell whether someone's made a decision to follow me because they find my ideas interesting or entertaining, or if they just were too lazy to change the defaults when they signed up.
I'm not complaining; I know a lot of people would love (or think they'd love) to be on the list. I've had some remarkable bits of serendipity, like my next door neighbor discovering me on the list. But I also missed the notification that my cousin was following me on the service because there's too much noise for me to turn on notifications. For the way I use the web, I value meaningful connections much more than I do sheer volume of followers.
Adding to the feeling that these aren't "real" connections is that almost nobody has gotten more than 200,000 followers or so without being on the suggested user list. I'd be curious to know the most popular account that's never been on the list, but at the very least the combination of prominently featuring follower count as a "score" on people's profile pages while also having the only path to earning a high score being an arbitrary selection through an opaque process is a recipe for leaving a lot of people frustrated or mystified. Indiscriminate followers might be of some value for a business that just wants to have a lot of people to talk to, but for an individual, being on the list only has value to those who want to brag about the number. I'll admit I've been tempted to use my follower count as a credential in my work lately as it's taken me to less tech-savvy corners of Washington, D.C., but the fact that the number is meaningless made me feel it'd be dishonest and would misrepresent my actual influence.
Because I've been privileged enough to be on the list, I've tried to use the power for good. I am very happy that I'll be able to promote my work with Expert Labs to a larger audience, though I don't think I have any way to translate this audience into followers of @expertlabs. I have also tried to promote worthy efforts by my friends or to support charities. But there's also generally a continuous stream of requests from spammers and schemers and just plain icky hustlers who want, expect or even demand that I promote their work to my large follower base. Explaining to them that these followers don't click on links, reply or retweet requests does nothing to dissuade them, unsurprisingly.
So if I had a choice in the matter and knew then what I know now, would I choose to be on the list? I'm not sure, but I think probably not. But, since I am, I wanted to try to do something interesting before either the suggested user list disappears or I ask (As Jay Rosen did) to be removed from the list.
Open to Suggestions
I want to see what interesting information we can tease out of my place on the suggested user list. There are a number of questions that immediately pop to mind, which I don't have specific answers for:
Has the rate of replies or retweets per day (or per week) increased as much as my follower count has?
Do I get more favorites from users, proportionate to the number of new followers?
I suspect there are lots of other bits of data that I think could be compelling, and the good news is that we might have a way to process some of that data. I've been running Gina Trapani's smart little Twitter application ThinkTank (formerly Twitalytic) since before I was added to the suggested user list. The app can pretty easily be customized to return whatever data queries we're interested in. As a result, I have an archive of all my followers, tweets and replies going back for months. So I'm proposing a simple contest to solicit ideas for what information people are interested in mining from the account of someone on the suggested user list, and I'll provide a prize to one random person who suggests an idea, as well as one random person who contributes code to help.
Here's the prizes and how to participate:
Have a question or specific bit of data that you'd like to know about an account on the Suggested User List? Submit it to Twitter with the hashtag #sulidea and one random person who makes a suggestion will get a $25 Amazon gift certificate.
If you're a programmer, watch ThinkTank on GitHub, commit any updates you have to the project, and one random person who commits code to the project will win a 500 GB portable hard drive. It's really cute!
I'll run the contest until January 15th, and then just pick a winner at random from people who tweet or submit code. I think there's great potential to discover some surprising insights about how the suggested user list really works.
The Twitter API is Finished. Now What?
Update: We've got some results already! Joseph Scott at Automattic mentions in the comments that he's added RSD support for the Twitter API to WordPress.com. I should also make clear that I am very confident that we'll be building apps on top of this API at Expert Labs, so insofar as I'm the Director of the labs, I've got a vested interest in seeing efforts around an open API succeed.
Twitter's API has spawned over 50,000 applications that connect to it, taking the promise of fertile APIs we first saw with Flickr half a decade ago and bringing it to new heights. Now, the first meaningful efforts to support Twitter's API on other services mark the maturation of the API as a de facto industry standard and herald the end of its period of rapid fundamental iteration.
From here, we're going to see a flourishing of support for the Twitter API across the web, meaning that the Twitter API is finished. Not kaput, complete. If two companies with a significant number of users that share no investors or board members both support a common API, we can say that the API has reached Version 1.0 and is safe to base your work on. So now what?
How We Got Here
Like a lot of folks, I've been thinking out loud and pondering the future of Twitter and open web APIs pretty much all year. Some key ideas have bubbled up:
[A]ny site or application can deliver realtime messages to a web-scale audience, using free and open technologies at low cost and without relying on any single company like Twitter or Facebook.
Understanding new tech needs to be a weekend-sized problem.
There has to be value before everybody has upgraded.
You have to be able to understand and explain it.
Those posts from this summer show that the ideas behind the Twitter API's "overnight" ubiquity have been kicking around in developer circles for months, if not more than a year. Finally, though, we have shipping examples of broad adoption of an API that's lightweight and suitable for today's most interesting applications. It's not just that Twitter's realtime, though of course that is compelling, but also that these APIs are simple enough for weekend hackers to build interesting projects on, and that they're easy to implement even on mobile devices and in almost any programming language.
So, today, we have support for the Twitter API from Twitter (of course), WordPress and Tumblr. I know I saw folks working on this for TypePad's free service when I was at Six Apart, so I'd assume they just wanted to finish OAuth support before supporting it as well. (See below.)
Of course, I don't need to make any suggestions to developers about what to do with these APIs — I'm sure the gears in everybody's heads are turning about cool new applications to build. Instead, I'd like to make a series of suggestions for the entire Open Twitter API ecosystem, based on what we've learned from past successes and failures in APIs around blogging.
What Server Developers Should Do
Please please please support OAuth: It's egregious that the newest implementations of the Twitter API are stil encouraging people to share their passwords with third-party sites. Five or ten years ago, this was common practice in APIs because we didn't have better options. Twitter started out using shared passwords, but mercifully has started to bring OAuth support online. But for new services to be encouraging the horrible practice of users entering their passwords into every application willy-nilly is just unacceptable. I think we have a two-week window or so within which the new services supporting the Twitter API could announce their intention to support OAuth and really catalyze client developers into doing the wrong thing, but I fear we may lose another generation of API evolution to this terrible practice. If just one or two services announce intent around OAuth by the end of the year, client developers will follow — if you use WordPress or Tumblr, encourage your service provider to do this. (This is usually where I'd insert a dozen examples of how sharing passwords screws users, services, and the ecosystem, but I know that developers often just use shared passwords because they're lazy. Do the right thing, guys. The client devs will follow along.)
Support Really Simple Discovery: The RSD format isn't sexy by today's standards, but grew organically out of some smart thinking from when blogging APIs were at the same state of maturity as today's tweeting APIs. Instead of reinventing the wheel, developers should look at supporting RSD and looking for something like a "tweetsapi" endpoint for these new services. That way, any arbitrary site can advertise that it supports the Twitter API, or even future versions of an open MetaTweets API. Pay attention to which APIs are listed as "preferred".
Think about overloading of source: The source element of status updates in the Twitter API is very interestingly open-ended, and supports use of URLs. Instead of merely advertising your client app, smart use of rel attributes and URLs here could help bootstrap some very interesting new potential.
What Client Developers Should Do
Support RSD: Same logic as above.
Start sharing parsing libraries: Client devs going to be doing a lot of duplicate work to parse out URLs and usernames and hashtags and maybe even slashtags. But almost every scripting language supports some similar variation on regular expressions, and if you're using that method to tease out meaning from short messages, then lighten your burden by sharing the load. John Gruber's work to share his URL parsing rules should be a model for a dozen other GitHub projects — compete on features and execution, but not on these fundamental interpretations of text.
Build in the big services, but support the little ones: You'll naturally want to offer menu options for users of the big, centralized hosted services. But (perhaps as part of supporting RSD), you should allow for all of us to have arbitrary Twitter API endpoints on our own domain names — this is good for the web!
What Every Developer Should Do
Think about piping Twitter API endpoints together: I think it will be common for some kinds of applications that support the Twitter API to be both clients and servers, supporting piping content through, and perhaps applying transformations to the updates. This idea of daisy-chaining services together is likely only going to happen if a lot of parts of the infrastructure support OAuth well, but has the potential to be truly revolutionary if the ecosystem allows it to happen.
Start looking at people's firehoses: Twitter's firehose of all status updates is about to be broadly available for developers, I know about the free TypePad firehose from my time at Six Apart, and I think WordPress will sell you access to theirs, but I haven't yet been able to find a reference for one for Tumblr. No matter — we should assume that free, open versions of these are coming, and start to figure out how to encourage similar collaboration around the reading side of things, now that the writing side of things is getting hashed out.
Consider adopting a "+2 Rule": The natural inclination right now for geeks of a certain type is to start dreaming up new standards bodies, or how they can participate in the Open Web Foundation to make a Super Awesome Twitter API Evolution Committee. Here's my recommendation: Don't. Don't do any of that shit, and don't run off to make membership badges for the Treehouse Club quite yet. Instead, just iterate and ship. Keep making new apps and see what you can do to stretch the limits of the existing methods and structures. I love the new geocoding and contributor aspects of the Twitter API, but as I said at the top of this post, I think the period of rapid iteration on the core Twitter API is ending, as new efforts going forward will have to reach consensus.
The good news is, consensus around evolution of the Twitter API can happen simply by saying to each other, "If two application developers who share no common investors or board members can reach agreement around an extension to the API, and between them they have a significant enough number of users to be relevant, then we should all just adopt their work."
This is important because it reframes the conversation from being about technical merits, and all the boys who like to play with APIs always think they know what's "better". I'm sure if I wanted to waste an afternoon, I could tell you a dozen ways in which the Twitter API could be "improved". But guess what? That shit does not matter. Adoption matters, and I'm heartened by the fact that people seem to be getting that.
So, get to work! Please give me feedback if I'm wrong or being stupid about one of my recommendations, but if not, then just start hacking. Stop encouraging people to share passwords, start encouraging services to share tweets, and let's all join in a hearty session of finger-pointing and mockery in Facebook's general direction for their sense of Not Invented Here having overshadowed their opportunity, because they could have really clearly done an "embrace and extend (and extinguish)" on the Twitter API if they hadn't wanted to make their own system a year ago, and now they've lost that power.
Finally, thanks a lot to Dave Winer for essentially inspiring a lot of players in blogging to move towards embracing the Twitter API. Sure, lots of us had the idea, and I've spent a lot of times in meetings arguing for this stuff across the industry, and Automattic and Tumblr and others were brave enough to embrace it. But I don't think anybody's done more to publicly advocate for an open Twitter API than Dave. I'm glad we've evolved as a community to the point where these kinds of breakthroughs aren't the contentious, immature shitfests they used to be.
The Most Interesting New Tech Startup of 2009
I love seeing people start new companies, especially in the tech world. But I've probably gotten a little bit jaded about new startups, especially when the story seems to be more about who's funding the effort than about the product itself. To me the distinction that makes a startup interesting is not just whether their own product or service is cool, but whether it's broad and ambitious enough that others can build interesting things on top of it.
So, after taking a pretty careful look at the tech scene (and of course with a number of my recent posts being focused on Facebook, Google, Apple and other giants of the tech industry), I think the most promising new startup of 2009 is one of the least likely: The executive branch of the federal government of the United States.
Now, .gov websites have historically been backwaters at best, a bunch of awkwardly-designed, poorly defined sites that only met the bare requirements of a web presence. But of course the current administration is comprised in great part of digital natives, and it's remarkable how quickly they've remade the .gov world into not just a number of compelling websites, but into a broad set of platforms that are going to inspire as much technological innovation as Twitter, Facebook or the iPhone did when they unveiled their technology platforms.
Need proof? Well, let's take a look at some of the most compelling new sites that have launched in just the few short months since President Obama took office:
Data.gov, providing open access to feeds of valuable facts and figures generated by the executive branch.
USAspending.gov, allowing any of us to drill down into the details of spending from various federal agencies.
Recovery.gov, perhaps one of the best-known of the new sites, offering up details of how resources from the Recovery Act are being allocated.
And of course, there's WhiteHouse.gov. You know about that one.
What's remarkable about these sites is not merely that they exist; There had been some efforts to provide this kind of information in the past. Rather, what stands out is that they exhibit a lot of the traits of some of the best tech startups in Silicon Valley or New York City. Each site has remarkably consistent branding elements, leading to a predictable and trustworthy sense of place when you visit the sites. There is clear attention to design, both from the cosmetic elements of these pages, and from the thoughtfulness of the information architecture on each site. (The clear, focused promotional areas on each homepage feel just like the "Sign up now!" links on the site of most Web 2.0 companies.) And increasingly, these services are being accompanied by new APIs and data sources that can be used by others to build interesting applications.
That last point is perhaps most significant. We've seen the remarkable innovation that sprung up years ago around the API for services like Flickr, and that continues full-force today around apps like Twitter. But who could have predicted just a year or two ago that we might have something like Apps for America, the effort being led by the Sunlight Foundation, Google, O'Reilly Media and TechWeb to reward applications built around datasets provided by Data.gov. The tools that have already been built are fascinating. And, frankly, they're a lot more compelling than most of the sample apps that a typical startup can wring out of its community with a developer contest.
More importantly, there's a different attitude about the web and leveraging online communities to help make our government work more effectively. I learned a bit about this first hand when I saw U.S. CIO Vivek Kundra speak at Wired's "Disruptive By Design" conference a few weeks ago:
One of the highlights of that clip happens at just 1:45 into the video, where Kundra outlines a vision where the default setting for information created by the government should be public, not secret. This is the same kind of "default openness" that turned ordinary collecting behaviors on sites like Flickr and Delicious into the foundation for remarkable communities that display phenomenally valuable emergent behaviors. We're seeing this right now, with an organization like Twitter looking to build the feature of retweeting into their own platform, after it having been pioneered by their community.
And it's just as essential to note the way in which these changes have happened. Something like the USA Spending dashboard would have taken half a year or more to deploy in any large-sized corporation; Our government got it done in just a few months. How did they do it? Well, the team in the CIO's office was working nights and weekends, borrowing time and resources as they were able in order to get something useful shipping as quickly as possible. In short, they were working startup hours, with a startup's level of intensity, because they knew they were making something cool and useful.
So What's Next?
While it's exciting to see the remarkable embrace of new technologies that's coming from inside the beltway, there are still some serious challenges that face the new startup-minded tech community within our government. In many ways, they echo the classic challenges that all startups face, but with a unique twist:
Defining a startup's culture is extraordinarily difficult, since there have to be clear values that are expressed in the way people act both in public and behind the scenes. In the case of the executive branch, this is doubly hard because it's redefining a culture which has been well-established for decades. Bringing organizational change and new technologies to an established way of working requires partners and suppliers to change the way they do business, as well.
Acquiring and retaining talent is hard, especially in a city that doesn't have as deep a well of people with tech startup experience. And of course, nobody works in government for the salaries. Fortunately, all of us who are citizens already have equity in this startup.
Marketing has never been the strong suit of those doing the most interesting work in the government sphere. Even some of the smarter folks I know in the tech world had never even heard of the sites I mentioned above, or had never bothered to check them out in much detail. It's going to take concerted effort to get the word out beyond the usual circle of those who were already interested in technology and government.
Of course, these efforts just represent a small start towards the incredible amount of work that remains to be done in making an entity like the U.S. government as responsive and interactive as today's web demands. There will be mistakes, and worse, there will be those who try to politicize this good work, even though our government making smarter use of the web benefits us all whether you agree or disagree with the policies of the present administration.
But I am hopeful, because I've seen a couple of cool applications come out, and more importantly I've seen every indication that, after literally decades of ignoring and neglecting the technology industry that defines so much of our culture, those in political power are eager to embrace those with technological ability. I personally can daydream about Pushbutton-enabling feeds from Data.gov to let us build realtime apps with government data, or deploying blogging tools at the FCC so that we find out about interesting filings from the organization that actually gets the filings. I can imagine all sorts of applications that could be built if we could find "all publicly-available government data on this neighborhood I'm considering moving to".
And while I'm sure that all of these things will get built, as someone who's paying for this stuff with my tax dollars, I am fundamentally most happy about the fact that data generated by my government can be created in a format that fits the way I consume and share information, instead of merely being printed on paper and filed away in a warehouse somewhere. For the way I live, and the way that all of my peers and friends live, the executive branch's new embrace of a startup mentality and the promise of the web means that its work is, for the first time, truly public.
What Works: The Web Way vs. The Wave Way
Google Wave is an impressive set of technologies, the kind of stunningly slick application that literally makes developers stand up and cheer. I've played with the Google Wave test sandbox a bit, and while it's definitely too complex to live up to the "this will replace email!" hype that greeted its launch, it certainly has some cool features. So the big question is whether Wave will succeed as overall in becoming a popular standard for communications on the web, because Google has made an admirable investment in documenting the underlying platform and making it open enough for others to build on and extend. I think the answer is no, and the reason is because the Wave way is not compatible with the Web way.
What do I mean by "the Web way"? Well, if we look at the history of new technologies being adopted to extend web sites and enhance communications, we see a few trends emerge:
Upgrades to the web are incremental. Instead of requiring a complete overhaul of your technical infrastructure, or radical changes to existing behaviors, the web tech that wins is usually the sort of thing that can be adopted piecemeal, integrated as needed or as a normal part of updating one's websites or applications.
Understanding new tech needs to be a weekend-sized problem. For a lot of web developers, long before they start integrating a new protocol or platform into their work, they hack together a rough demo over a long weekend to make sure they truly grasp how it works. And a weekend-scale implementation on a personal site usually translates roughly into a 90-day implementation cycle in a business context, which is a reasonably approachable project size. (In tech, three days in personal effort often translates to three months of corporate effort.)
There has to be value before everybody has upgraded. This is basically a corollary to Metcalfe's Law. While we know networks increase in value as they add more nodes, the nature of web tech is that, in order to be worthwhile, it has to provide value even if the people on the other end haven't upgraded their software or web browsers or clients or servers. Otherwise you're shouting into an empty room.
You have to be able to understand and explain it. Duh.
Now, if we take a look at some examples of what has worked, we can see how various successful technologies have displayed these traits. One great example is feeds. When RSS feeds were new, it was easy to understand their potential immediately, and since I was working at a newspaper at the time, I just spent an afternoon understanding the format and hacking together a quick feed of headlines that anybody could subscribe to. If nobody had adopted feedreaders yet, that was no problem, since there was no cost to just having the feed sit there with no subscribers — the "nobody's upgraded" problem would only result in me having wasted a few hours.
Ajax had a similar adoption pattern. It took a little bit more time to comprehend, but not much more than an afternoon, and the development effort required for adding Ajax enhancements to an application started as a weekend-scale project and has only gone down over time. Following the principles of progressive enhancement, well-designed implementations performed just fine on older browsers or systems that couldn't handle the new features. And most sites that have added Ajax features have done so by adding the requirements as a checklist item in the course of normal ongoing updates, not as standalone efforts to migrate to a new technology.
This brings us to Wave. Wave offers excellent opportunities to extend its core features and to add richness to its "wavelets", and I have no criticisms over its utility as a developer platform that third parties can build upon. But the fundamental Wave protocols are, I fear, a bit too complex to ever be fully and correctly implemented by anyone other than Google. Interoperability is likely to be a challenge that plagues the platform for its entire existence. In short: It's likely that nobody will ever build a fully-compatible clone of Wave that competes with Google's own implementation.
Why is that true? Let's look at what's built in to Wave:
Powerful realtime collaboration features
Unlimited versioning of content
Built around robust XMPP protocol
Combines chat, document editing, and message threading — wikis + blogs + comments + IM
Delivered as a very polished rich user interface
Each of these is a very compelling experience. But a lot of developers' reactions to seeing them was not just "I can't wait to use that!" but also "I want to add that one feature to my own existing application!". And that's where it gets tough. Let's take a look at Joe Gregorio's list of the protocols that power Wave. (Joe works at Google, but made this list before he was working on Wave. I appreciate his research and openness on this topic, and presenting his work here is a tribute to what makes Wave great, not a criticism of his effort.)
Federation (XMPP)
The robot protocol (JSONRPC)
The gadget API (OpenSocial)
The wave embed API (Javascript)
The client-server protocol (As defined by GWT)
That's a lotta stuff! XMPP alone is a bear to implement, let alone to deploy at large scale. (I can't think of anyone outside of Google, Earthlink and LiveJournal who have deployed XMPP to millions of users.) But if you wanted to make another application that truly interoperates with all that Wave can do, combining all of these pieces would just be the starting point.
And people aren't looking for a replacement for email, or instant messaging, or blogs, or wikis. Those tools all work great for their intended purposes, and whatever technology augments them will likely offer a different combination of persistence and immediacy than those systems. Right now, Wave evokes all of them without being its own distinctive thing. Which means it's most useful in providing reference implementations of particular new features.
If a developer wants one of the compelling individual features of Wave, like near-realtime collaboration, they're more likely to use something like (wait for it...) Pushbutton technologies. The infrastructure afforded by the components of the Pushbutton Platform comes nowhere near the richness and polish displayed by Google Wave. Pushbutton isn't even designed to offer the benefits demonstrated by Wave. But to its credit, Pushbutton displays nowhere near the complexity of Wave in its interoperability requirements. More importantly, integrating Pushbutton features into a website or application isn't a monolithic process of building dozens of cutting-edge features, but rather can be deployed incrementally by even non-expert webmasters.
In this context, it might help to think of Pushbutton tech as a "micro-Wave". As Gina Trapani said in mentioning Google Reader's support for PubSubHubBub:
Huh-wha? you ask. Yeah, I know. It's no Google Wave. But that's what makes this exciting. This kind of small Pushbutton implementation is how real web pages will easily use existing technology to notify one another of new updates. The Google Reader/FriendFeed integration is just the first tiny step in what will be a broad deployment of realtime-enabled sites. These sites and services will let one another know when they have new data to share without the sucky inefficiencies of polling. Check out how fast FriendFeed updates when you share an item in Google Reader in the video above.
In short, it's almost zero latency.
Why is this clearly "inferior" technology going to win? Well, as just one example, XMPP is way too complicated for any normal human to deploy. Whereas if you're reading this, you probably already have access to a regular HTTP web server that could talk to a Pushbutton hub. In fact, the only two backers I know who have worked extensively with XMPP are Brad Fitzpatrick and Artur Bergman, who co-created Djabberd. And they are both excited about PubSubHubBub. Realistically, someone like Yahoo might try to do all of this, and inevitably one or two open source projects will try to lash together open implementations of each of these pieces to make a kind of FrankenWave application. There are probably already one or two teams working on the inevitable "Enterprise Wave Server" platforms as well, though I haven't heard about them myself. These efforts may succeed, but that doesn't mean they'll ever be robust enough that people will trust them for communicating on the web.
More to the point, I'm a regular blogger who knows a little bit about scripting on a normal web server. I can poke around the documentation and add a few tweaks to my RSS feed (or, in my case, do nothing and have Feedburner automatically handle it for me), and all of a sudden my blog's feed is part of the Pushbutton web, ready for others to build on. I literally wouldn't even know where to start with the Wave developer documentation if I wanted to integrate it with my site or any of the little apps I like to hack on during a long weekend. What seems more realistic — that someone will figure out a way to incrementally build on top of realtime feeds to enable Wave-like experiences, or that all this talk of Waves, wavelets and blips is going to suddenly become easy to understand.
In short, web-way tech like feeds, Ajax and Pushbutton win because people who make good sites and applications have a place to start with it. Does this mean we get fancy realtime simultaneous editing right away, now that Pushbutton exists? Nope. In fact, Wave might even get the early jump on those kinds of features for web apps, simply because it's pioneered that part of the user experience. But Wave only runs to its full potential on the most cutting-edge web browsers. And there may only be a dozen companies in the world with the in-house expertise to clone the entire complement of technologies underlying Wave in order to make a full-fledged competitor. Worse, the monolithic nature of the Wave experience means it will even be a challenge to make a full-fledged open source competitor to the official Google service.
I hope that Wave succeeds, because I love to see ambition and innovation rewarded. But I think it's mostly likely that Wave's success will be in inspiring people to create similarly compelling experiences by adding incremental enhancements to their existing sites. That's how the web's always advanced in the past.
Apple is justifiably revered in the worlds of technology and culture for creating one of the most powerful brands in the world based on the combination of some key elements: Great user experience and design, and an extraordinary secrecy punctuated by surprising reveals. But the element of secrecy that's been required to maintain Apple's mystique has incurred an increasingly costly price. Apple must transform itself and leave its history of secrecy behind, not just to continue being innovative and to protect the fundamentals of its business, but because the cost of keeping these secrets has become morally and ethically untenable.
Some recent history:
Sun Danyong, a young man in Dongguan, China, who worked for Foxconn, one of Apple's most important iPhone suppliers, killed himself after misplacing a prototype iPhone device.
Apple prohibited the Google Voice application from being distributed on its iTunes application store with no public explanation of why, a refusal to offer any suggestions that could permit the application to be distributed, and no process for appealing the decision.
Apple removed third-party Google Voice-compatible applications by explaining that they violate a policy against applications that duplicate native iPhone functionality, despite this rule being wildly inconsistent in its enforcement. Again, Apple refused to offer any suggestions for how developers could comply with the guidelines, and offered no process for appealing the decision.
The circumstances of Danyong's suicide are murky -- it's possible that he was involved in supplying the iPhone prototype to copycat manufacturers which would create knockoff devices, but the theory has also been advanced that he was merely unable to cope with the stress of the extreme secrecy required for his work. Regardless of the reason for Danyong's death, copycat manufacturers are a fact of doing business in China; It is only the extraordinary veil drawn around the product that makes such disclosures so particularly fraught.
Similarly, every carrier (and nearly every mobile application platform) has some arduous or even capricious limitations on the applications that can be created by developers. But for better or worse, those limitations are spelled out clearly, in a way that developers can anticipate, and decisions to prohibit particular applications are explicit even when they are annoying or offensive to those of us who believe in open platforms.
This means that those of us who support Apple with our dollars and attention are supporting a company that chooses to operate with an extreme and excessive layer of secrecy, even when making reasonable business decisions. This squelching of communication about Apple's products results in customers being unhappy or uncertain of the future value of their purchases, developers being too afraid to bet their livelihoods on a platform whose fundamental opportunities could be destroyed at any time, and suppliers being forced to inflict unreasonable or even inhumane restrictions on their employees. And that's in addition to the incredible stress that Apple employees themselves have had to endure, from missing Christmas to get products ready for MacWorld without even being able to tell family members why they must do so, to public-facing communications staff having to endure the misery of telling developers that their products or businesses are being terminated by fiat, without so much as an explanation.
I'm certain the web's usual contingent of soulless Randists will believe this level of suffering is somehow acceptable despite its moral cost, because The Market has made Apple a success. But there's even a financial argument: Apple spends an enormous amount of money on protecting and obfuscating normal business operations that any other company can do in the open. It's hard to estimate just how much the overhead of this extreme secrecy costs the company, but it's obviously many millions of dollars extra per year. And it will only get more expensive as large-scale realtime communications get more and more commoditized.
The Case for Secrecy
Now, if being ultra-private about announcements has such a terrible cost, then why does Apple go to all the trouble? Apologists would say that Apple gets three significant benefits from its incredible secrecy:
An extremely disproportionate amount of extraordinarily favorable press from its "surprise" product launches
A significant lead time on the rest of the market being able to copy Apple innovations
An intangible benefit to the brand being so tightly controlled by the company
These benefits are real to some extent today, but in each case, the benefit is almost certainly not viable over the long term. Let's look at why:
"But they get so much free press from the element of surprise in their announcements!" This isn't true -- for almost every major announcement of the past several years, we've known the major points days, or even weeks, in advance. In fact, they earn the majority of their press from the extraordinary appeal of their products in design and user experience, as well as the pure showmanship they put into their signature launch events, which are unequalled thus far in the industry.
"But if they don't keep stuff a secret, other companies will be able to copy them!" Other companies already do copy Apple, and always have. And — dirty little secret — Apple has always copied other companies as well. This is a normal part of the business cycle (indeed, before its current bastardization, the patent system was designed to encourage this behavior), and no amount of secrecy will stop it. More to the point, if the only reason people are buying your product is because it has no viable competitors, then your standing in the marketplace is too tenuous to be defended anyway.
"But people love Apple's brand because it's so micromanaged!" This is the most insidious and inaccurate of all the justifications. In fact, since Apple's brand began to recover in the late 90s, two of the greatest and most influential global brands in the world have emerged: Google and Barack Obama. In both cases, they've embraced openness, transparency, and letting their communities define their brand. Despite my belief in my recent pointed criticisms of Google, it's worth noting that a number of high-profile Googlers responded personally, both privately and publicly, to the issues that I raised, all indicating that they took the discussion to heart. And President Obama has taken his penchant for talking things through to such an extreme that it's nearly become a let's-have-some-beers parody of itself.
In contrast, Apple's employees will be too cowed to publicly respond to this post, though I know they'll see it. Partners are tired of being bullied or facing petulant sanctions for accidental disclosures of relatively innocuous bits of information. And eventually, anyone talented and independent-minded enough to participate in the kind of innovation practiced at Apple is going to chafe at being constrained in how they can express themselves.
Real Artistry
Self expression matters because Apple has always explicitly tied itself to the world of the arts and expression. One of my favorite (possibly apocryphal) Steve Jobs quotes is "Real artists ship", a testament to the fact that an invention that never sees the light of day can't affect anyone. But if we're talking about real artists, then let's consider all of their traits.
Real artists also expose themselves, making themselves vulnerable through honest expression so that their audience can see their humanity, and thus form a connection to something universal in all of us. Apple is still holding on to the centralized, Pravda-style public relations that artists used in 1984 when the Mac was introduced. Back then, giant record labels and a few powerful media outlets could tightly control the flow of information around a tiny cluster of superstars. The superstars of 1984 -- Michael Jackson, Prince, Madonna -- subscribed to the doctrine of doing no interviews or press, and having their only communication with the public happen through tightly-managed events where they had total control.
Today's biggest and most influential artists, from Kanye West to Trent Reznor to Radiohead, are very nearly competing to see who can be most transparent. The immediacy and intimacy with which they communicate and create their works is dramatic, and they encourage their communities to get involved in a ritual that Apple used to encourage: Rip, Mix, Burn.
The sad truth is that Apple is still stuck in an anachronistic, 1984 mode of communicating with the world. If Apple doesn't evolve, it'll become a pathetic-looking giant, constantly playing whack-a-mole with information leaks, diminishing its relevance by antagonizing the very creators it has so long sought to identify with. Worse, while the fashions of 1984 might be back in style, the ability to tightly control a message is never going to come in vogue again, and the one thing Apple's brand can't withstand is suddenly becoming uncool. (I'm pretty sure Apple's also had a word or two to say about why today's world shouldn't be like 1984.)
Look Around And Learn
Every company, when facing a serious problem, suddenly starts blogging. From the giant auto manufacturers to troubled banks, it's been astounding to see how frequently companies figure out that embracing transparency yields an enormous improvement in how much their customers and community trust them. When Amazon screwed up by abusing their DRM powers over Kindle owners, they were a little slow to respond, but absolutely flawless in their message when they had Jeff Bezos himself post a simple, straightforward apology to Kindle owners in their own community, complete with open comments for people to respond. And it was an easy leap for Amazon to make -- they have extensive experience not just with consumer-facing blogs, but in talking directly to developers or business partners as well. While much was made of Amazon recalling George Orwell's titles, it's Apple's behavior that is most Orwellian overall.
This lesson isn't entirely lost on Apple; Once in a great while a missive will arrive from on high arrives in the form of a one-page letter from Steve Jobs on a significant issue. And when the debacle of MobileMe's bumbling launch got bad enough, Apple even launched a short-lived blog to address the issue. So it's not impossible that Apple can start to communicate in at least a semi-human, responsive way. Even better, Apple clearly has some parts of its corporate culture that want to do the right thing, as evidenced by its unusual willingness to offer refunds to a variety of disgruntled classes of customers over the years.
But the reason for Apple to embrace some open communications channels isn't merely because of the practical necessity of talking to customers, developers and partners. It's because this is the right thing to do. Apple has long been able to pride itself on being innovative even when the market wasn't demanding bold moves of them. Nothing could be more courageous than for Apple to take a decisive step to redefine a core part of their brand's history to be more in keeping with contemporary communication. Moving from the classic Mac OS to OS X or from PowerPC to Intel would be nothing compared to a transition from ultra-secretive to collaborative and expressive. It would show that Apple has the self-awareness to evolve into a better, more humane organization than they've been in the past.
The reckoning Apple has reached, whether it's admitted or not, is that its secrecy is compromising its humanity. Some of the smartest and most innovative developers on any platform are leaving and taking their creativity with them. The trade press who had embarrassed themselves with their effusive cheering for Apple in the past are rushing to cover absurdities like entire sites being dedicated to Kremlinology about Apple's platform decisions. If losing your cool doesn't move you, Apple, then what about people losing their lives to this domineering, outdated mindset?
It's incumbent upon Apple to do the moral thing here. Treat your employees, customers, suppliers and partner companies better, by letting them participate in the thing most of your products are designed for: Human self-expression. If the ethical argument is unpersuasive, then focus on the long-term viability of your marketing and branding efforts, and realize that a technology company that is determined to prevent information from being spread is an organization at war with itself. Civil wars are expensive, have no winners, and incur lots of casualties.
There is a path out of the current quagmire. Apple can start to see its customers as collaborators, and start to encourage them to use the very Apple products they've purchased as a conduit for sharing messages about the company and its products. Apple's fans have already shown a willingness to create fictitious print, television, and online advertising that exceeds other company's actual efforts in quality while still being slavishly faithful to Apple's brand guidelines. And being an open company doesn't mean that there can't be the occasional big surprise — in fact companies like Google often find it easier to have things "hide in plain site" because so much of what they do is open that the curious often don't dig past the surface to find out what else is going on.
Finally, there is the opportunity for Apple's employees themselves to act as ambassadors for the brand. Frankly, those Geniuses in the Apple stores aren't the most flattering face for the company. But instead of prohibiting all the other thousands of Apple employees from engaging in conversations about their professional lives on the web and in social media, perhaps they could be empowered to express the company's ideas in their own words. That would be an enormous resource that would be unleashed by Apple's evolution into a communicative company.
So Apple: Do the right thing. End your addiction to secrecy.
The Pushbutton Web: Realtime Becomes Real
Pushbutton is a name for what I believe will be an upgrade for the web, where any site or application can deliver realtime messages to a web-scale audience, using free and open technologies at low cost and without relying on any single company like Twitter or Facebook. The pieces of this platform have just come together to enable a whole set of new features and applications that would have been nearly impossible for an average web developer to build in the past.
Background
The most interesting area of new development on the web is the innovation happening around realtime messaging, the ability to deliver updates to a website or application in one or two seconds. While various systems like Yahoo News Alerts or feed readers like Google Reader have offered some simple ways of delivering fairly fast notifications, they are still built on an infrastructure that relies upon requesting a web page repeatedly. These systems do the equivalent of hitting the "reload" button in your web browser over and over.
While those systems have been using these inefficient methods to deliver updates, newer platforms like Twitter, Facebook and FriendFeed have focused on building the infrastructure for efficient large-scale delivery of updates using their own proprietary networks. A lot of attention has been paid to Twitter's 140-character limit, or Facebook's News Feed, but the compelling technology that enables the user experience on these platforms is the immediacy with which updates are delivered. Earlier systems like instant messaging or chat allowed realtime messaging on a one-to-one or small group basis, but it's been harder to deliver those realtime messages to anyone in the world who wanted to receive them unless you had a lot of money, expertise and infrastructure.
Another barrier is that, while there are many different programs and clients that let you connect to Twitter or Facebook with your own applications, there haven't been any free and open options for delivering realtime messages to a large audience if you couldn't, or didn't want to, rely on those companies.
But recently, a few key pieces have fallen into place that make it inexpensive and relatively easy to add realtime messaging as an incremental upgrade to existing websites and web applications. This set of related technologies, which I'm calling the Pushbutton platform, will yield a broad new set of capabilities for users, publishers and developers on the web. Best of all, Pushbutton technologies are free, open and decentralized, meaning that the arrival of realtime on the web will not be owned or controlled by any single company.
Defining Pushbutton
The concept and potential of Pushbutton is a lot like Ajax — it's not a single technology or invention, it's a whole family of technologies, some of which have been in development or deployment for nearly a decade, that together enable this new realtime web. Pushbutton's foundation is built on these systems:
Atom and RSS: The most common feed formats, for syndication on the web
Web Hooks: Simple web services for receiving messages, rather than sending them
Pushbutton systems rely on the web's fundamental HTTP protocol for communication between these component parts. The architecture of Pushbutton message delivery is also simple to understand. Before Pushbutton, in today's systems, when you create a message (a blog post, tweet or other update) that's published in your RSS or Atom feed, every application or site that wants updates from you has to repeatedly request your feed to know when it's updated. You can optionally notify ("ping") some applications to tell them it's time to come collect your new updates, but this is time-consuming and resource-intensive on both sides, especially if you want to notify a lot of people.
In the best case, the system we have now is analogous to a person coming by your house and saying "Hey, there's a new edition of your favorite newspaper today. You should go get it." And then you have to go to the newspaper's printing plant to pick it up. In a Pushbutton web, that person is delivering each story to your house the moment it's complete.
That's because Pushbutton-enabled applications will improve upon the current state of affairs by proactively delivering not just the notification that there's a new message, but the content of the message itself. And instead of requiring all those applications to come to your site to read the update, it uses a hub server in the cloud to pass along the message directly to all the receivers that are interested in it.
You, the Sender, create a message to be delivered via RSS or Atom
Your application gives the messsage to one or more PubSubHubBub or RSSCloud hubs, which reside in the Cloud
The PubSubHubBub or RSSCloud hubs deliver the message to any Receivers, the applications or sites that have requested updates from you
In this way, each time you create a new message, a large number of Receivers can consume that message in near realtime (usually less than a second) without a lot of complexity. This kind of messaging has been possible with custom-built or more obscure technologies in the past, but the Pushbutton ecosystem is a breakthrough for a few reasons:
Sending messages just requires a minor change to an RSS or Atom feed, and a simple, well-defined update notification, instead of major changes to the application where you create your messages.
Receiving messages is also very simple, only requiring a developer to handle incoming notifications of updates.
Most of the system's complexity is handled in the hub servers, which are well-documented, implementable in a variety of programming languages, and built around open code that will likely attract a large developer community.
Most of the scaling effort and expense happens at the hub level, and all current hubs are designed to run on inexpensive cloud systems like Google App Engine or Amazon's EC2.
The software for Sending, Receiving or running a hub is free, open source and available on almost any platform.
Messages sent on Pushbutton platforms are delivered via HTTP, which is familiar to any web developer and runs well on any hosting environment. All requests between the different layers of a Pushbutton system can be made as simple REST calls.
Pushbutton technologies can be adopted incrementally, so that features can be added piecemeal on either the sender or receiver side, without requiring a wholesale upgrade to infrastructure or application architecture.
Who's Behind Pushbutton?
Pushbutton technologies have been created and advocated by some of the most credible and experienced developers of social web technologies. Here's a brief overview of the impressive pedigree of these components:
XML-RPC update pings, RSS and the RSS Cloud ideas were pioneered by Dave Winer, who has been actively developing open implementations of each of these technologies.
Web Hooks have been evangelized by Jeff Lindsay, and have been deployed by a variety of different companies and platforms which all independently developed the technique.
In addition, Google has supported Brad and Brett's development of PubSubHubBub, and enabled it on the Google FeedBurner service. A number of smaller companies are deploying large parts of this infrastructure as well. In short, some of the best reputations in developing open web systems have made Pushbutton possible, from the biggest tech companies to the most steadfastly independent developers on the web.
Related Ideas and Prior Art
There are a lot of existing technologies that have influenced the creation and evolution of Pushbutton technologies; If you're familiar with any of these systems, you're probably already ahead of the curve in understanding part of what Pushbutton is trying to enable.
Twitter Firehose, FriendFeed SUP, TypePad Update Stream: These realtime delivery systems offer up the content of their respective platforms as an unending stream that developers can consume and use in their applications. At the present time, they all have varying licenses and degrees of openness, and slightly different formats for delivering updates, but have proven the utility of the "sending" part of Pushbutton's realtime functionality.
XMPP (Jabber), NNTP (Usenet), IRC: These older internet protocols all delivered various degrees of realtime messaging and distributed messaging capabilities, and can form a very useful base of experience for Pushbutton developers to learn from. In some cases, fundamental architectural choices about security, authentication or architecture were made when the Internet was less populated and less complex, making them inappropriate for today's applications. In all cases, these protocols are less-known by most contemporary web developers, and thus lack familiar toolkits and development resources, which make them quite challenging to deploy in common, inexpensive environments.
TrackBack and Pingback: These systems for delivering updates between blogging systems were very effective in enabling rich distributed conversations in the early days of the blogosphere. These have declined in usefulness due to poor or missing implementations of authentication, which led to spam problems, and a general lack of understanding of their utility by a lot of newer bloggers. Pushbutton may offer an opportunity to restore some of the value of the idea behind these systems.
Reverse HTTP may end up being a useful component of some Pushbutton deployments, as a complement or companion to Comet and related techniques.
What should we worry about?
A format war? If you're familiar with the communities around technologies like feeds, you may know they have a deserved reputation for being contentious and even breaking into heated disputes over arcane details. I don't think that's likely to happen this time, because there are only one or two viable formats for each layer of the platform, and the creators of each part have shown some consistent good-faith efforts to promote interoperability where possible and peaceful coexistence where necessary. In the Ajax community, for example, the "X" in Ajax often stands for JSON instead of XML, but this hasn't hindered its broad adoption at all. I'm also willing to personally commit to try to prevent any kind of interpersonal conflict that would inhibit the adoption of Pushbutton technologies. Worry? No.
Scaling issues? There will inevitably be some learning to do about how to scale the resource-intensive hub layer of a Pushbutton system. But because the hubs live on cloud systems that make enormous amounts of computing resources easily available, because the coders creating the reference implementations of the hub software have great experience making web-scale systems, and because it's relatively simple to introduce new hubs as needed, this will likely not be a gating factor for adoption of Pushbutton. Worry? No.
Intellectual Property Concerns? I'm not a lawyer, and this isn't legal advice. But there has already been a great deal of interest in these systems, and it's likely that any bad actors who were interested in throwing their patent lawyers at this sort of system would probably already be suing people left and right. And the main players who are already involved have shown a consistent desire to make truly open systems that don't have IP encumbrances. Put simply, I think anybody smart enough to invent these kinds of technologies is smart enough to not want to look like jerks by suing somebody for using them. Worry? Probably not.
Competition from centralized systems? Pushbutton technologies are not just free and open, they're decentralized, which is a serious threat to the "lobster trap" model of social software. We can expect serious competition from the centralized networks that are currently building these sorts of systems. If a threat arises to Pushbutton's adoption, this is the most likely source. Worry? Definitely.
Bad user experience? One of the worst things we can do in making use of new technologies is to ignore the social, personal or even political implications of their use. Messages that are immediately delivered can't, by their nature, be erased from all the places they appear. The idea of permanently archiving these types of messages is unfamiliar to a lot of less technically-savvy users. And whenever we see something shiny and new, we have the temptation to use technology for technology's sake, whether or not we're solving a real problem or providing a real value. If Pushbutton gets a bad rap early on despite having tremendous potential, this will be why. Worry? Hell, yes.
Conclusion
I have tremendous excitement about the new realtime era of web applications. While I'm fundamentally an optimistic person, I have great skepticism when it comes to mindless hype about new technologies, so it's with a bit of reluctance that I indulge in some hype myself. But I think the Pushbutton web has the opportunity to give individuals and organizations with distinct and passionate voices the ability to be even more immediate and expressive on the web, and after ten years of publishing on the web, that's the part I love the most.
I have no doubt that some skeptics will say "Pushbutton is just PubSubHubBub by another name", just like they said "Ajax is XMLHttpRequest by another name", and if that's what the super-geeky guys want to believe, I'm fine with that. And I'm sure there will still be some significant technical details to resolve. But I think by giving the overall concept an approachable, understandable name and (hopefully!) an explanation that can be understood by anyone with an interest, it can catalyze interest in a whole new area of innovation on the web. And to be honest, when I see folks like Brad Fitzpatrick and Dave Winer hacking on the same set of problems, I can't help but think something interesting will come of it.
Over the next few days, I'll be outlining some of the opportunities around Pushbutton, espousing more of the philosophy that has the potential to imbue Pushbutton with a bit more meaning than most new web tech, and providing some simple explanations of how you can get started both learning about and taking advantage of these technologies. Most of all, I hope you'll offer your pointed criticisms, thoughtful critiques, detailed corrections and even better ideas. I'll be following the conversation here in the comments, across the blogosphere, and on Twitter using the tag #pshb.
Google's Microsoft Moment
I'm not sure Google's new Chrome OS announcement is that big a deal, or that the eventual product that gets released will actually have that much impact, but it's a useful milestone in marking Google's evolution towards becoming an older company with a distinctly different culture than they used to have.
This is, for lack of a better term, Google's "Microsoft Moment". This is the point when the difference between their internal conception of the company starts to diverge just a bit too far from the public perception of the company, and even starts to diverge from reality. At this inflection point, the reasons for doing new things at Google start to change.
Let me be clear: I don't think Google is "turning evil". Hell, I've caught a lot of flack for the fact that basically I don't think Microsoft was evil. But there are some notable trends going on across Google today that could cause the company to compromise its stated values and that will certainly cause people to think Google is being evil, if not corrected. I'll try to outline a few key cultural indicators from around Google.
Designing for corporate synergy, not for users
Google's recent development work on applications for mobile devices has often been delivered exclusively as applications for their own Android platform instead of as iPhone applications, despite the fact that iPhones are roughly forty times more popular in the marketplace. iPhones are also much more popular outside of the United States than Android, further limiting the actual audience served by these applications. Now, it's obviously good company policy to make sure to support Google's own platforms, and Google does an admirable job of using generic open web technologies where possible to avoid having to choose between platforms at all. But choosing to leave the majority of users in a given market unaddressed because they are on a platform that is not part of your corporate goals is short-sighted and leaves a lingering sense of mistrust.
If you look at Microsoft ten years ago, or even as recently as five years ago, they had a tendency to say "Well, we've got a version that works on Windows Mobile." or "This works on Internet Explorer" and feel that they'd done their job for addressing mobile or the web. Or Windows Media Player would connect to XBox but not to any other systems for sharing media. They were putting their corporate agenda ahead of what the marketplace had chosen as its preferred platforms. But after all these years, Microsoft's internal teams have finally started to develop their web or mobile versions of products to work on competitor's browsers and competitor's mobile platforms, recognizing that they have to go where the users are, instead of favoring only the platforms created by their corporate siblings. Google appears to be headed the other way.
Forgetting what the real world uses, and favoring what's convenient for your own business goals is a quick way to have customers think you don't care, and to indicate to partners or developers that pleasing Google is more important than pleasing customers.
Multiple competing product lines: Chrome OS and Android
This is one of the simplest and most obvious examples, after this week's announcements: Google is now offering not one, but two mobile operating systems. While they undoubtedly share code, I can't help but think back to ten years ago, when Microsoft was vehemently protesting about how much code was shared between the Windows NT/Windows 2000 operating systems and the Windows 95/98/ME operating systems. If I make a screen two inches smaller, should I use Android instead of Chrome OS? If the keyboard works with my fingers instead of my thumbs, I should use Chrome OS and not Android? I know Google is convinced its employees are smarter than everyone else in the world, but this is a product management problem, not a computer science problem.
Changing methods of communication
Within Google, I'm sure the perception is that their public-facing communications are still very "Googley". Now, Google does an excellent job of maintaining and using an enormous number of official corporate blogs in dozens of languages for a rapidly-blossoming number of products and initiatives. But despite my admiration for that effort, and their commendable willingness to forgo the usual boring press releases, the way that the company communicates with the public has fundamentally changed, and not necessarily in a more human direction.
In lieu of blog posts or simple word-of-mouth, as helped popularize the Google search engine itself ten years ago, efforts like Chrome are being accompanied by television ads, complete with all of the production values of primetime TV. Instead of launching a new developer initiative by promoting an SDK on their blog, Google is filling convention centers, Apple-style, with day-long developer presentations and an Oprahesque giveaway of free phones under every seat. Instead of white papers, there are highly-produced comic books being distributed to the press to explain the value of Chrome.
Now, I actually support these types of outreach. Getting outside of the insular tech bubble requires higher production values and clearer messaging. But when Google evokes Apple or Microsoft or Oracle in its style of communicating ideas, and when cell phone ads on TV say "Powered by Google", an average consumer's conception of Google essentially shifts to seeing this company not as "those guys who do the search engine" but instead as another consumer electronics company, like Samsung or Sony, but a little more hip.
This would be okay, except that I doubt Google's internal self-image as an organization has changed to reflect this new reality. "We're not like some giant company with flashy TV ads — we're just a bunch of geeks in Mountain View!" And while that might be true for the vast number of engineers who define the company's internal culture, the external impression of Google being just another tech titan like Microsoft will gain footing, making the audience for Google's messages less tolerant of ambiguity and less forgiving of mistakes.
Only the last generation of companies can be evil, not us!
Though it's almost impossible to picture now, in the era when Microsoft was formed, IBM was synonymous with an almost Orwellian dominance of information technology. It's been a full 40 years since the antitrust actions against IBM, and IBM is seen as a bastion of open-sourceness now, but Microsoft's founding mindset clearly was shaped with the idea that "those old guys from the last generation are evil, and we're the nimble, smart upstarts who are going to humanize this industry". Sound familiar?
Though it's hard to believe, the FTC's first investigations against Microsoft began eighteen years ago. When Microsoft reached its apex in terms of public perception and industry respect, with the launch of Windows 95, the culture inside the company still largely saw themselves as upstarts against old, proprietary behemoths. Though Microsoft's headcount has increased fivefold since then, at the time of Windows 95's launch, they had about 17,000 employees.
Google's headcount just passed roughly 20,000 employees. And most of those staff members are firmly convinced that evil, or at least incompetence, is a trait of the last generation's dominant tech player: Microsoft. The idea that developers or customers might start to bristle at their dominance is met with the (true, yet irrelevant) argument about how open their data and platforms are. Eric Schmidt said yesterday that Chrome OS is so open that Microsoft could make Internet Explorer for it, though of course the effort of porting the browser would be prohibitively complex. By neatly inverting the framing of the conversation ("We didn't bundle a browser with our OS, we bundled an OS with our browser!"), Google's avoided having to confront the parallels between this moment in their corporate culture and Microsoft's similar moment of ascendancy 15 years ago.
Still haven't developed Theory of Mind
And finally, as I outlined two years ago, Google still hasn't developed theory of mind. From my piece then:
This shortcoming exists at a deep cultural level within the organization, and it keeps manifesting itself in the decisions that the company makes about its products and services. The flaw is one that is perpetuated by insularity, and will only be remedied by becoming more open to outside ideas and more aware of how people outside the company think, work and live.
Worse, because most of the dedicated detractors of Google have been either competing companies or nutjobs, it's been hard for Googlers to take criticisms seriously. That makes it easy to have defensiveness or dismissal of criticisms become a default response.
Conclusion
Google has made commendable steps towards communicating with those outside of its sphere of influence in the tech world. But the messages will be incomplete or insufficient as long as Google doesn't truly internalize and accept that its public perception is about to change radically. The era of Google as a trusted, "non-evil" startup whose actions are automatically assumed to be benevolent is over.
Years ago, GMail introduced context-sensitive ads and was unfairly pilloried for being anti-privacy or intrusive. And while there have been a few similar hand-slappings along the way, Google's never faced a widespread backlash against their influence or dominance from average consumers yet. Today, protestations of "but it's open source!" are being used to paper over real concerns about data ownership, and the truth is that open code doesn't necessarily imply that average users are in control.
And ultimately, once a tech company becomes dominant in its space, it's susceptible to a kind of reverse Hanlon's razor: Anything caused by stupidity or carelessness will instead be attributed to malice. Similar to the Law of Fail ("Once a web community has decided to dislike an idea, the conversation will shift from criticizing the idea to become a competition about who can be most scathing in their condemnation."), Google is entering the moment where it has to be over-careful not to offend, and extremely attentive to whether they are treading lightly.
Is Google evil? It doesn't matter. They've reached the point of corporate ambition and changing corporate culture that means they're going to be perceived as if they are. Whether they're able to truly internalize that lesson, accept it, and act accordingly will determine if they're able to extend their dominance in the years to come.
Update: There's been a phenomenal reaction to the ideas discussed here. I rounded up a lot of the responses in a follow-up post. But it's also worth noting that a number of people from both within and without Google have pointed out that in many cases, the release of an Android application has preceded its counterpart iPhone equivalent due to delays in Apple's opaque approval process for applications on that platform, or because the Android applications were only created as hobbyist projects by Googlers in their free time. Similarly, a number of people have pointed out significant differences between Chrome OS and Android, such as the primary development environments (HTML5 and Java, respectively), memory limitations for applications, and the distribution model.
While I've certainly not meant to gloss over any of these clarifications as insignificant, and appreciate the additional information, the key argument I'm advancing here is about the overall impact of changes in Google's culture and perception. Many more examples can (and have) been identified to support that larger trend, and I'm pleased that the larger dialogue has focused on that bigger issue, inspiring some great conversation.
Free Criticism, Science After Data, and Airport Books
When I saw Malcolm Gladwell doggedly dissecting Chris Anderson's upcoming "Free: The Future of a Radical Price" (see Chris' response here) my first reaction was: Brilliant! Chris Anderson is editor-in-chief of Wired, and Malcolm Gladwell is a top brand name at The New Yorker, and as corporate cousins, clearly Condé Nast's publicity machine must have engineered this beef, trying to boost sales of both their titles through a completely manufactured rivalry.
Their past titles have been champions of what I call the "Airport Books" genre: The elite class of business titles that I see sold in airport newsstands next to the magazines and crappy romance novels. (I might have unknowingly stolen "airport books" from someone else, but I can't find a citation.)
Alas, I'm assured that this particular contretemps isn't a planned corporate PR stunt. (Though I know lots of nice folks at Condé, they don't seem to mimic street-level hip hop marketing as often as one might hope.) Instead, it seems the criticism and counter-argument are sincere.
The core of Gladwell's argument is simple: "Free" fails to provide data to support its claims about the future of pricing, using anecdote and confident assertion in place of actual evidence. In his objection to this methodology, Gladwell seems uncharacteristically strident, compared to his usual measured tones. Whenever I see somebody getting their dander up, I think of one of the first things I ever blogged about ten years ago: We hate most in others that which we fail to see in ourselves. Ah hah!
Let's see what criticisms have been leveled at Malcolm Gladwell's The Tipping Point, Blink and Outliers, the juggernauts of the airport book genre:
The IUP Skeptical Inquirer's Wesley Cecil has a review of Blink which offers this up: "Gladwell relies heavily on anecdotal evidence. The centerpiece of each chapter is a short story or series of stories that are supposed to illustrate some aspect of his theory of snap judgments. ... [I]ntuitive feeling triumphs over careful study. ... [O]ne case does not an argument make."
More definitively, the New York Times' Michiko Kakutani made this point in her review of Outilers, but extends it to apply to all of Gladwelll's books: "[His] books are filled with colorful anecdotes and case studies that read like entertaining little stories. Both use PowerPoint-type catchphrases (like the 'stickiness factor' and 'the Rule of 150') to plant concepts in the reader’s mind. ... 'Outliers' Mr. Gladwell's latest book, employs this same recipe, but does so in such a clumsy manner that it italicizes the weaknesses of his methodology. ... [His examples are] all based not on persuasive, broadband research, but on a flimsy selection of colorful anecdotes and stories."
Joel Spolsky keyed off of Kakutani's review on his popular blog: "what's been driving me crazy over the last year... an unbelievable proliferation of anecdotes disguised as science, self-professed experts writing about things they actually know nothing about, and amusing stories disguised as metaphors for how the world works." (Bonus points to Joel for swiping en passant at airport book titan Thomas Friedman and his cartographic ironing board.)
Kevin Arthur's post built on Joel's rant, offering a slightly more measured, but still critical analysis: "I feel like clarifying my opinion on this... I think there is great value in pop science books, in articles written by non-experts, and in anecdotes. I read Joel's piece not as a rant against all those things but against those things badly done."
Peter Coclanis at Open Letters just gets downright mean: "[L]et me say from the get-go that my goal in this piece, which focuses on Outliers, is to demonstrate at once how wildly overstated such just-jacket claims are and how egregiously incomplete, insubstantial, and unconvincing Gladwell’s explanation of success actually is. His methodology stinks, too, and, from his dust-jacket photo, he appears to need a haircut." I know what fun it can be to bash someone from afar on the web, but I bet Coclanis is a lousy dresser. Just sayin'.
And not to belabor the point, but let's close up with Isaac Chotiner in The New Republic, taking a stridently snarky look at Outliers: "By the time Gladwell reaches his penultimate chapter, he is in full inspiration mode, and impervious to all forms of critical thinking. ... Here is the Gladwell method nicely on display: a questionable assumption, a partial walk-back of an earlier claim, and finally another questionable assumption synthesizing the half-reversal."
My goal is not to ennumerate all of the criticisms of Gladwell's books — I enjoyed reading all of them, and I like his New Yorker pieces, and that's kind of all I would ask of the guy. But I can't help but wonder if being ceaselessly criticized for using assertions and anecdotes in lieu of hard statistical data has left him much more inclined to criticize others for using the same technique.
I haven't had a chance to finish reading Free yet, but I am sure that both of these authors' books absolutely do lean more towards anecdotal evidence than statistical proof. And honestly, it's okay that these books don't necessarily follow the tenets of hard science. In many cases, they're arguing that a cultural trend is becoming true, or is about to become true, and the reality is that asserting that these trends are ascendent actually helps them come true. In short, these are books designed to create culture, presented in the guise of reporting on culture. I like that!
But of course there will always be those who disagree with the idea of starting from a premise first, and then finding examples to support it. Perhaps the last word in favor of using hard data to support social observations may be from a story package in Wired a year ago, which was headlined "The End of Science" and anchored by a story called The End of Theory:
This is a world where massive amounts of data and applied mathematics replace every other tool that might be brought to bear. Out with every theory of human behavior, from linguistics to sociology. Forget taxonomy, ontology, and psychology. Who knows why people do what they do? The point is they do it, and we can track and measure it with unprecedented fidelity. With enough data, the numbers speak for themselves.
... But faced with massive data, this approach to science — hypothesize, model, test — is becoming obsolete. ... The new availability of huge amounts of data, along with the statistical tools to crunch these numbers, offers a whole new way of understanding the world. Correlation supersedes causation, and science can advance even without coherent models, unified theories, or really any mechanistic explanation at all.
The author of this compelling argument in favor of using overwhelming amounts of data to help replace formulating theories about human behavior? Former scientist Chris Anderson.
Bonus link: If you're interested in actual debate about the content of the book, Mike Masnick's excellent overview over at TechDirt is a must-read.
In that era, before meetups and tweetups and mass political movements organized by bloggers, Brad recognized that not only were there real humans interacting on these sites, but that all of us who shared our thoughts online were part of a creative community every bit as legitimate and unifying as his work in theater.
While those systems have been using these inefficient methods to deliver updates, newer platforms like Twitter, Facebook and FriendFeed have focused on building the infrastructure for efficient large-scale delivery of updates using their own proprietary networks. A lot of attention has been paid to Twitter's 140-character limit, or Facebook's News Feed, but the compelling technology that enables the user experience on these platforms is the immediacy with which updates are delivered. Earlier systems like instant messaging or chat allowed realtime messaging on a one-to-one or small group basis, but it's been harder to deliver those realtime messages to anyone in the world who wanted to receive them unless you had a lot of money, expertise and infrastructure.
I have no doubt that some skeptics will say "Pushbutton is just PubSubHubBub by another name", just like they said "Ajax is 
