Assignment Zero: A Look Back and Lessons Learned

It’s not often that I’ll just run another story in full. But then again, it’s not often that I’m interviewed and quoted too.

Today Wired ran a story: Did Assignment Zero Fail? A Look Back and Lessons Learned.

Howe, who I am also helping with his book on Crowdsourcing did a good job with the article. (He also wrote a follow up piece on his blog which is perhaps the most public flattery I’ve ever received from a professional colleague).

When people ask me, I have to admit — Assignment Zero was not a complete success. It was a hard six months. I worked my tushie off, not everything got accomplished that we wanted and during the thick of it all, my life was very hectic. It was an uphill climb — but man, what a view.

And when I look back on it now, the project seems like an amazing success despite all the obstacles. Things did work when we really needed it to. As Jeff quotes me “the crowdsourcing gears kicked in.”

More importantly, I know it’s cheesy to say, but I made some friends. I am still in contact with many of the AZ contributors (and if you are an AZ Alumni — never hesitate to contact me), and I’m happy to think that through this project I’ve met scientists, programmers, artists, teachers, etc etc, and have worked with all of them.

I also believe that the final package of Assignment Zero (80 interviews and nine feature stories) is the largest collection of work on crowdsourcing to date. (hooray for us!).

And finally: While not everything worked at Assignment Zero, the beauty of the project was what we learned through the experiment (some of my lessons). I often said “AZ was like throwing pasta at the wall and seeing what sticks. The great part about the project — there were no noodles just hanging by a thread. All the ideas either fell to the ground or stuck perfectly. In my mind — if another project like AZ were to start from the ground, with the right organization, promotion and topic, I’m convinced it could work.

I will add this before I cut and paste the story. In one quote I sound overly harsh of the volunteer editors.

“What we really needed were people who understood online organizing,” says David Cohn, an Assignment Zero editor. “But many of the editors just didn’t have much experience with the internet.”

I won’t deny that I said this. But I also think it deserves a bit of explanation. That comment was for a few individuals, explaining that just having a warm body and calling them an editor doesn’t solve a problem.

The vast majority of editors that stuck with us were absolutely amazing. They volunteered their time and efforts to work with a crowd — and I can’t thank them enough.

There were a few (not a majority) who required me to teach them how to actually join and log into a community site. And you can imagine — if I had to teach them that, they weren’t exactly the most useful online community organizers.

But again, the vast majority of editors that we had were awesome. Some of them were absolutely clutch. Without a doubt, without the help of some of those editors, Assignment Zero would have been completely stagnant.

But anyways: Back to the story. (also see Tish Grier’s take on AZ)

Ambitions ran high when Wired joined forces in January with new media incubator NewAssignment.net to try a novel experiment in pro-am journalism.

Our goal: Have a crowd of volunteers write the definitive report on how crowds of volunteers are upending established businesses, from software to encyclopedias and beyond.

Citizen media initiatives are a hot topic in the media, and the new project, christened Assignment Zero, was widely reported. The New York Times gave it a lengthy, if skeptical, treatment. Would the crowd prove too tough to manage, the reporter asked?

Six months later, the jury is in, and the answer is mostly yes. Although Assignment Zero produced a strong body of work, consisting of seven original essays and some 80 Q&As, the real value of the exercise was discovery. We learned a lot about how crowds come together, and what’s required to organize them well. But many of the lessons came too late to help Assignment Zero.

In the 12 weeks the project was open to the public, it suffered from haphazard planning, technological glitches and a general sense of confusion among participants. Crucial staff members were either forced out or resigned in mid-stream, and its ambitious goal to produce “the most comprehensive knowledge base to date on the scope, limits and best practices of crowdsourcing” had to be dramatically curtailed in order to yield some tangible results when Assignment Zero ended on June 5.

And yet for all this, it might best be considered a highly satisfying failure. It fell far short of the original aim of producing over 80 feature stories, but in over a dozen interviews conducted by phone and e-mail, contributors uniformly described a positive, “though frequently exasperating, experience.” But then, Assignment Zero was full of contradictions.

Jay Rosen, the NYU journalism professor who initiated the project, has written that only 28 percent of Assignment Zero worked. That’s sobering assessment, but one that reflects the view expressed by other staffers and contributors. My own view is a little rosier: I found at least three-quarters of the Q&As to be equal to or exceeding the quality of thought and insight found in any national magazine. And if Assignment Zero failed to clear the especially high bar it set for itself, the fact it produced so large a body of work still speaks to the considerable potential of crowdsourced journalism.

“It’s like throwing a party. You program the iPod, mix the punch, dim the lights and at 8 o’clock people show up. And then who knows what is going to happen” Lauren Sandler, Assignment Zero editor, to New York Times reporter David Carr

Rosen has been a champion of bringing non-professionals into the production of journalism for years. In 2006 Rosen began conceiving of a vast project that would entail a large number of both professional and amateur contributors. In November, Rosen flew to San Francisco to meet with Wired News editor Evan Hansen. Newly acquired by Conde Nast, the publisher of Wired magazine, Wired.com was looking to experiment broadly and boldly.

So Hansen was looking for a platform to explore citizen journalism, and Rosen was looking for funds to create such a platform. The two decided on the rough scope of a project. It should be called, Rosen decided, Assignment Zero, a name indicative of the still-nascent character of citizen journalism. And the subject Rosen wanted to cover was the crowd itself the ways in which communities were coming together to create great things. Having coined the word crowdsourcing in a Wired magazine article earlier that year, I was brought in as the Wired writer and representative assigned to the project.

The choice of subject was a decision we would come to regret. The topic of crowdsourcing was too nebulous “too new” to gain immediate traction. One thing any volunteer project must inspire, be it citizen journalism, an open-source programming project or simply an AIDS drive is passion. Using the crowd to investigate crowdsourcing inspired, by contrast, confusion.

One of the key problems confronting the project managers was technological. How to build a site that would allow large numbers of contributors to sign up and participate in meaningful ways?

The site developers turned to was Drupal, an open-source publishing system that’s become one of the leading platforms for community-driven projects. The designers built in numerous topics for contributors to colonize when they arrived. But the AZ team chose to hold off recruiting editors until after the launch, with the result that when contributors signed up, they essentially arrived at a ghost town.

The flood of volunteers made Assignment Zero’s design flaws quickly apparent. Potential contributors which numbered roughly 500 after the first week were routed to a single Assignment Zero staffer, a former WashingtonPost.com editor, Steve Fox.

The net effect was to put the organizational onus on the volunteers themselves. Baffled by the overarching concept of crowdsourcing, confused by the design of the website and unable to connect directly to a manager or organizer, most of the initial volunteers simply drifted away. “What we learned, says Rosen, is that you have to be waaaay clearer in what you ask contributors to do. Just because they show up once doesn’t mean they’ll show up over and over. You have to engage
them right away.”

Over the course of the next two weeks, around 30 volunteer professional editors were assigned to manage various topics. Theoretically this should have solved Assignment Zero’s organizational problems. But this presented a new set of hurdles. For one, each editor needed to be trained to use our implementation of Drupal. But even once they were up to speed there was a lack of understanding of how open-source projects tend to work.

What we really needed were people who understood online organizing, says David Cohn, an Assignment Zero editor. â??But many of the editors just didnâ??t have much experience with the internet.

After roughly six weeks â?? halfway through its run Assignment Zero reached its nadir. Most of the initial volunteers were gone, the majority of topic pages were deserted and communications between staffers, volunteer editors and the few contributors that remained were uneven, resulting in frequent misunderstandings. A drastic change was
required.

“We are not a project without a mission.”

Jay Rosen at the midway point of Assignment Zero, in an e-mail to contributors.

Although much had already gone awry, Rosen and the Assignment Zero team had one advantage working in its favor: hard-earned knowledge from six weeks of trial and error.

The first order of business was a site redesign. After consultation with the rest of the team, Cohn, who doubled as webmaster, rebuilt each topic page to include social networking features. The relevant editor’s picture and contact e-mail were placed at the top of each page, and each topic area now included a forum. The idea was to make each topic a sort of home page, a community gathering-place.

The effect of this reorganization was felt immediately, as contributors could now collaborate openly with each other and review one another’s reporting. This certainly reinforced one of the lessons that was learned from reporting on various crowdsourcing projects: Essentially, it’s all about the community.

However, the majority of topic pages had yet to attract a base of interested volunteers. This demonstrated another lesson: The community controls the scope and direction of the project. “We had to jettison most of the topics we’d started off with, says Cohn.” Instead, we concentrated on the topics that people were most clearly interested in.

At about the same time the Assignment Zero team made the decision to shift the goal from producing scores of feature stories to producing scores of interviews. Asking contributors to write the story on open-source car design had all the appeal of asking people to rewrite their college term papers. Asking them to talk to someone they admire and respect was met with a far warmer response.

All efforts were now focused on interview week, intended to be a five-day flurry of activity, with contributors conducting and transcribing roughly 50 Q&As. Given a concrete, clear task, Assignment Zero volunteers quickly rallied. “The crowdsourcing gears kicked in,” says Cohn.

Meanwhile, the half-dozen topic areas still functioning showed signs of life. The volunteer editor in charge of crowdsourced religion, the Detroit Free Press David Crumm, even fulfilled Assignment Zero’s original ambitions. By rewriting assignments and going out and gathering a community around his topic page, Crumm essentially made his topic his own party. â??He had
40 people reporting for him at one point, and others doing writing, says Cohn.

With interview weeks at hand, contributors became more active, helping to schedule interview times and suggesting additional subjects. Soon the list of sources had grown from 50 to more than 80.

In the final two weeks of the project, Assignment Zero began to resemble a professional journalism outfit. Editors and contributors discussed potential questions; the interviews were scheduled, conducted, transcribed, filed and edited. And as they began pouring in, it became clear that many would exceed the team’s diminished expectations.

In my rough count, at least 60 of the 80 interviews would stand up to professional scrutiny, which is to say the interviewer was

well-informed, asked challenging questions and managed to elicit interesting (and occasionally fascinating) commentary from his or her subject. The gems are too numerous to mention, but a few standouts include Randy Burgeâ??s interview with Innocentive co-founder Alpheus Bingham and J. Jack Unrau’s inteview with crowdsourcing scholar Karim Lakhani. But these are the tip of the iceberg.

What the interviews make clear is that contributors volunteered to tackle subjects about which they were passionate and knowledgeable. In this they held a considerable advantage over professionals, who often must complete interviews with little time (or inclination) for advance research.

It is a community’s ability to allocate intellectual resources organically in this way that can make it a more efficient machine than a traditional, hierarchical organization. In the final stretch Assignment Zero embodied the best of crowdsourcing as well as studying it.

“Why are these people willing to work for free?” Jay Rosen in Wired News, on the launch of Assignment Zero

As flawed as the project’s mechanics proved to be, the final result met the goal of being the most comprehensive exploration of crowdsourcing to date. So what did that exploration reveal? Some common themes emerged:

One is that Assignment Zero isn’t the only project still fumbling its way toward a working model. A mutual embrace of experimentation runs throughout all the interviews, a cheerful admission that the kinds of collaborative efforts enabled by the internet are both powerful and also in their infancy.

We need to try different things, the prominent political blogger Susan Gardner told Anna Haynes. “The process of elimination is undervalued. What’s wrong with trying something, assessing and taking it as positive information that this particular model doesn’ t work. That’s not failure. That’s important information.”

And one model that doesn’t work is attempting to use crowdsourcing simply as a cost-saving measure. Communities must be cultivated, respected and deftly managed if they are to come together to create economic value. This takes talented staff, and a set of skills not taught in journalism or business schools.

“(Crowdsourcing) rarely works as a free-for-all,” Charles Leadbeater, the author who wrote The Pro-Am Revolution,
told Lilly Evans. “It thrives on decentralized cooperation and people taking responsibility for working together. So it needs leadership that makes the conditions for that possible.”

Assignment Zero, which required a considerable expenditure to bring it to completion, is one illustration of the labor required to make a collaborative project work. But then so is the open-source cinema project, A Swarm of Angels, or the crowdsourcing company Cambrian House.

Another conclusion that can be drawn is that people who participate in crowdsourcing do so for various reasons. In discussion with Assignment Zero interviewer M. Six Silberman, the legal professor and economic theorist Yochai Benkler said that simply attempting to harness the crowd by offering explicit rewards is a mistake: “People who relentlessly pursue the design of such systems, rather than systems based on transparency and conversation, will end up losing.”

What is clear from reading the entire set of interviews is that people participate for a complex web of reasons, including the enhancement of one’s status within the community, the opportunity to learn or perfect a skill, the chance for financial gain or simply the intangible rewards from working with others toward a shared goal.

As Howard Rheingold told Scott Rosenberg, an Assignment Zero volunteer and as a co-founder of Salon.com, no stranger to professional journalism: “(We) need some way of aggregating individual actions into something that’s greater than the sum of its parts.  “Let’s find out by observation and experiment what the limits and the capabilities of these phenomena really are.”

Let the experiments continue.

Jeff Howe is a contributing editor at Wired magazine.
He is currently writing a book about crowdsourcing that will be
published in June 2008, and he keeps a blog on the subject at
Crowdsourcing.com (http://crowdsourcing.typepad.com/cs/).