Paul Knox asks, as a news organization, what do you actually do when someone dumps 250,000 secret diplomatic cables on your electronic doorstep? How do you sift through the bonanza, wash out the nuggets, figure out what’s genuine and decide what to publish? And how do you go about prospecting for more?

I was hoping to find a few answers at the Dec. 16 Secrecy and Journalism conference put on by the Nieman Foundation at Harvard University. I did find some, but what also emerged was a picture of how technology, imagination and the expectations of audiences are changing the way closely held information becomes public.
 
Journalists have always used documents as the starting point for high-impact reporting. Some are lying around in open pits waiting to be scooped up. Others are hidden deep underground in hard-to-reach veins. Still others are dumped on our desks by people with a wide variety of motives.

What’s new in the digital world is the sheer volume of documents created, duplicated and retained. They proliferate like rabbits on fertility drugs. For secret-keepers this means the risk of disclosure – accidental or intentional – is much greater than it used to be. For journalists and citizen, it translates into opportunities.

At the Nieman event, executive editor Bill Keller of The New York Times shared details of how his newspaper handled the latest Wikileaks trove. The Times and several other newspapers received a quarter of a million cables written by U.S. diplomats reporting from dozens of countries on politics, personalities and military affairs. The files – intended to remain secret for decades – contained frank assessments of world leaders and judgments about events, many of which seemed to be at odds with public statements and official policy. The Times had earlier dealt directly with Wikileaks, receiving dumps of cables about Afghanistan and Iraq. But Wikileaks founder Julian Assange was upset with the newspaper for several reasons, and this time the relationship was indirect.

It was out of the question for anyone at the Times to read all the documents. So Keller’s staff turned them into searchable files and created a data base to hold them. Specialists in computer-assisted journalism, working with military and diplomatic beat reporters, built searches that seemed likely to draw sensitive, explosive or just plain interesting needles from the haystack. This, Keller said, produced “clumps” of cables in which, it was clear, there were newsworthy stories to tell. 

Then
the process of verification and story development began. “They had to establish
that [the information] was real,” Keller said. Veteran reporters familiar with
the terminology and format of such cables pronounced them plausible. The Times scrubbed everything it intended to
publish to remove names of private individuals and non-decision-makers who
might become reprisal targets. “It was obvious that a low-level elder in
Kandahar would probably be killed,” Keller said. “In most case we managed to
tell the story without publishing the actual documents.”

The next step was to give government officials a chance to point out errors or challenge the veracity of the documents as received, and to make arguments as to why they shouldn’t be published. “Sometimes they wanted us to omit things that were just embarrassing,” Keller noted – but that didn’t happen. In general, he said, the Obama administration’s reaction to the affair was “sober, professional and grown-up … [they] resisted the opportunity for an orgy of press-bashing.” This, Keller added, was in marked contrast to 2005, when George W. Bush told Keller he would be held personally responsible for the next attack against the United States if the newspaper went ahead and published details of anti-terrorism bank surveillance procedures. 

Of course Keller talked about more than the mechanics. He laid out the reasons why publishing so-called secret information should not be considered a criminal act, an ethical lapse or even particularly rude. Chief among them is that relatively little government information really needs to be secret. Elaborate classification regimes are more about inter-government and inter-agency rivalry than protecting individuals, preserving commercial confidentiality or conducting diplomacy.

Secrets are made to be revealed. Political capitals are bazaars in which information is the hottest currency around. One of Keller’s predecessors, Max Frankel, put it this way in a court affidavit filed in support of the Times’ plans in 1971 to publish material from the “secret” Pentagon Papers: “Government hides what it can, pleading necessity as long as it can, and the press pries out what it can, pleading a need and a right to know. Each side in this ‘game’ regularly ‘wins’ and ‘loses’ a round or two.”

Putting the point more bluntly to the Nieman audience, Associated Press executive editor Kathleen Carroll characterized the position of government officials the world over as: “I’m in office, you’re not, and what I’m doing is none of your damn business.”

She also underscored just how high the stakes can get. The late French president François Mitterrand was obsessed with covering up his government’s malfeasance and his own private life, Carroll noted. In 1985, France denied any involvement in blowing up Greenpeace’s Rainbow Warrior as it prepared to sail to the French South Pacific nuclear-testing zone. Twenty years later it emerged that Mitterrand himself had ordered the attack. During his presidency, an elite anti-terrorism squad was detailed to spy on people who knew about his adulterous affairs and out-of-wedlock daughter.

Like Keller, Carroll was sceptical about the game-changing claims being made for Wikileaks and the World Wide Web. Technology can make original documents available to a wide public, she said, and “a few hardy folks … are applying journalistic principles in their own self-publishing.”  But few outside the mainstream have the motivation or resources to “dig out what governments are doing.”

This kind of reporting, Carroll observed, can be hard, boring, repetitive and lonely. People who feel threatened can be mean and rude. Governments are fighting back, making investigations longer and more expensive. Large news organizations know the law and can develop counter-strategies, but few people who aren’t paid to do this kind of work have the stomach or the stamina for it.

“Imagine the barrier if you’re just an average citizen who wants to ask the government questions,” Carroll said. And how much more so for those who live under authoritarian regimes, or in new or threatened democracies? International journalists currently studying at Harvard spoke about the difficulties they face – sometimes from their own co-workers.

How does that work? Stefan Candeia, founder of the Romanian Centre for Investigative Journalism, told of colleagues at news organizations hacking into others’ computers and selling the information they found to the targets of investigations. Alejandra Matus, who fled Chile to avoid jail after writing a book on abuses in the judicial system, said journalists can be corrupted as well as business people and politicians, and that in her experience a mainstream newsroom was not a place where much could be learned about journalism.

Nevertheless, even where conditions are less than ideal, investigative journalism can thrive. Matus got courageous, conscientious people inside the system to recount their stories, then added more reporting to provide context and analysis. Panelists from South Africa and Cambodia spoke about uncovered documents that proved the existence of abuses, irregularities and cover-ups.

American reporters, too, recounted their experiences – some from the new foundation-supported centres that are moving into investigative work as traditional news organizations stampede for the exits. Maggie Mulvihill, a journalism veteran now with the New England Center for Investigative Reporting, trains young newshounds in the techniques of prospecting for data and sifting through it. She fears they don’t understand that “government works for them and not the other way around.”

If the story of Wikileaks proves anything, it’s that the documentary base of news is assuming new forms and styles that all of us must become familiar with. Toward the end of the day there were a couple of glimpses of what the future might look like.

Aron Pilhofer, editor for interactive news technologies at The New York Times, is also the co-founder of Document Cloud , which describes itself as “an index of primary source documents and a tool for annotating, organizing and publishing them on the Web.” News organizations have used it to analyze, annotate and publish original material alongside the stories based on it. Software makes the documents searchable and extracts key words and names, which are linkable across a data base. He outlined a vision of decentralized document storage in which no single organization could control the fate of a project (as Amazon and others have sought to do with Wikileaks).

Science writer John Bohannon, who has written extensively on the casualty tolls in Iraq and Afghanistan, warned that those with a stake in keeping information secret aren’t likely to stand still. Cryptographic technology allows indelible time-and-place stamps to be embedded in documents, he said, and programs that analyze individuals’ use of language can make it hard to conceal authorship.

We also heard about a project called Basetrack, in which a team of writers and photographers plans to work with a U.S. Marine battalion to chronicle its entire deployment in Afghanistan on social media. This new journalistic form may look as if it transcends the documentary base, but in another sense it transforms it, creating a semi-official public record of action and experience on a non-traditional platform.

What came across through the day was a sense that although the mechanics of tipping off, whistleblowing and information processing have changed dramatically, the tasks and ethos of journalism have not. News advances “by inches and feet, not in great leaps,” Keller said. “The age of open data is more a promise than reality,” said Clint Hendler, a staff writer at the Columbia Journalism Review, adding that just because data is accessible doesn’t mean it’s comprehensible.

Matus, the Chilean author, said Wikileaks and similar phenomena have opened new doors for journalistic watchdogs. But she asked: “Who is going to pay professionals to study, to go and read boring documents and make sense out of it?” That puzzle has not been solved, and Matus added: “I don’t think Wikileaks helps solve it.”

Paul Knox is a journalism professor at Ryerson University and former chair of its School of Journalism. He spent more than 30 years as a newspaper reporter, editor and foreign
correspondent, and continues to write and broadcast on international
affairs and media issues.