The Future of Comments

When many news websites were shutting down their comments sections, Alaska Dispatch News executive editor David Hulen was determined to keep his. Like every news site, ADN’s comments had problems, but Hulen had also seen the good that comments could do, as well as how they could serve as an important community forum for a large state that often feels like a small town.

Six years ago, when he was an editor of the Anchorage Daily News (purchased and absorbed by Alaska Dispatch in 2014), Julia O’Malley wrote a seven-part series about a heroin addict’s journey toward recovery. Concerned that someone who exposed such intimate details of her life for readers would be attacked in the comments, editors decided to ask specific questions of commenters—“Has your life been affected by heroin?”—to guide the discussion, deleting anything that was off-topic or abusive.

The result, Hulen says, was many insightful, informative, and touching comments that added to the overall series and showed that giving a comments section a little more direction could improve discourse, as long as the newsroom had the manpower to weed out the comments that didn’t.

But that lack of manpower was the problem.

ADN is the biggest paper in Alaska but it’s still comparatively small, with a staff of 60. There are no dedicated comment moderators, so several staff members split those duties, and they increasingly found themselves losing a game of whack-a-mole with abusive comments and trolling commenters. By the end of 2015, ADN’s comments sections were a “rough dudespace,” as Hulen describes it, dominated by angry men shouting each other down and leaving racist and sexist comments. “All the usual stuff.”

And then came the spam. ADN had switched to Facebook’s commenting platform in the hopes that commenters having to use their real names would keep them civil. It didn’t. Now the comments were getting carpet-bombed by fake Facebook identities urging readers to click on their links for amazing weight loss secrets or information on how to work from home and make six figures a year. Moderators couldn’t keep up. It was time for a change.

In the past few years, how newsrooms think about comments—which had remained largely unchanged since outlets began introducing them in the mid-2000s—has changed as well. While many have elected to kill comments sections, ceding that community to third parties such as social media, others are looking at them as a key part of their audience engagement strategies—and seeing their audience engagement strategies as a key part of their business model.

Several publications, including The Guardian, have taken deep dives into their comments sections for a data-centered look at what their future should be. In two separate surveys, FiveThirtyEight and the University of Texas’s Engaging News Project asked thousands of commenters—more than 9,000 between the two—why, when, and how they comment. The Financial Times overhauled its comments strategy last spring and, in January, The Washington Post launched a weekly newsletter showcasing the best conversations and comments from online articles posted that week. For most outlets, these efforts are part of a larger strategy to listen more to its readership—and let it know they’re listening back—and ultimately give the audience a product worth paying for. Following the election of a candidate few journalists saw coming and many of whose supporters eschewed traditional journalism for hyper-partisan publications that told them what they wanted to hear instead of what was demonstrably true, this kind of thinking is more important than ever.

“We do have a complicated relationship with our audience,” says Mónica Guzmán, co-founder of Seattle-focused newsletter The Evergrey and an early proponent of comments and community in journalism. “And I think we’re learning how valuable deeper connections can be and how valuable incorporating contributions can be. For so long, we were the ones talking and they were the ones listening.”

These efforts are part of a larger strategy to listen more to its readership—and let it know they’re listening back

That’s not good enough anymore. Neither are the comments sections that do little more than give trolls another pulpit. But effectively giving those communities away to social media isn’t a solution, either. “There are ways that we can lead and guide our own community and design whole spaces that do that for them,” Guzmán says. “That’s part of the service we provide. It’s a responsibility.”

That service may be made easier with new technology that will help achieve those goals. Civil Comments puts the onus on commenters to moderate each other by forcing them to rate randomly selected comments before they can comment themselves. The Coral Project hopes to introduce a suite of tools that will unify and integrate audience engagement, including comments, across news sites. Some publications have partnered with annotation platform Genius, which allows reporters and readers to place line-by-line notes directly next to a webpage’s content and have a focused discussion about it. The New York Times is working with Google on technology that uses machine learning to advance automated moderation in ways never achieved before.

In 2006, The Washington Post became the first major United States news site to enable comments on articles. (It allowed comments on its blogs starting in January 2005.) The impetus behind the decision, then-executive editor of WashingtonPost.com Jim Brady says, was seeing how many Post articles were being discussed on other people’s blogs: “People are talking about what The Washington Post writes all over the internet. Why would we want people to go to 20 different sites to read reaction to the story when we could get it ourselves?”

While comments did give readers an on-site place to discuss articles, Brady acknowledges that it was a tough sell to reporters to convince them to join in. Most were just fine keeping their readers at arm’s length. Brady cites Chris Cillizza as a journalist who, through his “The Fix” blog and under his articles, was responsive to and involved in comments from the beginning, though he’s since changed his mind. “It turned into the loudest and most obnoxious person on your block appointing himself mayor,” Cillizza says. “He isn’t broadly representative of your neighborhood but he’s the loudest.”

Cillizza now favors eliminating comments sections under politics articles—he thinks people are too passionate about the subject matter to really have productive and interesting discussions—though he believes that comments sections under stories about other subjects can still work. He’s also a fan of Quora, a question and answer site that has partnered with outlets like Newsweek and Slate to publish particularly illuminating or interesting answers on their sites. So far, though, this is more of a syndication deal than an engagement strategy, as the audience doing the interacting is Quora’s.

One way that outlets can use a third party to engage with their actual audience is with annotations. The Washington Post and Los Angeles Times have partnered with the annotation platform Genius to add context from journalists and readers to transcripts of speeches by politicians and actors. The Los Angeles Times annotated actor Jesse Williams’ speech when he accepted the humanitarian honor at the June 2016 Black Entertainment Television Awards. Readers didn’t do much annotating but instead responded to the Times’ annotations, sharing their own experiences with racism, noting the distinct lack of minorities in their history textbooks, and recommending works that expanded on points Williams had made.

While most places are looking at ways to combine humans and technology to create better comments sections—with an emphasis on better human moderation—The New York Times is taking a slightly different tack: teaming up with Google’s Jigsaw incubator to create technology that may be able moderate comments for the same things that it was assumed only humans could do, such as tone or going off-topic. The Times’s biggest problem with its comment sections is that its hands-on approach to moderating doesn’t scale; its moderators are only able to look at approximately 11,000 comments each day, which is why commenting is currently available on only about 10 percent of the paper’s online articles. If this partnership works, that problem will be solved and the Times will conceivably be able to open all of its articles to comments–and Google hopes to make the technology open source and available to other outlets down the road.

The New York Times’s moderators have been tagging disapproved comments with reasons for their rejection for years. Unbeknownst to them at the time, those tags would come in handy when they decided to work with Jigsaw to use machine learning to predict which comments would and would not be approved by a human moderator. This goes way beyond how technology assisted moderating in the past, which mostly relied on filters to catch comments with bad words in them. Community editor Bassey Etim says the Times should start rolling the algorithm out within the next few months.

As for the human moderators, they will be able to spend less time making sure commenters are behaving themselves and more time on community-building and engagement tasks like curating comments to feature in The New York Times’ reporting. “The best thing you can do for a community is to actively show people that somebody at the organization is listening,” says Etim. “The more you do on that end, the less intense moderation you need to have.”

Though the workload of managing comments can be significant, newsroom engagement can have a positive effect—and commenters are often hoping for journalists to join in. A recent joint survey by The Coral Project and the University of Texas’s Engaging New Project, which garnered more than 12,000 responses, found that more than 75 percent of commenters on news sites would like if reporters clarified factual questions in the comment section, and nearly half said they’d like it if newsrooms highlighted quality comments. According to another survey, from 2014, by the Engaging News Project, the likelihood of an “uncivil” comment decreased by 15 percent when journalists participated in comments sections. Even so, participation like that is rare. That same study also cited a 2010 survey that found that, while 98 percent of newspaper reporters said they read comments, 80 percent of them said that they “never” or “rarely” responded to them.

This attitude appears to be changing, slowly but surely. A 2016 Engaging News Project survey of 34 journalists found that all of them read comments at least occasionally, though a third refused to respond to them. The majority, however, did respond to comments and saw such engagement as part of their jobs.

Guzmán remembers what a key role comments (and the audience that left them) were when she worked as an online reporter at the Seattle Post-Intelligencer. Though she received her share of mean comments, the constructive ones were a regular source of ideas for her columns. “I got so much out of my comments,” Guzmán says. “Half the time my next story would come from the comments on the last one. I really took them seriously. It was so valuable to me that I just got to the point where I looked around went, ‘Why isn’t everybody doing this?'”

Guzmán says her experience with comments and community has inspired The Evergrey, a newsletter she co-founded that incorporates feedback from its audience into stories, from a love letter to Seattle aggregated entirely from reader suggestions to sections that answer reader questions, such as how to know what makes a credible news site when so much “fake news” is spread through social media.

Yet comments sections still tend to be neglected. A Engaging News Project survey, published in 2016, of 525 editors and news directors across all mediums showed that, 10 years after comments became widespread on news sites, only 61 percent had staff moderators for their comments sections and only 22 percent had written policies on how they should do this. The bright spot: 87 percent said they responded in comments sections and/or on social media, though the survey didn’t make a distinction between on-site comments and those made on Twitter or Facebook.

In her report on audience engagement for the American Press Institute, released last May, Guzmán stressed the importance of creating collaborative and mutually beneficial relationships with readers—”making sure your work matters to your audience,” which then, for the business side, “helps ensure that work finds the public support it needs to endure.” Comments are part of an audience engagement strategy that many news outlets are finally starting to realize is editorially and commercially essential.

The push to make comments a free speech zone where all viewpoints are welcome may have had the opposite effect. “Sometimes we have erred on the side of allowing everybody speak without realizing that that effectively silences certain groups of people,” says Mary Hamilton, The Guardian’s executive editor for audience. The Guardian, a proponent of audience participation in its news stories, is in the midst of examining its own approach to comments and community. In April, The Guardian revealed the results of its analysis of 70 million comments left on its site between 1999 and 2016  (though the vast majority of which were made after 2006) for a series called “The Web We Want,” which looks at online abuse.

Newsroom engagement can have a positive effect—and commenters are often hoping for journalists to join in

The data team looked at how many comments had been blocked and which sections and authors tended to attract the most blocked comments. Only a small minority of the 70 million comments had been blocked: 1.4 million, or 2 percent. But social minorities bore the brunt of those abusive comments. Articles written by women had a higher percentage of blocked comments than those written by men, and articles about feminism and rape were among those with the highest percentage of blocked comments. Of the top 10 authors who received the most abuse, eight were women, four were white and four were of color. The two men were black, and one was gay. All 10 of the writers who received the least abuse were men.

This deep dive is part of editor in chief Katharine Viner’s strategic strategic vision for how to increase audience loyalty, and then convert that loyalty into paying customers. Commenters, Hamilton says, tend to be the most invested and dedicated of all readers: “Even though they might be a very small proportion of the readership, that readership is some of the most loyal. That method of engagement, if done well and if done with commitment and understanding of where it fits specifically can be hugely valuable to the organization.”

Hamilton says The Guardian is also trying to assess the value of the comments themselves and how having quality, constructive discussions might also be a draw for readers: “We’re thinking about how we measure the value of comments to the people who read them alongside our journalism,” Hamilton says.

Aron Pilhofer, who was The Guardian’s executive editor of digital before joining the faculty of the School of Media and Communication at Temple University last summer, considers audience engagement—of which comments are part but, he stresses, but not all—to be “fundamentally core to any publication that considers conversion to be an important thing. By that I mean paywall, subscriptions, membership, or donations.”

Before moving to The Guardian in 2014, Pilhofer was the New York Times’ associate managing editor for digital strategy. He’s seen the Times experiment with its commenting policies and platforms as well as with other ways to engage its audience, citing its annual “The Lives They Loved” feature—where readers can submit photos and stories about people they know who died in the past year—as one example of an “amazing piece of collaborative journalism” that can result from such initiatives. But, Pilhofer says, these projects are usually one-offs. He wanted a platform that would be a “toolkit for newsrooms to do this kind of collaborative journalism.”

The New York Times teamed up with The Washington Post and Mozilla to try to create this kind of community platform. In June 2014, the Knight Foundation awarded what would become known as The Coral Project a $3.9 million grant over the next three years. Project lead Andrew Losowsky says they plan to release two tools by June 2017, all of which will be open source—free to any newsroom that wants them.

Those two tools are called Ask and Talk. The first to be released, Ask, allows reporters to ask readers for contributions or answers to questions. It made its debut on Philly.com for the election, where it was used to solicit reports from citizens on polling experiences and then to collect and publish reader responses to the presidential election’s results. ProPublica has been doing something similar as part of its “Get Involved” initiative, asking readers to submit information about their apartment rentals as part of an investigation into New York City landlords or asking for stories about how readers’ medical information had been compromised as part of an investigation into patient privacy violations.

Talk, which Losowsky hopes will be available in early 2017, is, at its most basic level, a comments section. Losowsky says Talk gives the moderators much more data about the community, which could then be used to identify troublemakers and sources. Community editors can look for users who often post original comments but seldom reply to others to determine who might be a possible troll. Or they can look for commenters with higher percentages of flagged comments or comments deleted by moderators.

More positively, the filters could be used to find commenters who tend to leave longer and possibly more substantial posts that are generally well-received by other commenters. If a journalist is looking for a source in comments who has personal experience with or is an expert on the article or its subject, these filters could make that process much faster.

Of course, this tool is only as effective as the newsroom’s moderation team. “It’s not a ‘set-it-and-forget-it, this is going to do all the moderation for us,'” Losowsky says. “It’s a way of making your moderation actions scalable and predictable.”

Civil Comments, in which commenters have to rate other comments on the site for quality and civility before they’re allowed to post their own, is another approach to streamlining the moderation process. “We just need to find ways to correct the small minority of bad actors who are ruining the experience for everybody else,” says Christa Mrgan, who co-founded Civil with Aja Bogdanoff in January 2015.

Civil chooses comments at random for the user to rate, which prevents them from upvoting their friends and downvoting their enemies. If users give too many outlier scores, their accounts will be flagged, as will comments whose rating dips below a certain threshold. But Bogdanoff sees it as more of a behavior modification system than a filter. Not only does this force commenters to think twice about civility before submitting, it also makes them moderators themselves. The more comments a site gets, the more ratings it also gets on existing comments. This makes it infinitely scalable, and frees up staff moderators or community editors to work on other audience engagement strategies.

ADN, David Hulen’s newsroom, is one of the largest outlets to try Civil so far. Other publications include The Register-Guard in Eugene, Oregon, Honolulu Civil Beat, and Canada’s The Globe and Mail. A sixth publication, Willamette Week, went back to Disqus after six months with Civil, saying there was a decrease in the number of comments as well as user engagement on the site.

A month after it installed Civil Comments, ADN ran a story about a man looking for his birth parents after he was abandoned as baby in a cardboard box. It was both an update of what had been a major Anchorage story when he was first found and a story of a man who desperately wanted to know something about his origins. It was also a story that could have attracted a lot of toxic comments with ADN’s old system. For the most part, it didn’t. Hulen thinks the new platform had something to do with that. He points to two comments that stood out to him.

The entire culture around comments sections has to change, from those who leave them to those who moderate them

One commenter remembered going to church with the man and his foster family, and then babysitting him after he was adopted. “You and my daughter loved playing together,” she wrote. “I have thought about you SO MANY TIMES over the years. Sending love and prayers for success.”

Another woman said she was 17 years old and seven months pregnant when the baby was found, so his story hit home for her. “I sobbed for that baby,” she wrote. “I prayed, and thanked God I was blessed with a safety net, my mom.” She said she would continue to pray that he would find “the answers you deserve, and desire, so that you may feel some sort of closure.”

And there were others who had suggestions of websites that might help him find his biological family, or who shared their own stories of being adopted and looking for their birth family. There was no spam, and, aside from a few exceptions, “the comments were just sweet,” Hulen says. “Just more human.”

These new comments platforms may make moderation a much easier task, but that’s only a start. Winning back the good commenters who may have abandoned sections when they were at their worst, identifying and punishing toxic community members, and convincing journalists and their outlets to play a bigger role in shaping the communities under their words must happen, too. The entire culture around comments sections has to change, from those who leave them to those who moderate them. The result is a product valued not just for the information it delivers but also for the community it provides.

“You see the diverse revenue streams, you see sustainability, and you see a great community,” Guzmán says of the publications that have done this successfully. “It’s not a coincidence.”