Educating Journalists
In asking young journalists to write about their experiences, what we wanted to provide was a forum in which they could express their views, concerns and ideas about the way in which journalism is practiced today. What we found as their articles arrived is that these young journalists raise questions that are relevant for seasoned journalists to ponder. As one correspondent writes: “I know in the future…I’ll take opportunities to listen to interns and recent college graduates who other folks in the newsroom might dismiss as starry-eyed idealists. I hope that listening to their perspectives will help me remember why I chose to become a journalist in the first place.”
After two years of traveling the country working with reporters and editors on computer-assisted reporting techniques, it was clear that no story invokes more fear than the dreaded annual school report. Editors know educators will complain. After all, the experts say, each student is different. Parents demand impartial reviews of schools—until their children’s schools are shown to fare poorly. And school administrators say it’s not just unfair but also downright destructive for journalists to try to do anything other than repackage their press releases.
These complaints, though, fly in the face of what the school systems themselves do. After his rankings of school districts was published in the paper one reporter heard from the man who was principal at a school he’d attended. The principal complained that his district did not score as well as he had expected, arguing that there were too many factors that numbers could not take into account. The reporter reminded him that the school system never had trouble labeling him as an underachiever based only on standardized test scores and grade point averages. Apparently, it only became unfair when the schools, not the individual students, were being graded.
Some reporters, including the ones writing in this issue, have addressed the problem by making it clear to educators that it’s the newspaper’s or station’s job to provide information to parents, not to hide it. They let “the experts” know they won’t be deterred from getting this important information to parents. The experts can help them make it as fair as possible or they can live with—and furiously debate—the results. Still, reporters are often left on our own to figure out ways to measure the quality of the schools we cover, and often on deadline. Here are five tips to help guide journalists through the minefields of data. These tips are culled from numerous discussions with reporters and their editors who are tackling this kind of project annually.
1. Fight for Student Level Data
One of the key failings of many school report stories is that they rely on statistics generated by the school systems. This problem is driven by the way these data are offered. Instead of providing a database of basic information about the students (with names and exact addresses redacted), the schools provide hundreds of summary statistics. This makes the analysis a lot harder than it has to be.
A skilled reporter needs to know only about 10 things about each student to generate meaningful summaries. For example, if you had the grade level, ethnicity, reading test result and math test result for every child in the district, in an independent analysis the median or average score could be calculated by school along with the percentage of kids who took the test.
Instead, school officials present a series of pre-made calculations to reporters, turning 10 columns of information into hundreds: the percentage taking the test by ethnicity, the percentage of each ethnic group passing, the percentage of high achievers, and the like. Then they provide fields for the school, the district, the county and the state. In the end, their numbers and analysis amount to a sea of numbers with little meaning. And the data they provide turn a rather straight-forward analysis into a nightmare for someone just starting out in computer-assisted reporting.
So fight for all the information it is possible to get about individual students, not schools. You may have to sign away some rights to use the data. A caveat common in the use of computer files on children (and patients in health care) is that you won’t try to track down anyone you find through the data.
Just be sure you’re allowed to interview students in the schools—as way of illuminating what the data demonstrate—without trying to find an individual student.
2. Copy Someone Else’s Methodology
Don’t try to reinvent school rankings each year. Although reporters grow bored with the same annual analysis, parents appreciate consistency. And it’s not always even necessary to come up with a ranking system on your own. You can borrow a method from others who have spent more time studying how best to present this information. Education Week, for example, grades school systems across the country each year in its Quality Counts project. The researchers there have spent years trying to come up with a way to compare one school with another, one district with another, and one state with another. Borrow their techniques and ask for their help.
Also, look for any standards developed in your state to measure school performance. In Maryland, for example, the state has threatened to take over schools that fail to meet certain standards. This yardstick is written into the regulations and provides a good least common denominator.
3. Define “Quality” Early On
One of the big criticisms of school report stories is that they’re subjective. As school kids might put it, “And your point?” Of course a ranking or grading of schools is a qualitative, not a purely quantitative, endeavor. There will always be an element of opinion in your work based on what you choose to measure and exactly how you combine those factors. Be prepared to defend your choices.
The best reason for doing this kind of independent analysis is your job as a watchdog of local educators. Remember that the grading is of the job that educators, administrators and legislators are doing, not the kids themselves. Staying focused on grading the adults is the reason that many reporters adjust results to account for the difficulties kids bring from the home, usually by using statistical regression techniques. Giving the school system credit for advantages kids bring to school from home isn’t telling your audience much about the quality of the school. Neither is blaming the school for problems the kids bring as a result of the impact of lower family incomes.
U.S. News & World Report dubbed this kind of adjustment “value added,” and includes it in the annual ranking of colleges. It’s a good signal to parents that what is being graded is the school system, not the children.
4. Go Beyond the Report Card
In Baltimore, reporters examined the experience of teachers in inner-city schools. Reporters there found that the least experienced teachers were assigned to the most difficult schools. As soon as they got a little experience, they moved.
In Broward County, Florida, reporters found that, despite a court order, schools in minority districts have received little extra money to bring their facilities up to par with schools in wealthier districts. And in Charlotte, North Carolina, a groundbreaking story in January detailed the potential effects of returning to neighborhood schools at Ground Zero of 1970’s-era forced desegregation. About half of the region’s children would attend schools with virtually no diversity if the busing were to end.
These stories and others use the data that generate the school report cards to focus their reporters in terms of the ongoing coverage of education news.
5. Give Parents Choices
On-line editions provide readers with powerful tools. Let them use the data on Web sites by giving them the option of examining the schools in any way they’d choose—by diversity, combined test scores, math scores or student-teacher ratios. This option doesn’t just stand out as a public service, but it can also counter the criticism that there are many ways to measure the performance of a school, and the newspaper chose to illustrate its case with only one or two.
Of course, the biggest tip is one that is so obvious it doesn’t require its own heading: Write in depth about schools, teachers and students rather than relying on the numbers to tell the story. Focusing on schools that do well—as well as on those that don’t appear to be working to their potential—will result in compelling stories. Finding teachers and administrators who have come up with ways of organizing a school or teaching a particular subject that works well for students is also a way of telling essential stories about education. So is reporting that uncovers complacency in teachers or burdensome regulations or rules that serve to hold back kids who might otherwise succeed in school.
Put the numbers in their place, in charts and on line, and let reporters go out and illustrate with words the stories that are responsible for the numbers we can now reveal.
Sarah Cohen was Training Director for Investigative Reporters and Editors, Inc. for two years and traveled the country teaching computer-assisted reporting techniques to reporters, editors and producers. She was recently hired as the Database Editor for The Washington Post.
These complaints, though, fly in the face of what the school systems themselves do. After his rankings of school districts was published in the paper one reporter heard from the man who was principal at a school he’d attended. The principal complained that his district did not score as well as he had expected, arguing that there were too many factors that numbers could not take into account. The reporter reminded him that the school system never had trouble labeling him as an underachiever based only on standardized test scores and grade point averages. Apparently, it only became unfair when the schools, not the individual students, were being graded.
Some reporters, including the ones writing in this issue, have addressed the problem by making it clear to educators that it’s the newspaper’s or station’s job to provide information to parents, not to hide it. They let “the experts” know they won’t be deterred from getting this important information to parents. The experts can help them make it as fair as possible or they can live with—and furiously debate—the results. Still, reporters are often left on our own to figure out ways to measure the quality of the schools we cover, and often on deadline. Here are five tips to help guide journalists through the minefields of data. These tips are culled from numerous discussions with reporters and their editors who are tackling this kind of project annually.
1. Fight for Student Level Data
One of the key failings of many school report stories is that they rely on statistics generated by the school systems. This problem is driven by the way these data are offered. Instead of providing a database of basic information about the students (with names and exact addresses redacted), the schools provide hundreds of summary statistics. This makes the analysis a lot harder than it has to be.
A skilled reporter needs to know only about 10 things about each student to generate meaningful summaries. For example, if you had the grade level, ethnicity, reading test result and math test result for every child in the district, in an independent analysis the median or average score could be calculated by school along with the percentage of kids who took the test.
Instead, school officials present a series of pre-made calculations to reporters, turning 10 columns of information into hundreds: the percentage taking the test by ethnicity, the percentage of each ethnic group passing, the percentage of high achievers, and the like. Then they provide fields for the school, the district, the county and the state. In the end, their numbers and analysis amount to a sea of numbers with little meaning. And the data they provide turn a rather straight-forward analysis into a nightmare for someone just starting out in computer-assisted reporting.
So fight for all the information it is possible to get about individual students, not schools. You may have to sign away some rights to use the data. A caveat common in the use of computer files on children (and patients in health care) is that you won’t try to track down anyone you find through the data.
Just be sure you’re allowed to interview students in the schools—as way of illuminating what the data demonstrate—without trying to find an individual student.
2. Copy Someone Else’s Methodology
Don’t try to reinvent school rankings each year. Although reporters grow bored with the same annual analysis, parents appreciate consistency. And it’s not always even necessary to come up with a ranking system on your own. You can borrow a method from others who have spent more time studying how best to present this information. Education Week, for example, grades school systems across the country each year in its Quality Counts project. The researchers there have spent years trying to come up with a way to compare one school with another, one district with another, and one state with another. Borrow their techniques and ask for their help.
Also, look for any standards developed in your state to measure school performance. In Maryland, for example, the state has threatened to take over schools that fail to meet certain standards. This yardstick is written into the regulations and provides a good least common denominator.
3. Define “Quality” Early On
One of the big criticisms of school report stories is that they’re subjective. As school kids might put it, “And your point?” Of course a ranking or grading of schools is a qualitative, not a purely quantitative, endeavor. There will always be an element of opinion in your work based on what you choose to measure and exactly how you combine those factors. Be prepared to defend your choices.
The best reason for doing this kind of independent analysis is your job as a watchdog of local educators. Remember that the grading is of the job that educators, administrators and legislators are doing, not the kids themselves. Staying focused on grading the adults is the reason that many reporters adjust results to account for the difficulties kids bring from the home, usually by using statistical regression techniques. Giving the school system credit for advantages kids bring to school from home isn’t telling your audience much about the quality of the school. Neither is blaming the school for problems the kids bring as a result of the impact of lower family incomes.
U.S. News & World Report dubbed this kind of adjustment “value added,” and includes it in the annual ranking of colleges. It’s a good signal to parents that what is being graded is the school system, not the children.
4. Go Beyond the Report Card
In Baltimore, reporters examined the experience of teachers in inner-city schools. Reporters there found that the least experienced teachers were assigned to the most difficult schools. As soon as they got a little experience, they moved.
In Broward County, Florida, reporters found that, despite a court order, schools in minority districts have received little extra money to bring their facilities up to par with schools in wealthier districts. And in Charlotte, North Carolina, a groundbreaking story in January detailed the potential effects of returning to neighborhood schools at Ground Zero of 1970’s-era forced desegregation. About half of the region’s children would attend schools with virtually no diversity if the busing were to end.
These stories and others use the data that generate the school report cards to focus their reporters in terms of the ongoing coverage of education news.
5. Give Parents Choices
On-line editions provide readers with powerful tools. Let them use the data on Web sites by giving them the option of examining the schools in any way they’d choose—by diversity, combined test scores, math scores or student-teacher ratios. This option doesn’t just stand out as a public service, but it can also counter the criticism that there are many ways to measure the performance of a school, and the newspaper chose to illustrate its case with only one or two.
Of course, the biggest tip is one that is so obvious it doesn’t require its own heading: Write in depth about schools, teachers and students rather than relying on the numbers to tell the story. Focusing on schools that do well—as well as on those that don’t appear to be working to their potential—will result in compelling stories. Finding teachers and administrators who have come up with ways of organizing a school or teaching a particular subject that works well for students is also a way of telling essential stories about education. So is reporting that uncovers complacency in teachers or burdensome regulations or rules that serve to hold back kids who might otherwise succeed in school.
Put the numbers in their place, in charts and on line, and let reporters go out and illustrate with words the stories that are responsible for the numbers we can now reveal.
Sarah Cohen was Training Director for Investigative Reporters and Editors, Inc. for two years and traveled the country teaching computer-assisted reporting techniques to reporters, editors and producers. She was recently hired as the Database Editor for The Washington Post.
