Last year, two very talented educators — Ted Appel, the extraordinary principal we have at our school, and Kelly Young, creator of much of the engaging curriculum we use at our school through his Pebble Creek Labs — brought-up the same point in separate meetings with teachers at my school: The importance of not being “data-driven” and, instead, to be “data-informed.”
These conversations took place in the context of discussing the results of state standardized tests. Here’s the point made by Ted:
If schools are data-driven, they might make decisions like keeping students who are “borderline” between algebra and a higher-level of math in algebra so that they do well in the algebra state test. Or, in English, teachers might focus a lot of energy on teaching a “strand” that is heavy on the tests — even though it might not help the student become a life-long reader. In other words, the school can tend to focus on its institutional self-interest instead of what’s best for the students.
In schools that are data-informed, test results are just one more piece of information that can be helpful in determining future directions.
Since that conversation took place, I’ve written several posts about the topic. I thought it might be useful to bring together several related resources.
Here are my choices for The Best Resources Showing Why We Need To Be “Data-Informed” & Not “Data-Driven”:
First, I’m going to list the post I wrote immediately after that conversation — “Data-Driven” Versus “Data-Informed”
Next, a Dilbert cartoon that Alexander Russo shared on his blog:
The cartoon reminded of what the New York judge said earlier this month when he ruled that the School District can publicly release the names of teachers and their “Teacher Data Reports.” Here is what the judge said (and I kid you not):
“The UFT’s argument that the data reflected in the TDRs should not be released because the TDRs are so flawed and unreliable as to be subjective is without merit,” the judge wrote, citing legal precedent that “there is no requirement that data be reliable for it to be disclosed.”
Data-Driven…Off a Cliff is the title of an excellent post by Robert Pondiscio.
An article in Educational Leadership is a year-old, but it’s new to me and certainly worth sharing. It’s called The New Stupid, and has the subtitle “Educators have made great strides in using data. But danger lies ahead for those who misunderstand what data can and can’t do.” It’s written by Frederick M. Hess.
It’s an article worth reading (though I do have concerns about some of its points), and relates to what I’ve written about being “Data-Driven” Versus “Data-Informed.”
Here are a couple of excerpts:
…the key is not to retreat from data but to truly embrace the data by asking hard questions, considering organizational realities, and contemplating unintended consequences. Absent sensible restraint, it is not difficult to envision a raft of poor judgments governing staffing, operations, and instruction—all in the name of “data-driven decision making.”
First, educators should be wary of allowing data or research to substitute for good judgment. When presented with persuasive findings or promising new programs, it is still vital to ask the simple questions: What are the presumed benefits of adopting this program or reform? What are the costs? How confident are we that the promised results are replicable? What contextual factors might complicate projections? Data-driven decision making does not simply require good data; it also requires good decisions.
The Truth Wears Off: Is there something wrong with the scientific method? by Jonah Lehrer is an exceptional article from The New Yorker. David Brooks from The New York Times wrote a nice summary of the article:
He describes a class of antipsychotic drugs, whose effectiveness was demonstrated by several large clinical trials. But in a subsequent batch of studies, the therapeutic power of the drugs appeared to wane precipitously.
This is not an isolated case. “But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain,” Lehrer writes. “It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable.”
The world is fluid. Bias and randomness can creep in from all directions. For example, between 1966 and 1995 there were 47 acupuncture studies conducted in Japan, Taiwan and China, and they all found it to be an effective therapy. There were 94 studies in the U.S., Sweden and Britain, and only 56 percent showed benefits. The lesson is not to throw out studies, but to never underestimate the complexity of the world around.
Talking To Students About Their Reading (& Their Data) is a post I’ve written.
In a Data-Heavy Society, Being Defined by the Numbers is by Alina Tugend at The New York Times.
Data-Driven Instruction and the Practice of Teaching is by Larry Cuban.
The Obituaries for Data-Driven ‘Reform’ Are Being Written is by John Thompson.
California Governor Puts the Testing Juggernaut On Ice is by Anthony Cody at Education Week.
Data-Driven To Distraction appeared on Larry Cuban’s blog.
Policy by Algorithm is a nice post over at Ed Week.
Professional Judgment: Beyond Data Worship is by Justin Baeder at Education Week.
Bias toward Numbers in Judging Teaching is by Larry Cuban.
The False Allure Of Statistics is by John Thompson.
‘Moneyball’ and making schools better is by John Thompson.
“Why Do Good Policy Makers Use Bad Indicators?” is by Larry Cuban.
New Hope for the Obama/Gates School of Reform is by John Thompson.
“It’s amazing how much it’s possible to figure out by analyzing the various kinds of data I’ve kept,” Stephen Wolfram says. To which I say, “I’m looking at your data, and you know what’s amazing to me? How much of you is missing.”
This is the last paragraph of Robert Krulwich’s article at NPR, titled Mirror, Mirror On The Wall, Does The Data Tell It All? In it, he compares authors of books, one by Stephen Wolfram, creator of a the Wolfram search engine, and Bill Bryson, author of a biographical account of growing up in Iowa. The column, though not specifically about schools, hits a “bulls-eye” on our current data-driven madness.
Tired of the Tyranny of Data is by Dave Orphal.
Big Data Doesn’t Work if You Ignore the Small Things that Matter is from The Harvard Business Review.
Invisible Data is from Stories From School.
On the Uses and Meaning of Data is by David B. Cohen.
Friday Thoughts on Data, Assessment & Informed Decision Making in Schools is from School Finance 101.
Be evidence informed, but not data driven. There are faces behind data, so use data judiciously. -Hargreaves & Fullan
— Joe Clark (@DrJoeClark) February 3, 2013
Data: No deus ex machina is by Frederick M. Hess & Jal Mehta.
Bill Gates is naive, data is not objective is by Cathy O’Neil and is really good.
Bill Gates and the Cult of Measurement is by Anthony Cody.
Sure, Big Data Is Great. But So Is Intuition. is from The New York Times. Here’s an excerpt:
It’s encouraging that thoughtful data scientists like Ms. Perlich and Ms. Schutt recognize the limits and shortcomings of the Big Data technology that they are building. Listening to the data is important, they say, but so is experience and intuition. After all, what is intuition at its best but large amounts of data of all kinds filtered through a human brain rather than a math model?
At the M.I.T. conference, Ms. Schutt was asked what makes a good data scientist. Obviously, she replied, the requirements include computer science and math skills, but you also want someone who has a deep, wide-ranging curiosity, is innovative and is guided by experience as well as data.
“I don’t worship the machine,” she said.
Beware the Big Errors of ‘Big Data’ is from Wired.
Data-Informed Versus Data-Driven PLC Teams is from All Things PLC.
David Brooks, who generally loses all coherence when he writes explicitly about education issues, has just written an eloquent case for the importance of being data-informed, and not data-driven. Read his column titled What Data Can’t Do. Here’s an excerpt:
The Problem with Our Data Obsession is from MIT.
Data Without Context Tells a Misleading Story is from The New York Times.
“Big (Dumb) Data” is by John Thompson.
Data are no good without theory is from The Washington Post.
The Perils of Economic Thinking about Human Behavior is from School Finance 101.
What You’ll Do Next is by David Brooks
At the risk of being accused of taking a “cheap shot,” I just can’t resist embedding two segments from The Colbert Show about the now well-known mistake by the two economists whose work has been cited endlessly to support austerity. And I can’t resist adding it to this list:
The Great Lakes Center has released an excellent report on Data-driven Improvement and Accountability. The Washington Post published an excerpt, Six principles for using data to hold people accountable.
The Tyranny of the Datum is by John Kuhn.
Garbage In, Garbage Out: Or, How to Lie with Bad Data is from Medium.
How ‘data walls’ in classrooms can humiliate young kids is by Valerie Strauss.
Select Your Conclusions, Apply Data is from The Shanker Blog.
How ‘platooning’ and data walls are changing elementary school is from The Washington Post.
Big data: are we making a big mistake? is from The Financial Times.
Ainge: Analytics Sometimes Leads To Shortcuts is from RealGM Basketball.
— Andy Hargreaves (@HargreavesBC) May 15, 2014
Misusing Test Data is from Renee Moore’s blog.
‘There’s Nothing More Innovative Than Care’ is the title of one of my related columns at Education Week Teacher.
Shoot Your Data: 5 Kinds of Photos That Reveal More Than Numbers is by Angela Stockman. This is a very practical post that has policy implications.
School Districts Embrace Business Model of Data Collection is from The New York Times.
The Illusion of Knowing is by David Didau.
Using Data In Education: The Good, the Bad and the Numbers with Myron Dueck, Dr. Jenni Donohoo, and Nancy Fichtman Dana is one of my BAM! radio shows.
How To Use Data – & How Not To Use It – In Schools is the headline of one of my columns in Education Week Teacher.
Quote from one of my favorite books.. pic.twitter.com/zWwV6FJNqk
— Doug Lemov (@Doug_Lemov) March 1, 2016
5 Doubts About Data-Driven Schools is from NPR.
Eight Daily Classroom Data Sources to Empower Student-Directed Learning is by Kathy Dyer.
How to Use Metrics Instead of Being Used by Them is from Bloomberg.
Help Your Team Understand What Data Is and Isn’t Good For is from The Harvard Business Review.
Data Was Supposed to Fix the U.S. Education System. Here’s Why It Hasn’t. is from The Harvard Business Review.
6 Steps to Equitable Data Analysis is from Edutopia.
Data Can Only Mean One Thing by Scott Adams https://t.co/meIc0OnMbL on the importance of being data- informed instead of being data-driven
— Larry Ferlazzo (@Larryferlazzo) January 25, 2020
Does Studying Student Data Really Raise Test Scores? is from Ed Week. It reviews research that suggests teachers analyzing student data tends to not result in student improvement, primarily because it doesn’t result in teachers changing instructional practices. It reminds of one day when I was interpreting in conversation between a colleague and a parent and her child. My colleague was saying the child could come in for tutoring after-school anytime, and the student interjected, “But he just tells me the same way he did it in class and I don’t understand!”
Additional suggestions are welcome.
If you found this post useful, you might want to consider subscribing to this blog for free.
You might also want to explore the over 600 other “The Best…” lists I’ve compiled.