geralt / Pixabay

Last year, two very talented educators — Ted Appel, the extraordinary principal we have at our school, and Kelly Young, creator of much of the engaging curriculum we use at our school through his Pebble Creek Labs — brought-up the same point in separate meetings with teachers at my school: The importance of not being “data-driven” and, instead, to be “data-informed.”

These conversations took place in the context of discussing the results of state standardized tests. Here’s the point made by Ted:

If schools are data-driven, they might make decisions like keeping students who are “borderline” between algebra and a higher-level of math in algebra so that they do well in the algebra state test. Or, in English, teachers might focus a lot of energy on teaching a “strand” that is heavy on the tests — even though it might not help the student become a life-long reader. In other words, the school can tend to focus on its institutional self-interest instead of what’s best for the students.

In schools that are data-informed, test results are just one more piece of information that can be helpful in determining future directions.

Since that conversation took place, I’ve written several posts about the topic. I thought it might be useful to bring together several related resources.

Here are my choices for The Best Resources Showing Why We Need To Be “Data-Informed” & Not “Data-Driven”:

First, I’m going to list the post I wrote immediately after that conversation — “Data-Driven” Versus “Data-Informed”

Next, a Dilbert cartoon that Alexander Russo shared on his blog:

Dilbert.com

The cartoon reminded of what the New York judge said earlier this month when he ruled that the School District can publicly release the names of teachers and their “Teacher Data Reports.” Here is what the judge said (and I kid you not):

“The UFT’s argument that the data reflected in the TDRs should not be released because the TDRs are so flawed and unreliable as to be subjective is without merit,” the judge wrote, citing legal precedent that “there is no requirement that data be reliable for it to be disclosed.”

Data-Driven…Off a Cliff is the title of an excellent post by Robert Pondiscio.

An article in Educational Leadership is a year-old, but it’s new to me and certainly worth sharing. It’s called The New Stupid, and has the subtitle “Educators have made great strides in using data. But danger lies ahead for those who misunderstand what data can and can’t do.” It’s written by Frederick M. Hess.

It’s an article worth reading (though I do have concerns about some of its points), and relates to what I’ve written about being “Data-Driven” Versus “Data-Informed.”

Here are a couple of excerpts:

…the key is not to retreat from data but to truly embrace the data by asking hard questions, considering organizational realities, and contemplating unintended consequences. Absent sensible restraint, it is not difficult to envision a raft of poor judgments governing staffing, operations, and instruction—all in the name of “data-driven decision making.”

and…

First, educators should be wary of allowing data or research to substitute for good judgment. When presented with persuasive findings or promising new programs, it is still vital to ask the simple questions: What are the presumed benefits of adopting this program or reform? What are the costs? How confident are we that the promised results are replicable? What contextual factors might complicate projections? Data-driven decision making does not simply require good data; it also requires good decisions.

The Truth Wears Off: Is there something wrong with the scientific method? by Jonah Lehrer is an exceptional article from The New Yorker. David Brooks from The New York Times wrote a nice summary of the article:

He describes a class of antipsychotic drugs, whose effectiveness was demonstrated by several large clinical trials. But in a subsequent batch of studies, the therapeutic power of the drugs appeared to wane precipitously.

This is not an isolated case. “But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain,” Lehrer writes. “It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable.”

 

 

The world is fluid. Bias and randomness can creep in from all directions. For example, between 1966 and 1995 there were 47 acupuncture studies conducted in Japan, Taiwan and China, and they all found it to be an effective therapy. There were 94 studies in the U.S., Sweden and Britain, and only 56 percent showed benefits. The lesson is not to throw out studies, but to never underestimate the complexity of the world around.

Talking To Students About Their Reading (& Their Data) is a post I’ve written.

“Using data for progress, not punishment”

In a Data-Heavy Society, Being Defined by the Numbers is by Alina Tugend at The New York Times.

Data-Driven Instruction and the Practice of Teaching is by Larry Cuban.

The Obituaries for Data-Driven ‘Reform’ Are Being Written is by John Thompson.

California Governor Puts the Testing Juggernaut On Ice is by Anthony Cody at Education Week.

Making the wrong “Data-Driven Decisions” is by Carl Anderson (thanks to Dean Shareski for the tip).

Data-Driven To Distraction appeared on Larry Cuban’s blog.

Larry Cuban has written another interesting post titled Jazz, Basketball, and Teacher Decision-making. John Thompson relates it to school data at Thompson: Duncan Can Shoot — But Can He Rebound?

“Not everything that matters can be measured”

“You Are Not An Equation” (And Neither Are Your Students)

Policy by Algorithm is a nice post over at Ed Week.

Professional Judgment: Beyond Data Worship is by Justin Baeder at Education Week.

This Is Why Our School is “Data-Informed” & Not “Data-Driven”

Bias toward Numbers in Judging Teaching is by Larry Cuban.

The False Allure Of Statistics is by John Thompson.

‘Moneyball’ and making schools better is by John Thompson.

Here’s Another Reason Why We Need To Be Data-Informed & Not Data-Driven

Data Gone Wild

“Why Do Good Policy Makers Use Bad Indicators?” is by Larry Cuban.

New Hope for the Obama/Gates School of Reform is by John Thompson.

“It’s amazing how much it’s possible to figure out by analyzing the various kinds of data I’ve kept,” Stephen Wolfram says. To which I say, “I’m looking at your data, and you know what’s amazing to me? How much of you is missing.”

This is the last paragraph of Robert Krulwich’s article at NPR, titled Mirror, Mirror On The Wall, Does The Data Tell It All? In it, he compares authors of books, one by Stephen Wolfram, creator of a the Wolfram search engine, and Bill Bryson, author of a biographical account of growing up in Iowa. The column, though not specifically about schools, hits a “bulls-eye” on our current data-driven madness.


What Does “Stop & Frisk” Have To Do With What’s Happening With Our Schools?

What Does The NYPD Have In Common With Many Data-Driven Schools?

Tired of the Tyranny of Data is by Dave Orphal.

Big Data Doesn’t Work if You Ignore the Small Things that Matter is from The Harvard Business Review.


Test Scores Often Misused In Policy Decisions
is from The Huffington Post.

The Data-Driven Education Movement
is from The Shanker Blog.

Data Overload

Invisible Data is from Stories From School.

Don’t Let Data Drive Your Dialogue is from The Canadian Education Association.

“The Goal Is The Goal”

On the Uses and Meaning of Data is by David B. Cohen.

Friday Thoughts on Data, Assessment & Informed Decision Making in Schools is from School Finance 101.

The New York Times Has Discovered The Perils Of Being Data-Driven — I Just Wish Arne Duncan Would, Too

Here’s a Part One and Part Two series of posts on the use of data in education, and they’re both from Larry Cuban’s blog.

Data: No deus ex machina is by Frederick M. Hess & Jal Mehta.

Bill Gates is naive, data is not objective is by Cathy O’Neil and is really good.

Bill Gates and the Cult of Measurement is by Anthony Cody.

Sure, Big Data Is Great. But So Is Intuition. is from The New York Times. Here’s an excerpt:

It’s encouraging that thoughtful data scientists like Ms. Perlich and Ms. Schutt recognize the limits and shortcomings of the Big Data technology that they are building. Listening to the data is important, they say, but so is experience and intuition. After all, what is intuition at its best but large amounts of data of all kinds filtered through a human brain rather than a math model?

At the M.I.T. conference, Ms. Schutt was asked what makes a good data scientist. Obviously, she replied, the requirements include computer science and math skills, but you also want someone who has a deep, wide-ranging curiosity, is innovative and is guided by experience as well as data.

“I don’t worship the machine,” she said.

Beware the Big Errors of ‘Big Data’ is from Wired.

The NYPD Probably Didn’t Stop All That Crime

Data-Informed Versus Data-Driven PLC Teams is from All Things PLC.

David Brooks, who generally loses all coherence when he writes explicitly about education issues, has just written an eloquent case for the importance of being data-informed, and not data-driven. Read his column titled What Data Can’t Do.

The Problem with Our Data Obsession is from MIT.

Data Without Context Tells a Misleading Story is from The New York Times.

Big Data is “not a replacement for the classic ways of understanding the world”

Quote Of The Day: “Data & data sets are not objective”

“Big (Dumb) Data” is by John Thompson.

Data are no good without theory is from The Washington Post.

The Perils of Economic Thinking about Human Behavior is from School Finance 101.

What You’ll Do Next is by David Brooks

Quote Of The Day: “The Dictatorship of Data”

How The NBA Finals Taught A Lesson About Not Being “Data-Driven”

Second Quote Of The Day: The Dangers Of Being “Data-Driven”

The Great Lakes Center has released an excellent report on Data-driven Improvement and Accountability. The Washington Post published an excerpt, Six principles for using data to hold people accountable.

The Tyranny of the Datum is by John Kuhn.

Garbage In, Garbage Out: Or, How to Lie with Bad Data is from Medium.

How ‘data walls’ in classrooms can humiliate young kids is by Valerie Strauss.

Select Your Conclusions, Apply Data is from The Shanker Blog.

How ‘platooning’ and data walls are changing elementary school is from The Washington Post.

Big data: are we making a big mistake? is from The Financial Times.

Ainge: Analytics Sometimes Leads To Shortcuts is from RealGM Basketball.

Misusing Test Data is from Renee Moore’s blog.

‘There’s Nothing More Innovative Than Care’ is the title of one of my related columns at Education Week Teacher.

Statistic Of The Day: The Myth Of Data-Driven Instruction


Quote Of The Day: Our Students Are More Than A Number

Shoot Your Data: 5 Kinds of Photos That Reveal More Than Numbers is by Angela Stockman. This is a very practical post that has policy implications.

Second Quote Of The Day: Numbers, Statistics, Data & All That Stuff

The Difference Between “Evidence-Based” & “Evidence-Informed” Education

Quote Of The Day: “How Not to Drown in Numbers”

School Districts Embrace Business Model of Data Collection is from The New York Times.

Quote Of The Day: “Algorithms do not fall from the sky.”

Quote Of The Day: Gladwell On Data


Quotes Of The Day On The Perils Of Being Data-Driven

The Illusion of Knowing is by David Didau.

Using Data In Education: The Good, the Bad and the Numbers with Myron Dueck, Dr. Jenni Donohoo, and Nancy Fichtman Dana is one of my BAM! radio shows.

How To Use Data – & How Not To Use It – In Schools is the headline of one of my columns in Education Week Teacher.

Quote Of The Day: The Dangers Of “Data Walls”

5 Doubts About Data-Driven Schools is from NPR.

Polls convinced me Hillary Clinton wouldn’t lose. As an education researcher, the result was a wake-up call is from Chalkbeat.

“Management [& Teaching] Is Much More Than a Science”

Eight Daily Classroom Data Sources to Empower Student-Directed Learning is by Kathy Dyer.

How to Use Metrics Instead of Being Used by Them is from Bloomberg.

Help Your Team Understand What Data Is and Isn’t Good For is from The Harvard Business Review.

ANOTHER EXAMPLE OF WHY WE NEED TO BE “DATA-INFORMED” & NOT “DATA-DRIVEN”

Data Was Supposed to Fix the U.S. Education System. Here’s Why It Hasn’t. is from The Harvard Business Review.

6 Steps to Equitable Data Analysis is from Edutopia.

QUOTE OF THE DAY: ANOTHER REASON TO BE “DATA-INFORMED” & NOT “DATA-DRIVEN”

DATA WALLS DON’T WORK – FOR ANYBODY

Here’s another reason why we need to be “data-informed” and not “data-driven“: Students: Amistad Faked Suspension Stats

Does Studying Student Data Really Raise Test Scores? is from Ed Week. It reviews research that suggests teachers analyzing student data tends to not result in student improvement, primarily because it doesn’t result in teachers changing instructional practices. It reminds of one day when I was interpreting in conversation between a colleague and a parent and her child. My colleague was saying the child could come in for tutoring after-school anytime, and the student interjected, “But he just tells me the same way he did it in class and I don’t understand!”

COULD POLLING ERRORS IN THE 2020 ELECTION TEACH US SOMETHING ABOUT THE USE OF “DATA” IN EDUCATION?

“DATA ALONE IS NOT KNOWLEDGE”

THE DANGERS OF BEING “DATA-DRIVEN”

MORE RESEARCH POINTS TO BEING “DATA-INFORMED” INSTEAD OF BEING “DATA-DRIVEN”

Another Reason To Be Careful About Being “Data-Driven”

Analysis: There’s Lots of Education Data Out There — and It Can Be Misleading. Here Are 6 Questions to Ask is from The 74.

Looking at Data Through an Equity Lens comes from ASCD Educational Leadership. I like what the authors say about looking through a “asset-based perspective” lens, which is similar to the “improvement rubric” concept I talk about in some links at The Best Rubric Sites (And A Beginning Discussion About Their Use).

PROOF POINTS: Researchers blast data analysis for teachers to help students is from The Hechinger Report.  You might also be interested in A Look Back: Studies Find That Teachers Analyzing Data Is No Help To Students If It Doesn’t Result In Different Instructional Practices .

Three tools to help educators better understand what students need is from MindShift.

Data Walls Demoralize Students. Assessment Expert Lorrie Shepard Explains Why is an older, but useful, report from NEPC.

Quote Of The Day: The Data-Driven Train To Nowhere

Why one-size-fits-all metrics for evaluating schools must go appeared in The Washington Post.

Standardized Tests Aren’t the Only Meaningful Data on Student Achievement is by Shane Safir.

I like the quote in this next tweet. It seems like a different way to say “data-informed” instead of “data-driven.” I’m adding it to

What Netflix Would Look Like if an Edtech Company Built It is by Dan Meyer.

Watch Two Schools Experience the Street Data Process is from Cult of Pedagogy.

A GOOD EXAMPLE OF BAD DATA-DRIVEN DECISION-MAKING

How I’m Visualizing Assessment Data For Students

THREE CHALLENGES I’M FACING IN MY ELL NEWCOMERS CLASS & HOW I’M RESPONDING TO THEM

Additional suggestions are welcome.

If you found this post useful, you might want to consider subscribing to this blog for free.

You might also want to explore the over 600 other “The Best…” lists I’ve compiled.