Last year, two very talented educators — Ted Appel, the extraordinary principal we have at our school, and Kelly Young, creator of much of the engaging curriculum we use at our school through his Pebble Creek Labs — brought-up the same point in separate meetings with teachers at my school: The importance of not being “data-driven” and, instead, to be “data-informed.”
These conversations took place in the context of discussing the results of state standardized tests. Here’s the point made by Ted:
If schools are data-driven, they might make decisions like keeping students who are “borderline” between algebra and a higher-level of math in algebra so that they do well in the algebra state test. Or, in English, teachers might focus a lot of energy on teaching a “strand” that is heavy on the tests — even though it might not help the student become a life-long reader. In other words, the school can tend to focus on its institutional self-interest instead of what’s best for the students.
In schools that are data-informed, test results are just one more piece of information that can be helpful in determining future directions.
Since that conversation took place, I’ve written several posts about the topic. I thought it might be useful to bring together several related resources.
Here are my choices for The Best Resources Showing Why We Need To Be “Data-Informed” & Not “Data-Driven”:
First, I’m going to list the post I wrote immediately after that conversation – “Data-Driven” Versus “Data-Informed”
Next, a Dilbert cartoon that Alexander Russo shared on his blog:
The cartoon reminded of what the New York judge said earlier this month when he ruled that the School District can publicly release the names of teachers and their “Teacher Data Reports.” Here is what the judge said (and I kid you not):
“The UFT’s argument that the data reflected in the TDRs should not be released because the TDRs are so flawed and unreliable as to be subjective is without merit,” the judge wrote, citing legal precedent that “there is no requirement that data be reliable for it to be disclosed.”
Data-Driven…Off a Cliff is the title of an excellent post by Robert Pondiscio.
An article in Educational Leadership is a year-old, but it’s new to me and certainly worth sharing. It’s called The New Stupid, and has the subtitle “Educators have made great strides in using data. But danger lies ahead for those who misunderstand what data can and can’t do.” It’s written by Frederick M. Hess.
It’s an article worth reading (though I do have concerns about some of its points), and relates to what I’ve written about being “Data-Driven” Versus “Data-Informed.”
Here are a couple of excerpts:
…the key is not to retreat from data but to truly embrace the data by asking hard questions, considering organizational realities, and contemplating unintended consequences. Absent sensible restraint, it is not difficult to envision a raft of poor judgments governing staffing, operations, and instruction—all in the name of “data-driven decision making.”
First, educators should be wary of allowing data or research to substitute for good judgment. When presented with persuasive findings or promising new programs, it is still vital to ask the simple questions: What are the presumed benefits of adopting this program or reform? What are the costs? How confident are we that the promised results are replicable? What contextual factors might complicate projections? Data-driven decision making does not simply require good data; it also requires good decisions.
The Truth Wears Off: Is there something wrong with the scientific method? by Jonah Lehrer is an exceptional article from The New Yorker. David Brooks from The New York Times wrote a nice summary of the article:
He describes a class of antipsychotic drugs, whose effectiveness was demonstrated by several large clinical trials. But in a subsequent batch of studies, the therapeutic power of the drugs appeared to wane precipitously.
This is not an isolated case. “But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain,” Lehrer writes. “It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable.”
The world is fluid. Bias and randomness can creep in from all directions. For example, between 1966 and 1995 there were 47 acupuncture studies conducted in Japan, Taiwan and China, and they all found it to be an effective therapy. There were 94 studies in the U.S., Sweden and Britain, and only 56 percent showed benefits. The lesson is not to throw out studies, but to never underestimate the complexity of the world around.
Talking To Students About Their Reading (& Their Data) is a post I’ve written.
“Using data for progress, not punishment”
In a Data-Heavy Society, Being Defined by the Numbers is by Alina Tugend at The New York Times.
Data-Driven Instruction and the Practice of Teaching is by Larry Cuban.
The Obituaries for Data-Driven ‘Reform’ Are Being Written is by John Thompson.
California Governor Puts the Testing Juggernaut On Ice is by Anthony Cody at Education Week.
Making the wrong “Data-Driven Decisions” is by Carl Anderson (thanks to Dean Shareski for the tip).
Data-Driven To Distraction appeared on Larry Cuban’s blog.
Larry Cuban has written another interesting post titled Jazz, Basketball, and Teacher Decision-making. John Thompson relates it to school data at Thompson: Duncan Can Shoot — But Can He Rebound?
“Not everything that matters can be measured”
“You Are Not An Equation” (And Neither Are Your Students)
Policy by Algorithm is a nice post over at Ed Week.
Professional Judgment: Beyond Data Worship is by Justin Baeder at Education Week.
This Is Why Our School is “Data-Informed” & Not “Data-Driven”
Bias toward Numbers in Judging Teaching is by Larry Cuban.
The False Allure Of Statistics is by John Thompson.
‘Moneyball’ and making schools better is by John Thompson.
Here’s Another Reason Why We Need To Be Data-Informed & Not Data-Driven
Data Gone Wild
“Why Do Good Policy Makers Use Bad Indicators?” is by Larry Cuban.
New Hope for the Obama/Gates School of Reform is by John Thompson.
“It’s amazing how much it’s possible to figure out by analyzing the various kinds of data I’ve kept,” Stephen Wolfram says. To which I say, “I’m looking at your data, and you know what’s amazing to me? How much of you is missing.”
This is the last paragraph of Robert Krulwich’s article at NPR, titled Mirror, Mirror On The Wall, Does The Data Tell It All? In it, he compares authors of books, one by Stephen Wolfram, creator of a the Wolfram search engine, and Bill Bryson, author of a biographical account of growing up in Iowa. The column, though not specifically about schools, hits a “bulls-eye” on our current data-driven madness.
What Does “Stop & Frisk” Have To Do With What’s Happening With Our Schools?
What Does The NYPD Have In Common With Many Data-Driven Schools?
Tired of the Tyranny of Data is by Dave Orphal.
Big Data Doesn’t Work if You Ignore the Small Things that Matter is from The Harvard Business Review.
Test Scores Often Misused In Policy Decisions is from The Huffington Post.
The Data-Driven Education Movement is from The Shanker Blog.
Invisible Data is from Stories From School.
Don’t Let Data Drive Your Dialogue is from The Canadian Education Association.
“The Goal Is The Goal”
On the Uses and Meaning of Data is by David B. Cohen.
Friday Thoughts on Data, Assessment & Informed Decision Making in Schools is from School Finance 101.
The New York Times Has Discovered The Perils Of Being Data-Driven — I Just Wish Arne Duncan Would, Too
Here’s a Part One and Part Two series of posts on the use of data in education, and they’re both from Larry Cuban’s blog.
Data: No deus ex machina is by Frederick M. Hess & Jal Mehta.
Bill Gates is naive, data is not objective is by Cathy O’Neil and is really good.
Bill Gates and the Cult of Measurement is by Anthony Cody.
Sure, Big Data Is Great. But So Is Intuition. is from The New York Times. Here’s an excerpt:
It’s encouraging that thoughtful data scientists like Ms. Perlich and Ms. Schutt recognize the limits and shortcomings of the Big Data technology that they are building. Listening to the data is important, they say, but so is experience and intuition. After all, what is intuition at its best but large amounts of data of all kinds filtered through a human brain rather than a math model?
At the M.I.T. conference, Ms. Schutt was asked what makes a good data scientist. Obviously, she replied, the requirements include computer science and math skills, but you also want someone who has a deep, wide-ranging curiosity, is innovative and is guided by experience as well as data.
“I don’t worship the machine,” she said.
Beware the Big Errors of ‘Big Data’ is from Wired.
The NYPD Probably Didn’t Stop All That Crime
Data-Informed Versus Data-Driven PLC Teams is from All Things PLC.
David Brooks, who generally loses all coherence when he writes explicitly about education issues, has just written an eloquent case for the importance of being data-informed, and not data-driven. Read his column titled What Data Can’t Do. Here’s an excerpt:
December 12, 2010
by Larry Ferlazzo
I support developing more effective ways to evaluate teachers — using multiple measures.
What I don’t support, however, is the present effort by the Gates Foundation that’s spending millions of dollars using student scores on standardized tests as THE MEASURE used to evaluate teachers.
I have no objection to scores from existing standardized tests being a part — a small part — of those multiple measures. If present efforts to create a “new generation” of state assessments actually invite teachers to work with them and develop more accurate performance-based assessments, I would have no objection to their proportional weight being increased — a little.
Accomplished California Teachers (of which I am a member) published a report earlier this year that I think accurately reflects my thinking on teacher evaluation:
To support collaboration and the sharing of expertise, teachers should be evaluated both on their success in their own classroom and their contributions to the success of their peers and the school as a whole. They should be evaluated with tools that assess professional standards of practice in the classroom, augmented with evidence of student outcomes. Beyond standardized test scores, those outcomes should include performance on authentic tasks that demonstrate learning of content; presentation of evidence from formative classroom assessments that show patterns of student improvement; the development of habits that lead to improved academic success (personal responsibility, homework completion, willingness and ability to revise work to meet standards), along with contributing indicators like attendance, enrollment and success in advanced courses, graduation rates, pursuit of higher education, and work place success.
I’ve written at the Washington Post what these ideas look like on the ground at our school (see The best kind of teacher evaluation).
I’m not going to spend a lot of time here reviewing the reams of research that have shown how evaluating teachers using student test results are unstable and inaccurate. You can find more than enough evidence for that at The Best Resources For Learning About The “Value-Added” Approach Towards Teacher Evaluation.
But right now my big concerns about the Gates Foundation efforts are how I fear they might be minimizing two key tools that can have a huge impact on improving teacher effectiveness — videotape and student surveys.
As I’ve previously written (There Are Some Right Ways & Some Wrong Ways To Videotape Teachers — And This Is A Wrong Way) Gates is funding a massive effort to videotape teacher lessons and then have them evaluated by people who have never visited the school nor have any kind of relationship with the teacher, and rate them using checklists and correlate them to value-added scores.
Contrast that way with how videotape is being used to universal acclaim at our school (led by principal Ted Appel) where a talented consultant (Kelly Young at Pebble Creek Labs), who has been working with us for years, meets with us to review an edited version of a taped lesson, with us initially giving our own critique and reflections followed by his comments. This process is entirely outside of the official evaluation process, and is focused on helping teachers improve their craft. It has been one of the most significant professional development experiences I’ve had. At my request, Kelly and I subsequently showed the video and shared our critique with my class, which was a transforming experience for all involved. Teacher Magazine will be publishing my account of that class period in early January.
As part of their massive project, Gates is also having thousands of students complete anonymous surveys evaluating their teachers and, you guessed it, correlating the answers to student test scores.
I’m a huge fan of getting student feedback. In fact, I’ve posted My Best Posts On Students Evaluating Classes (And Teachers). To help students see that I take their responses seriously, I always reprint the results in this blog (you can see them and the questions at that “The Best…” list) and email the results to teachers and administrators at my school.
But I want to know more from students than what Gates is asking. I want to know if they think I’m patient and if they believe I care about their lives outside of school. Yes, I certainly want to know what they think I could do better, and I also want to know what they think they could do better. I want to learn if they think their reading habits have changed and, for example, when I’m teaching a history class, are they more interested in learning about history than they were prior to taking the class. I want to find-out what they believe are the most important things they learned in the class and, for many, it might be learning life skills like the fact their brain actually grows when they learn new things or the fact that they had in them the capacity to complete reading a book or writing an essay for the first time in their lives. And, in the discussion that follows (one thing I learned as an organizer is that a survey’s true use is as a spark for a conversation) we discuss all these things and many more, including the differences between what might be what we like to do best and what we learn the most from.
By trying to connect videotaping teachers to anonymous checklist evaluators and test scores, and doing the same to student surveys, I fear the Gates Foundation may succeed in framing the public conversation about these tools as just a means to one end — better scores on assessments that don’t accurately measure learning.
This minimizes these potentially powerful tools, contributes toward seeing both teachers and students as replaceable widgets, and unfortunately reinforces a school reform debate where many worship at the alter of multiple choice test results.
Using videotaped teacher lessons and student surveys for the primary purpose of connecting them to teacher evaluation by test scores is like using a Stradivarius and a Grand Piano to play “Mary Had A Little Lamb” to evaluate the musician. In both instances, the tools have far more value to everyone if used in more expansive ways.
No, we all deserve better…
(Here’s a link to the article I wrote about my evaluation)
December 4, 2010
by Larry Ferlazzo
Today, The New York Times is running two articles on videotaping teachers for evaluation purposes. They are:
Teacher Ratings Get New Look, Pushed by a Rich Watcher
Video Eye Aimed at Teachers in 7 School Systems
They both talk about a Gates Foundation-funding effort to videotape teacher lessons and then have them evaluated by people who have never visited the school nor have any kind of relationship with the teacher, and rate them using checklists.
Here’s a criticism voiced in the article that I agree with wholeheartedly:
Randi Weingarten, president of the American Federation of Teachers, which has several affiliates participating in the research, also expressed reservations. “Videotaped observations have their role but shouldn’t be used to substitute for in-person observations to evaluate teachers,” Ms. Weingarten said. “It would be hard to justify ratings by outsiders watching videotapes at a remote location who never visited the classroom and couldn’t see for themselves a teacher’s interaction and relationship with students.”
I’d call this a wrong way to use videotape of teachers.
I’ve previously written about what I think is a right way to use videotaped teachers (Now, This Is What A Useful & Effective Teacher Assessment Might Look Like).
Our school, led by principal Ted Appel, has begun having Kelly Young, an extraordinarily talented consultant on instructional strategies who we have been working with for years, videotape our lessons (I’ve written much about Kelly in this blog). He then meets with us to review an edited version of the tape, with us initially giving our own critique and reflections followed by his comments. This process is entirely outside of the official evaluation process, and is focused on helping teachers improve their craft.
This process has been universally acclaimed by teachers so far, and it has been one of the most significant professional development experiences I’ve had.
As I mentioned in that previous post on my videotaped lesson, I had suggested to Kelly that we show the video and discuss the critique with my class as an experiment.
We did this a few days ago, and it was truly an amazing one hour.
I’ve written an article for Teacher Magazine about what happened, and they’ll be publishing it after the holidays. After reading it, I think you’ll agree that there are far better ways to use videotaped lessons than what the Gates Foundation is planning.
March 28, 2010
by Larry Ferlazzo
As regular readers know, I think very highly of Bill Ferriter, writer of The Tempered Radical blog and a colleague in the Teacher Leaders Network.
He’s just written a very thoughtful and honest post titled “Learning from the Met: Great Expectations?” It’s his reflection on the year he spent teaching in a non-suburban school, why he left, and how it connects to a recent survey of teachers from MetLife that show over 50% of teachers questioning whether every student could succeed academically (you can read my thoughts on the MetLife survey here).
Bill concludes his post this way:
Nope. I don’t think every student can succeed academically.
But instead of being the result of unmotivated or incapable children, that’s a direct result of the callous and under-informed approach that policymakers take towards addressing the challenges of students living in high-poverty communities.
Their unwillingness to invest tangible resources—dollars, people and time—equitably instead of equally is evidence of our unwillingness to care for other people’s children as much as we care for our own.
Maybe I’m not the only one who should be ashamed.
I can understand, and agree, with most of what Bill has written in his post.
Except for the part about not believing that every student can succeed academically.
I believe they can. And I also believe that most of my colleagues feel the same way, and it’s a school culture that is supported by our principal, Ted Appel. And it’s that belief which keeps me going.
Many of the students at our inner-city high school have huge challenges — not having a home situation that can provide many educational enrichment activities; lack of health insurance; unstable family life; self-control issues; gangs; English as their second language, etc.
But, though they might have a long list of deficits, they also have many assets — their potential; their life experiences; their resiliency.
Many years ago, I had a conversation with a man who worked with Gandhi in the struggle for Indian independence. He told me, “Larry, the key to Gandhi’s success was that he looked at every problem as an opportunity, not as a pain in the butt.”
Now, I’m not sure that Gandhi would have put it in quite the same way. But I’ve been able to use that pearl of wisdom as a key guide in my life.
I teach in Sacramento’s largest inner-city high school. I experience many of the typical frustrations of any inner-city teacher, and I write about many of them in this blog.
I agree with Richard Rothstein, who writes that we can only narrow, not bridge, the achievement gap without public policies that will impact the problems outside the schoolhouse doors that affect student learning. And there are some days when I come home feeling emotionally-drained and wonder what it might be like teaching at a suburban school. And there are students who — for one reason or another — I am not able to reach during an entire school year, and have hopes that some other teacher will down the line.
But those days and disappointments are more than off-set by the successes I see — the students who had never read a book before and now are doing so regularly; the ones who are able to develop their own capacity for self-control and discipline; the boys and girls (and young men and young women) who go on to college after telling me in ninth-grade that they don’t need to work on their writing because they would never need it as a professional skateboarder or professional basketball player.
As a teacher, I’m a subscriber to what New York Mets pitcher Tug McGraw said in 1973 when the team improbably won the National League pennant (I’m a New York native), “You gotta’ believe!”
(You might want to also read what Renee Moore, another Teacher Leaders Network colleague, writes about the same topic)
November 24, 2009
by Larry Ferlazzo
I read, hear, and even write a lot about “techniques” that are supposed to improve schools and classroom instruction. Often times, professional development books and workshops (and teacher hand-outs at staff meetings) are filled with zillions of them — how to use multiple intelligences, technology, specific instructional strategies with students that have special needs, etc.
These techniques are obviously important.
I wonder, though, if we teachers and our students, schools, and districts might be better off if we spent a little more time focusing on — for lack of being able to come up with better terms — our “cultural orientations” or basic “ways of thinking”?
What am I talking about?
Please bear with me as share my thinking on all this. Usually, I don’t post a piece like this which is more of a “process post” — I don’t necessarily have as much clarity as I would like, and, instead, am sharing my thoughts and hoping that feedback from readers will helping move my thinking along.
Last week marked the 100th anniversary of Peter Drucker’s birth. Drucker was the renowned business and management philosopher, writer, theorist, analyst. His thinking also says a lot to community organizing (my previous career) and teaching (my present one) Someone (and I’m sorry that I can’t remember who) wrote about National Public Radio’s coverage of this anniversary, which pointed out that his most important idea was:
the importance of a company having a sense of mission or a purpose, and that that’s not identical with its strategy, it’s not identical with its business model, it’s why it exists and what social good or greater good that it’s serving. That’s a very important Drucker idea.
When I’m talking about a “cultural orientation” or “way of thinking,” I think mean something like what Drucker meant. But something more than “whatever is good for kids.”
I’d like to give three examples of what I mean — in the classroom, in a school and, in the context of schools connecting with parents.
IN A CLASSROOM
In the first part of each school year, in most of my classes I lead a discussion with students asking what they want our class to be — “A Community of Learners” or a “Classroom of Students.” I write about this more extensively in my book “Teaching English Language Learners: Strategies That Work” (which will be out next summer), but I’ll give a short description here.
I write the two columns on the overhead and give some examples of the difference between the two. In a classroom of students, a teacher does most of the talking. In a Community Of Learners, students work in small groups and are co-teachers. In a “classroom” people laugh when others make mistakes, while in a “community” people are supported when they take risks. In a “classroom” the teacher has to be always be the one to keep people focused, while in a “community” students take responsibility to keep themselves focused.
Most students say their previous classes had been more like a “Classroom of Students.” I ask students to share what other differences they might see between the two types. Here are a couple of examples students said this year:
In a “classroom” “students start a fight and end up hurting each other.” In a “community” “they don’t start a fight, they talk it out.”
In a “classroom” “the only way to succeed is doing exactly what the teacher says.” In a “community” “you have more than one choice in succeeding.”
After adding to the list, students then decide which one they’d rather have. No one has every chosen a “classroom of students.”
By starting with this basic “cultural orientation” or “way of thinking,” students developed their own ways of approaching (I guess you could almost call it their own “techniques”) how the class would operate. It provided a framework for looking at numerous issues throughout the whole school year, and respected their judgment and wants.
IN A SCHOOL:
Ted Appel has done a tremendous job working with teachers over the past few years at our school to develop a “cultural orientation” or “way of thinking.” Basically, it’s not acceptable for students to not do well — everybody succeeds. That way of thinking operates almost universally among the faculty, and is amazingly prevalent among students as well.
Our tutoring project, where students hire (and fire) teachers of their choice, is an example of this way of thinking. We didn’t set-up an after-school tutoring center and then blame the students for not showing-up. Ted and our staff began with the thinking that some students needed help, and looked at what were the barriers to them getting the most effective assistance they could get so they could do well and thought outside the box.
IN A PARENT ENGAGEMENT STRATEGY
In my book, Building Parent Engagement In Schools, I highlight the differences between parent involvement and parent engagement. Some of those differences include the primary “involvement” tool schools use is their mouths to talk, while the primary “engagement” tool is their ears to listen. Involvement is often about one-way communication, while engagement can be about two-way conversation. The invitation to involvement is often “irritating” — challenging parents to do something the schools want them to do, while with engagement it’s often “agitation” — challenging parents to do something that they say they want to do.
Obviously a few examples are useful to illustrate each of those parent engagement elements, but if schools are committed to that kind of criteria, they can judge their own possible actions against them. They don’t necessarily need a long laundry list of what they should or shouldn’t do.
I guess all I’m wondering is how many schools and districts are skipping looking these big kinds of cultural orientations or ways of thinking?
I wonder if there should be more of an investment in developing our compasses instead of giving us road maps?
What do you think?
November 19, 2009
by Larry Ferlazzo
Our very talented and innovative principal Ted Appel (you’ll be able to read an interview with him in Public School Insights that Claus von Zastrow will post there in a couple of weeks) recently came up with another excellent idea — this one related to students getting tutors.
Our school is a recipient of a multi-year federal grant to support Small Learning Communities (we have seven SLC’s which are made up of roughly 300 students and twenty teachers each — those students pretty much stay in those groupings for their high school career). Part of that grant provides substantial funding for “tutoring.”
Ted thought, “Why not use the money to have students hire teachers of their choice as tutors and they can arrange the time (before –school, lunchtime, after-school, weekends) and location?”
As Ted explained it to me, many schools might use these kinds of funds for “tutoring centers” that don’t easily offer the flexibility that might be needed to make tutoring an attractive idea to students.
Of course, the National No Child Left Behind Act also requires all schools in Program Improvement to offer the students the option of receiving tutoring paid by NCLB funds (known as supplemental educational services, SES). There has been a fair amount of criticism of the unevenness of those providers (see Research shows key NCLB provision not helping students and Evidence Thin on Student Gains From NCLB Tutoring ), and often it’s difficult for those tutors to really communicate well with students’ teachers to coordinate on curriculum and student strengths and challenges.
The way that our school has set it up has teachers identifying students who appear that they might need tutoring and initiating a discussion about how the program works. It’s quite simple — we have a list of teachers at the school who have said they’d be open to being a tutor; the student can review the list (with help from a teacher who knows him/her and who can provide suggestions on who might be a good fit); the student arranges to “interview” potential tutors; and then they develop a contract that is agreed to by all parties (including parents). There can be one or twenty sessions, and the teacher is paid the regular hourly rate paid by the District for extra activities.
I love the way it has transformed some conversations I’ve had with students. Several of my students are having major challenges in their math classes. I’ve been able to approach them to share my concern about what I hear from their math teacher, and explain to them that they could get their own individual tutor; they could hire a teacher of their choice; they can interview several if they want; and then they help determine when and where the tutoring takes place. If they try it out, and don’t like it, then they can fire their tutor and find someone else. We review the names of available teachers, and I can help them narrow down who might be a good fit — temperamentally and language and content-wise. It sends a message, I believe, that we really are going the extra mile, and that they have power in their hands. It maximizes the benefit for students as well, since it’s very easy for teachers to communicate with each other about the individual student, and since we have common curriculum in many of our classes, students won’t using content that is “parachuted” in by an outside provider.
As Ted puts it, “It reflects our school’s thinking. We don’t just want to put on a tutoring center after school, say we provided it, and then blame the kids for not coming. ”
Even if some students don’t follow-through, it removes an excuse that they can give themselves (and others) for why they aren’t doing well.
Are any of your schools doing anything like this?