Last month, I wrote about a questionable study that touted the idea of “loss aversion” as a successful strategy to get students to do better at tests. In other words, they gave students a trophy or some cash before they took the test and told them they would have to give it back if they didn’t do well. As one of the researchers told an interviewer:
Yeah, it’s hard when you rip a trophy out of the hands of an eight year old.
You can read more about that study at Can’t Economists Stay Away From Schools? Don’t They Have Enough Other Things To Do? and at Part Two Of “Can’t Economists Stay Away From Schools?” — My Worst Fears Realized.
Well, most of those same economists, with the addition of Roland G. Fryer, Jr, are back at it. This time, though, they’re applying the idea of “loss aversion” to teachers (thanks to the Shanker Blog for the tip).
In this new study, Enhancing the Efficacy of Teacher Incentives through Loss Aversion: A Field Experiment, they found that if they gave teachers several thousand dollars at the beginning of the year and told them they’d have to return it if their students didn’t do well on math tests, then students did better on those tests (though it doesn’t appear to me to be significantly better and it didn’t track results past one school year — I’d be interested in hearing from others more versed on statistics, and readers might be interested in The Best Resources For Understanding How To Interpret Education Research).
I questioned what kind of positive classroom culture a “loss aversion” strategy would create with students, and I wonder what kind of affect a similar plan with teachers would have on school culture. The usual kind of teacher merit pay is bad enough (see The Best Resources For Learning Why Teacher Merit Pay Is A Bad Idea), but it seems to me that this kind of threatened “take-away” strategy might even be more offensive. It seems to exemplify what behavioral economist Dan Ariely said as part of the National Research Council report criticizing the role of testing in education:
“These policies are treating humans like rats in a maze. We keep thinking about how to reorganize the cheese to get the rats to do what we want. People do so much more than that.”
Abraham Maslow said “If you only have a hammer, you tend to see every problem as a nail.” It seems to me that many (though not all) economists see teachers and students only through the lens of financial incentives.
If these economists have money they want to spend on seeing what improves academic achievement in schools, I’ve shared my suggestions for how they spend it in a Washington Post column.
I also wonder how the professors who wrote the study would feel if their salaries incorporated a loss aversion component?
I suspect they might say their situation is different because, as they claim in their study, “teacher quality and aptitude has declined significantly in the past 40 years,” so we K-12 teachers are the ones who need this kind of program — not them.
The teachers I work with wouldn’t take the money and ask for it to be put into the supplies and materials budget.
Larry,
The study isn’t as crazy as you might think — one of the arguments against the type of incentive study Fryer’s done in the past is that it has ignored the behavioral economics literature (which is often really cognitive psych literature), which includes the asymmetry of incentives (i.e., we feel loss and potential losses more acutely than gains). This study addresses that criticism.
I’m at home and don’t have the same type of access to NBER papers as I do on campus, but from the abstract I don’t think you could implement this type of program outside a research context with informed consent. At least in U.S. law, you can’t take away wages once agreed to.
In a broader context, teachers typically use a mix of gain and loss strategies in grading students. For some things, students gain points for different things they do. In other cases, I’ve heard teachers explain that students have an “A” in a category at the beginning of the semester (e.g., attendance), and they lose that “A” if they fail to perform. My vague understanding of the loss aversion incentive is that it works best if there is a concrete task students are in control of (such as attendance), and because of the power of loss aversion, it’s best to make sure that it’s in an area that makes the most difference in how students learn. S
Sherman,
I’m not questioning the sanity of the study — I’m questioning its ethics. Just because something might get the result you want doesn’t mean it’s right to actually do it — not to mention the issue of how wise it is to place increasing test scores on a pedestal as THE determining mark for academic improvement.
I also don’t question that “loss aversion” might have some reasonable role in life and in the classroom. I always apply the idea you mention — letting students know that they start the semester with an “A”, for example. It seems to me, though, that there is a qualitative and quantitative difference between doing that and physically giving cash and rewards to adults and children and then, as one of the study’s author said, ripping it out of their hands if they don’t meet certain standards.
Larry
Are you talking about the research or using this as policy? I don’t consider Fryer’s research unethical. If the research had the consent of the participants, that means Fryer et al. had to go through an informed-consent procedure approved by Harvard’s Institutional Review Board.
There’s a difference between research and policy. Social scientists conduct all sorts of experiments using designs that we would not foist on people without their consent. That’s how we discover an enormous amount about how people behave.
Sherman,
I question both the ethics behind the research and the ethics of the policy. Just because someone says it’s okay for something to be done to them does not mean — at least in my eyes — that what is being done to them can be assumed to be ethical.
Larry
“they found that if they gave teachers several thousand dollars at the beginning of the year and told them they’d have to return it if their students didn’t do well on math tests, then students did better on those tests (though it doesn’t appear to me to be significantly better and it didn’t track results past one school year”
What a disgustingly sick research idea. If they came to me and tried to give that money to me I’d have to stick it into them where the sun doesn’t shine, pull it out and then make them eat it-metaphorically speaking of course.
And Larry, you are correct in questioning the ethics of it
“risk loss aversion” These types of studies and the policies and practices that evolve out of them are concerned mainly about one thing–control. Control in a subtle, most of the times not to be exposed, manipulative fashion. Sorry, but that’s just wrong.
There really is one main thing for “control” in the class, as O. Redding wrote in the song A. Franklin made popular R-E-S-P-E-C-T, respect for the dignity of each individual student through not manipulating them, respecting them as they are not as you “want” them to be.
I am in agreement ethically with the points you make, Larry. In response to your request for statistical clarification, I offer you one more example, admittedly anecdotal evidence, on the interpretation of statistics in education research.
http://wp.me/p2iCdW-3M
In this study, the increase in scores was statistically significant, showing an admirable increase in understanding – until other data were examined and the story behind the scores was included.
For those who are concerned about the study not as policy but as research, please consider the way that social-science literature is often the target of political attacks, such as the attempt earlier this year to defund the political-science program in the National Science Foundation. Should research really be prohibited because someone is uncomfortable with the question being asked? If so, say good-bye to most research on sexually-transmitted diseases.
I will defend Fryer’s right to conduct this research under the supervision of Harvard’s Institutional Review Board (as my colleagues at USF conduct human-participant research under our Institutional Review Board). I urge you to as well, ESPECIALLY if you disagree with the model assumptions of the authors.
Sherman,
I haven’t heard anyone say Fryer doesn’t have a right to do the research. However, having the right do something doesn’t mean it’s always a good idea to do it. In this study, as the sentence I quote clearly indicates, and in many of his other studies, he clearly is pushing a policy agenda. When a researcher enters that realm, he/she better expect a pushback. That’s what public life is all about…
Larry
Sorry, I have to disagree that research for research sakes should be conducted even with supposed peer review. First off, there is no such thing as “social science”. Studying societies, people, etc are not amenable to “scientific study”. No, it’s a study and I don’t have a problem with that, but to say that it is “scientific” is fantasy at best if not lunacy. It’s a bastardization of the concept of “scientific study” to try to use “science” to describe what are better described as “studies’
Duane,
Fine — don’t call it social science. Call it “swooshing around with people and ideas.” Still want to punish folks who do “swooshing around with people and ideas” based on the content of their ideas?
Sherman,
Not interested in punishing anyone, never liked that concept of punishment, grew up in the catholic church and rejected that crap a long time ago. Only interested in obtaining practices and results that can come close to “truth” and those that, in education “cause no harm”.
I have to agree with Sherman (though I think there might be a couple of different arguments going on here). There is a distinction between research and policy, and dismissing the former as wrong or a bad idea because you don’t like how it’s being used for the latter is a dangerous road (and, as Sherman notes, threatens much of social science research).
That’s not to say that I don’t personally struggle with this distinction (I do), and one can hardly blame teachers for being fed up with all this stuff. It’s easy for me to be detached when it’s not my life and job that might be affected.
But this paper is one small part of a body of evidence from multiple disciplines, including Fryer’s own prior work. It has legitimate research value. Not everyone will keep every study in perspective, but it’s not the existence of the research that is responsible for that.
MD
Matt,
Are you really suggesting that before researchers embark on a study that they should not consider how the results might be inappropriately used? I’m not trying to put words in your mouth, just trying to make sure I have an accurate understanding of your position. Of course, even if you’re not suggesting that, it’s possible that this may very well be a common perspective in the world of academia — that arena is fairly unfamiliar to me. And that would be troubling….
I have to reiterate, too, that, even if that is the case, it’s pretty clear in words used in this study, and in past actions by Fryer, that he uses his research to push a clear policy agenda. I’m not just talking about others using the research in ways I don’t agree with — I’m talking about the researchers using it to promote policies I don’t agree with. And I’m not just singling out Fryer as doing this — the Chetty study from earlier this year and the subsequent comments by the researchers made that pretty clear:
http://larryferlazzo.edublogs.org/2012/01/07/the-best-posts-on-the-ny-times-featured-teacher-effectiveness-study/
I appreciate both you and Sherman’s engagement in this dialogue. It’s thought-provoking to me, and I’m sure it’s the same to other readers.
Larry
Well, no, that wasn’t my point. And I don’t feel comfortable making a blanket statement as to whether researchers should consider how their work will be used. The best I can do is: Occasionally, definitely yes; usually, it’s up to them.
I agree that Fryer’s views on this particular policy often seem evident (though I try not to assume too much, and I’m guessing his opinions are nuanced). But, even if he was pushing an agenda, as you say, I don’t see how that makes the research a bad idea. Would you say the same thing if someone was using their work to argue against merit pay? If not, then it sounds like this discussion boils down to policy disagreements, rather than the value or proper use of research.
MD
Matt,
Your points are well taken — at least related to the merit pay study. I do have to say, though, I still have questions about why a researcher would not always consider how their work would be used by others. I know it’s probably unfair to say this, but I’ve got to say I’m reminded of lyrics from the old Tom Lehrer song:
“Once the rockets are up, who cares where they come down?
That’s not my department,” says Wernher von Braun.
In regards to merit pay’s “sister study” on using loss aversion with students, I doubt if anyone is every going to be able to convince me that it’s ethical to give trophies to little children and then “rip them out of their hands” in the name of research.
Larry
Matt,
A problem with policy-driven research is that, by definition, it has an agenda. That may not sound important, but research is a nasty dish when dealing with ethics, and the question of ethics goes beyond “Is this ok to do to people?” Researchers, those with or without an agenda, can (un)intentionally avoid variables that could discredit their premise. But with the US’s history of using studies to drive education policy, my biggest problem is that research, including the research being discussed, too often removes human experience.
I don’t just want to hear about test scores. I want to know what happens to the students whose trophies are taken back. If it was me, I would be bitter and embarrassed seeing my peers with their trophy still tightly grasped and mine back in the teacher’s desk. Or maybe I would pretend it didn’t bother me. Feign indifference, say “I don’t care about school/that trophy anyway.” Then the next test comes. Do I get another trophy before this test, or do I get a promise (again) that if I do well I’ll get my lost trophy back? Suppose, again, that I don’t do well. I urge everyone to imagine the consequences of continual loss. On the other hand, what if I do well on the test? I do well, and I keep my trophy. Then the next test comes. And I still keep my trophy. Maybe I get a new one. But is a trophy going to satisfy anyone through their whole school career? Even if it could, is that we want? A nation of young people who will strive to succeed only when offered incentive?
I understand this is an initial study, and I don’t know where they plan to take their findings next, but it seems right now that this study is trying to bake eggs to make a cake. There’s too much missing for anything worthwhile to emerge.
Travis,
I think Larry was talking in general about loss aversion. As far as I’m aware, no trophies were given to children or taken away as part of this study.
Sherman,
The “sister” loss aversion study to the merit pay one, done by most of the same researchers and in the same schools and at the same time with students, did, in fact, involve giving — and taking away — trophies from children as young as eight years old. I include the link to the network interview where one of the researchers said it was hard to “rip” trophies out of the arms of the children in the study.
Larry
“Motivation is a fire from within. If someone else tries to light that fire under you, chances are it will burn very briefly.” ~S. Covey
This sort of “money matters most” solutions also explicitly celebrate financial success, over-elevate the significance of any single measurement of student achievement, and require extensive measures to prevent corruption. It’s a clever, yet inappropriate, use of behavior economics in public schools. It’s also impossible to actually implement so let’s refocus on what can make a difference in our students’ lives today.