Last week, my colleague Katie Hull and I did a quick survey with our students, and I’ll be writing a longer piece about it. But, first, I need some help interpreting the results.
We asked two questions. The shortened versions were:
1) Do standardized tests accurately measure how smart you are? Why or why not?
2) Should teachers be graded on how well students do on those standardized tests? Why or why not?
Katie asked her students in mainstream classes to respond, and I had my IB students and my Beginning/Intermediate ESL students answer.
The answers to the second question were consistent in all the classes — about 85% said student test results should not be used to evaluate teachers.
It’s the answers to the first question that were surprising to me.
Among my IB students, the ones who score the highest on the tests, only 5% said they were accurate measurements of their intelligence.
In Katie’s mainstream classes, about 20% said they were accurate measures.
In my ESL classes, comprised of students who have the most difficulties with the tests, over 60% said they were accurate measures.
How would you interpret these survey results?
Hi Larry,
I’m going to take a stab at it.
My guess is that people have a tendency to undervalue things that are easy, and overvalue things that are difficult. The mentality among the high-performers may be that “That was not challenging, I know I can do more than that.” Therefore, it’s a poor measure of what I know.
Among the low-performers, the attitude may be, “This is really hard. Only the smart kids do well on these tests. That’s not me.” The tests may be confirming what the students already believe about themselves – that they’re not good at school or not good at taking tests, or not smart, etc. Therefore, it’s a “good measure.”
That’s all speculation, of course. You’ll have to ask the students themselves for more information to get the real answer.
Susan Wetenkamp-Brandt
Susan,
Good thoughts. Thanks!
Larry
I’m curious as to why you asked the students whether the tests produced an accurate assessment of their intelligence (as opposed to an accurate assessment of what they have learned). These are far from being the same thing (and it looks as if your more advanced students were on to this distinction, while your strugglers were not).
Hi Larry
Difficult to know without knowing more about your study.
There may be cultural reasons in your ESL students, depending where they come from (they may come from countries where standardised testing is widely considered to be a good way of measuring intelligence).
Also I presume that your lower and intermediate ESL students are not necessarily students of low ability- it is just that their English is at a lower level. They may be confounding standardised testing as a measure of their intelligence with standardised testing as an accurate measure of their English language skills.
You asked them if tests measure how smart they are, so maybe they recognized that tests also measure (or at least should) how much they have learned and how hard they worked.
A better question to ask is do standardized tests accurately measure your knowledge and skills.
Hi Larry, thanks for posting this. I think those are great questions to ask your students, and I’m glad to hear their voice on the matter. I am surprised at how low those percentages are, especially given the current policy environment the kids are suffering through, but I am heartened to hear that they are apparently not buying into it. I do wonder however, if the low percentages for the first question were in response to the particular wording of the question…I don’t think any standardized-test proponent would argue that they are a measure of “how smart” anyone is, so much as “how much they have learned” on a particular topic (depending on what test, of course). Did the students include any written responses with their answer? It could shed light on the details of why they responded to #1 like that. I would love to hear whether they think standardized tests are an accurate measure of how well they know the topic the test is on. I’m guessing in that case the percentages would be higher, but hopefully still pretty low overall, because, IMHO standardized tests are not accurate measure of even what they purport to be measure of…be that “smartness” or “amount of knowledge in a particular discipline” whatever that could possibly mean.
Luke
Thanks for everybody’s responses so far. They’re very helpful, and I’ll need to think more about them.
I know that much of the rhetoric around testing says that it is designed to evaluate how much students have learned. However, in reality, it seems to me that what is more often communicated to students is that it evaluates how “smart” they are — by the labels that are given them (in CA it’s Far Below Basic, Below Basic, Basic, Proficient, and Advanced); by the messages we send to them about the kinds of classes they can take, etc. It’s similar to the labels give to schools — and, more and more, to teachers — based on student test scores: Bad, Good, Failing, etc. That was my reasoning behind the wording of the question. I’m not entirely convinced the responses would have been different if I had worded as several of you suggest, but it’s an important point to consider.
I’ll be writing a lengthier piece soon including the reasons that students gave for their answers.
Larry
Larry – Your questions and the posts that I’ve read about them leave me with two thoughts. First: The continual widening of the income gap and diversification of the American population (and therefore that in our classrooms) makes it clear that we count on a huge amount of life experience and background knowledge to drive success in schools. This is why national tests are considered by many to be biased against certain racial, ethnic, and geographic groups. It is why students who grow up in environments rich with literature, culture, science, and vocabulary tend to do better in schools. It is the reality that we can’t claim to ever measure what’s learned in a class until our grades reflect their growth instead of their achievement on some supposedly objective metric. Second: Since I believe the problem described above turns schools into places that try to teach students as learners but ends up measuring the lives they’ve lived as people, there are bound to be connections between grades and self-worth. Although these connections aren’t legitimate, only the successful students will have the confidence and fortitude to disregard the results. Others, like your ESOL students, may have unfortunately succumbed to the hype that these tests measure their intelligence and overall ability.
My reaction is that the more you understand about standardized, multiple-choice, multiple-guess tests, the less you value the results. I was always a good test taker and am not being immodest to state that at this point I qualify as a leading expert on improving performance on SAT/ACT and the like. As a kid, I was frustrated by my very good but not outstanding math score on the SAT and proud of my top-notch English score. When I last took the GRE (1991), having spent much of the intervening years between taking it the first time, in 1973, I scored 2300 out of a possible 2400. But nice as that was for my graduate school applications, I knew just how little that meant in terms of things that matter. If you’ve got kids in high school already figuring that out, I’d take that as a sign of cultural wisdom growing (or that you’ve got a wise crop of kids).
Those who do less well are likely to see their weaker performance as meaningful for a number of reasons. One is that they don’t want to be seen as devaluing the test just because they don’t blow it away. Another is that they actually have swallowed enough cultural Kool-Aid to be convinced that there’s something lacking in them that matters and which these tests truly reveal.
ELL students may simply not have enough context in which to make an informed judgment about the meaning of such tests, and hence may simply assume that if schools give them, they must be important and meaningful.
Hi Larry,
I think the article linked below clarifies a great deal. It seems from your survey results that your students are possibly (sub-consciously) aware of the basic premise that it’s hard work that gets the results…
http://m.wired.com/wiredscience/2011/10/why-do-some-people-learn-faster-2/