Standardized Errors

What is it with standardized tests? They are the primary method of evaluation for the No Child Left Behind law (NCLB) and “school accountability”. Student test scores are being used as the principal data source for “value-added” assessments of teachers, and from that for “merit-based” salary and retention decisions. Many universities and workplaces use standardized tests to influence admissions, hiring, or promotions. Most people have settled, and largely incorrect, opinions about standardized tests. Almost universally, people believe that standardized tests accurately measure something, either basic intelligence or specific knowledge. Even people who consider themselves smart, but who don’t perform well on standardized tests, assume that their test results indicate that there’s something lacking in their level of intelligence or knowledge. They assume the test is correct.

To all of this, let me simply (to begin with) say what more and more researchers have been discovering and demonstrating in recent years: Standardized tests primarily measure one thing—a person’s ability to take a standardized test. That’s about it. What that means, as a result, is that the best way to improve your score on a standardized test is to practice taking standardized tests. The corollary reality is that whatever anyone does to prepare for a test will have very limited impact on their score.

Those are the simple facts. There are reasons why minorities and women, on average, don’t score as well as white men on tests. But it’s not all white men. Because of life experiences, upper middle class white men have an advantage over anyone in the lower middle and lower classes, in any situation where decisions are based on test scores.
In case you haven’t heard, there is a requirement in the No Child Left Behind law that every student in every school be able to pass the proficiency exam by 2014? That is a ticking time bomb for all public schools. The requirement is an impossible task, one akin to requiring every person in the U.S. to “proficiently” sing a Verdi aria and a Lady Gaga song. No matter what, there will be students who cannot “demonstrate proficiency” if it means passing a standardized test.

Perhaps you, like most people, do not believe that. Maybe you think that anyone can be taught to get 70 percent right on a generalized test, given skilled teachers and enough preparation time. Maybe you believe that to be a relatively simple task. After all, what could get in the way of that? What could? Plenty. Let me give you a list of the human limits students and their teachers constantly come up against.

a. Limited background knowledge. It’s not just curriculum. The people who write and review standardized tests are almost entirely white, euro-centric, and upper-middle class. The students who take the tests have a wide variety of life experiences and cultural expectations, but most of them have not grown up in the same social milieu as the test makers. A person’s background knowledge determines how they respond to new information as well as to “logical” assumptions and arguments. This includes the many people who grew up in non-European cultures and speaking languages other than English. It’s not simply that test-takers lack knowledge, it’s their approach to knowledge and reasoning that can be different. And that fact can, and does, affect their test scores.

b. ADD/ADHD. People with Attention Deficit (Hyperactivity) Disorder have difficulty maintaining focus on one activity. Imagine the difficulty of answering questions, one after another, when in the middle of a paragraph you get distracted by another student coughing or erasing an answer and you have to start over on that question. Got your answer? Before you can find your place on the bubble sheet a random thought pops into your head, and by the time you manage to dismiss it you forget your answer, and you have to start over. Or worse, you get distracted and fail to fill in a bubble and then you answer twelve questions on the wrong answer lines before you realize your mistake, and you have to erase them all … and start over.

c. Dyslexia. Students with many forms of dyslexia have difficulty distinguishing written language. Printed words can seem fuzzy, or vibrating rapidly, or reversing so that it’s hard to tell between a “d” and a “b”, or a “b” and a “p”, or all three of these conditions at once, or more. As you might expect, this mipht  affect someone’s adility to read and comqrehenb the questions on a test, and that becomes even more of a problem when there is a limited time to complete all the answers.

d. Vision problems. Many students have common conditions such as near- and far-sightedness, astigmatism, and accommodation difficulties. For a variety of reasons, including poverty and social stigma, these conditions are often not corrected adequately. These problems are not generally as severe as dyslexia, but can also make the testing process difficult.

e. Weak motor skills. In order to maximize scores on a standardized test it is important to fill in every answer “bubble” with a solid dark mark that completely fills the small space within the lines on the answer sheet, without leaving any traces outside the correct bubble. If your mark is not dark or wide enough it may not register. If you inadvertently allow your pencil tip to contact the answer sheet while you are concentrating on the question, or while using your pencil as a pointer to find your location on the answer sheet, or you don’t erase a wrong answer completely, and that leaves a mark in the wrong bubble, the automatic scoring machine may not accept your answer. And unlike the process in a voter recount, there will not be a human reviewer looking at your answer sheet to make a reasonable effort to correct any machine errors.

f. Low self esteem and test anxiety. There are students who absolutely believe they cannot score well on any test. They have learned this from parents or teachers and have internalized it. It has become a self-fulfilling assumption. When they get into a test situation their brain and body conspire to defeat them, through confusion, anxiety, denial of logical results, rejection or “loss” of previously learned knowledge.

g. Stereotype threat. Related to low self esteem is the stereotype problem, activated in “any situation where an individual faces the potential of confirming a negative stereotype”. Any common sociocultural stigma will enhance this effect. A woman who is reminded, before a math test, that women in general do not perform well in math, will not do as well on the test as she would have if she had not received that reminder. Underlying social assumptions do affect how humans respond to life experiences.

h. Low motivation. There are many, many reasons why some students simply don’t care about testing. This can be the result of low self esteem, but even some students who do very well in all other aspects of school, and who do well on college entrance tests, don’t bother trying to do well when faced with a standardized test that will not impact them personally. Students have been known to mark answers randomly, or in visual patterns of bubbles, or not to bubble answers at all.

i. Other mental or physical conditions. There are many of these. There is verbal blindness, in which the connection between the visual recognition center of the brain is separated from the verbal center, so that a person clearly sees words and letters but cannot recognize them as related to meanings or sounds. There are degrees of this; it was said that Picasso had difficulty with math because, for example, the numeral four looked like a nose. There are similar conditions like aphasia and alexia. There are other people who strongly link specific words with colors and smells, so they are frequently distracted by intrusive sensory irrelevancies. There are temporary emotional and health conditions that will affect performance on test day. The human nervous system is an extremely complex structure.

The items listed above are not exclusive. Many people have several of these conditions, and the effects on their test scores are accumulative. The result of all this is that on any test there will always be a large percentage of people who will not perform to a level that represents their true abilities. On any test there will also always be a small percentage who will not be able to “pass”, or to “display proficiency”, however proficient or intelligent they may be in a non-testing situation.

Standardized tests are limited and oversimplified assessment tools that will always misrepresent reality. It is high time we stop trying to use them to make important societal decisions.

Note about the author of this article: No, I am not a disgruntled failure at test taking. I am a person who has scored very well in every standardized test I have ever taken, without any efforts at pre-test studies or other forms of test preparation. ACT, SAT, GRE, LSAT, ASVAB, NTE, and more; for some reason I have always been good at choosing the correct answers in “multiple-guess” testing. I have also administered many standardized tests and served as a test coordinator. All my experiences, quite simply, support my beginning assertion that “standardized tests primarily measure one thing—a person’s ability to take a standardized test”.


This entry was posted in Education and tagged , , , , , , , . Bookmark the permalink.