Rationing Education in an Era of Accountability

["PRISCILLA GUTIERREZ" <pgutpgut@msn.com>]

This is an excellent article about the unintended consquences of NCLB 
"accountability."  Mrs. Dewey's exeriences are not unusual.  I know several 
educators in 3 states who are in the same position - they've been told by 
district administrators to ignore the students who won't make enough 
progress to score well on standardized tests, and to focus on the borderline 
kids who can pull scores up and help the district make AYP.


Phi Delta Kappan
Rationing Education In an Era of Accountability
By Jennifer Booher-Jennings
November 9, 2006
http://www.pdkintl.org/kappan/k_v87/k0606boo.htm

MEET Mrs. Dewey, 46 years old and a veteran fourth-grade teacher at Marshall 
Elementary School. Mrs. Dewey entered the teaching profession in the wake of 
A Nation at Risk and has weathered the storm ever since. For the last 20 
years, she has survived the continuous succession of faddish programs that 
has characterized American education reform. Year after year, administrators 
have asked Marshall teachers to alter their practice to conform to the 
latest theory. Mrs. Dewey's colleagues, frustrated by the implementation of 
such silver-bullet approaches, have often flouted the administrative 
directives and chosen instead to serve as the sole arbiters of their 
classroom practice.

But it is the newest of the new solutions that worries Mrs. Dewey most. The 
language of accountability is swift and uncompromising: hold educators 
responsible for results. Identify those teachers who, as President Bush 
says, "won't teach." Fair enough, Mrs. Dewey thinks. The consummate 
professional, Mrs. Dewey always looks for the silver lining.

Like other reforms, accountability requires teachers to embrace a new 
strategy. Data-driven decision making, a consultant told the faculty at a 
professional development session, is the philosophy Marshall teachers must 
adopt. The theory is simple. Give students regular benchmark assessments; 
use the data to identify individual students' weaknesses; provide targeted 
instruction and support that addresses those areas. Mrs. Dewey remembers 
nodding approvingly. After all, this approach -- gathering textured 
information on each student to guide instructional activities -- was one she 
had been using for 22 years.

The consultant moved on. "Using the data, you can identify and focus on the 
kids who are close to passing. The bubble kids. And focus on the kids that 
count -- the ones that show up at Marshall after October won't count toward 
the school's test scores this year. Because you don't have enough special 
education students to disaggregate scores for that group, don't worry about 
them either." To make this concept tangible for teachers, the consultant 
passed out markers in three colors: green, yellow, and red. Mrs. Dewey heard 
someone mutter, "What is this? The traffic light theory of education?"

"Take out your classes' latest benchmark scores," the consultant told them, 
"and divide your students into three groups. Color the 'safe cases,' or kids 
who will definitely pass, green. Now, here's the most important part: 
identify the kids who are 'suitable cases for treatment.' Those are the ones 
who can pass with a little extra help. Color them yellow. Then, color the 
kids who have no chance of passing this year and the kids that don't count 
-- the 'hopeless cases' -- red. You should focus your attention on the 
yellow kids, the bubble kids. They'll give you the biggest return on your 
investment."

As the bell tolls a final warning to the boisterous 9-year-olds bringing up 
the rear of her class line, Mrs. Dewey stares blankly into the hallway. 
Never did she believe that the advice offered by that consultant would 
become Marshall's educational mantra. Focus on the bubble kids. Tutor only 
these students. Pay more attention to them in class. Why? It's data-driven. 
Yet this is what her colleagues have been doing, and Marshall's scores are 
up. The community is proud, and the principal has been anointed one of the 
most promising educational leaders in the state. At every faculty meeting, 
the principal presents a "league table," ranking teachers by the percentage 
of their students passing the latest benchmark test. And the teachers talk, 
as they always do. The table makes perfect fodder for faculty room gossip: 
"Did you see who was at the bottom of the table this month?"

Mrs. Dewey has made compromises, both large and small, throughout her 
career. Every educator who's in it for the long haul must. But this 
institutionalized policy of educational triage weighs heavily and hurts 
more.1 Should she focus only on Brittney, Julian, Shennell, Tiffany, George, 
and Marlena -- the so-called bubble kids -- to the exclusion of the other 17 
students in her class? Should Mrs. Dewey refuse to tutor Anthony, a 
persistent and eager little boy with no chance of passing the state test 
this year, so that she can spend time with students who have a better shot 
at passing? What should she tell Celine, a precocious student, whose mother 
wants Mrs. Dewey to review her entry for an essay contest? Celine will 
certainly pass the state test, so can Mrs. Dewey afford the time? What about 
the five students who moved into the school in the middle of the year? Since 
th! ey don't count toward Marshall's scores, should Mrs. Dewey worry about 
their performance at all?

In her angrier moments, Mrs. Dewey pledges to ignore Marshall's approach and 
to teach as she always has, the best way she knows how. Yet, if she does, 
Mrs. Dewey risks being denounced as a traitor to the school's effort to 
increase scores -- in short, a bad teacher. Given 22 years of sacrifices for 
her profession, it is this reality that stings the most.

Mulling over her choices, Mrs. Dewey shuts her classroom door and begins her 
class.2

Unintended Consequences of Accountability Systems: Educational Triage

Test-based accountability systems aim to direct the behavior of educators 
toward the improvement of student achievement. The No Child Left Behind 
(NCLB) Act codified accountability as our national educational blueprint, 
requiring schools to increase test scores incrementally so that all students 
are proficient in reading and math by 2014. Yet, despite the stated intent 
of NCLB to improve outcomes for all students, particularly those who have 
been historically neglected, educators and others may adopt a series of 
"gaming" practices in order to artificially inflate schools' passing rates. 
Such practices include giving students a special education classification to 
exclude them from high-stakes tests,3 retaining students in grade to delay 
test-taking,4 diverting attention away from subject! s not evaluated on 
high-stakes tests,5 teaching to the test,6 and cheating.7

In what follows, I discuss two of the dilemmas presented by a 
less-well-known gaming practice: educational triage. The insights offered 
here derive from an ethnographic study of an urban elementary school in 
Texas, to which I have assigned the pseudonym "Beck Elementary." Educational 
triage has become an increasingly widespread response to accountability 
systems and has been documented in Texas, California, Chicago, Philadelphia, 
New York, and even England.8 By educational triage, I mean the process 
through which teachers divide students into safe cases, cases suitable for 
treatment, and hopeless cases and ration resources to focus on those 
students most likely to improve a school's test scores. The idea of triage, 
a practice usually restricted to the direst of circumstances, like the 
battlefield or the emergency room, poignantly captures the dynamics of many 
schools' re! sponses to NCLB. In the name of improving schools' scores, some 
students must inevitably be sacrificed. And the stakes are high -- for 
schools, which face serious sanctions for failing to meet adequate yearly 
progress targets; for students, who increasingly face retention if they do 
not pass state tests; and for teachers, who are judged by the number of 
students they ''save."

Dilemma 1. Data can be used to improve student achievement, but they can 
also be used to target some students at the expense of others. Data-driven 
decision making has become something of a sacrosanct term in education 
policy circles. Who could be against it? The public face of data-driven 
decision making -- identifying the needs of each individual child and 
introducing interventions to remediate any learning difficulties -- is 
sensible and beyond question.

But the Achilles' heel of education policy has always been implementation. 
When I listened closely to the conversations that educators at Beck 
Elementary School had about "being data-driven," the slippage between 
evaluating the individual needs of every student and deciding which students 
to target to maximize school performance quickly became evident. As I moved 
closer and closer to the classroom, the administrators' ideal version 
dissipated and gave way to a triage-based understanding of data-driven 
decision making. Teachers were most attuned to the chasm between 
administrators' theoretical proclamations and how the same administrators 
expected them to operate: teachers understood that the bottom line in this 
numbers game was the percentage of students who passed. Because of the 
unrelenting pressure to increase test scores, one mode of using data became 
dominant at Beck: the diversion of resources (e.g., additional time in 
class; enrichment sessions wit! h the literacy teacher; and after-school, 
Saturday, and summer tutoring) to students on the threshold of passing the 
test, the "bubble kids."

All my questions about which students received extra help were met with the 
deferent maxim, "It's data-driven." When I asked one teacher how the school 
allocated additional services to students -- for example, the reading 
specialist or after-school and Saturday tutoring -- she provided the 
following response:
It's all data-driven. . . . We do projections -- how many of them do you 
think will pass, how many of them do you think will need more instruction, 
how many teachers do we have to work with, what time limit do we have. Based 
on that, who are we going to work with? It comes down to that. . . . We 
really worked with the bubble kids . . . that's the most realistic and 
time-efficient thing we can think of.
In this conception of data-driven practice, the choice to privilege one 
group of students over another is viewed as neutral and objective. The 
decision to distribute resources to those most advantageous to the school's 
pass rates is not understood as a moral or ethical decision. Instead, it is 
seen as a sterile management imperative. Protected by its scientific 
underpinnings, the data-driven focus on the bubble kids is difficult for 
teachers to attack. In sum, at Beck Elementary, the invocation of the phrase 
"data-driven" obscures, neutralizes, and legitimates a system of resource 
distribution that is designed to increase passing rates rather than to meet 
the needs of individual students.

The blunt vocabulary of triage infiltrated every corner of Beck. The tenor 
of the phrases used to describe students -- "the ones who could make it" and 
"hopeless cases" -- speaks not only to the perceived urgency to improve test 
scores but also to the destructive labeling of those children who find 
themselves below the bubble. Driven by the pressure to increase the passing 
rate, teachers turned their attention away from these students. As one 
teacher related in an interview:
I guess there's supposed to be remediation for anything below 55%, but you 
have to figure out who to focus on in class, and I definitely focus more 
attention on the bubble kids. If you look at her score [pointing to a 
student's score on her class test-score summary sheet], she's got a 25%. 
What's the point in trying to get her to grade level? It would take two 
years to get her to pass to the test, so there's really no hope for her. . . 
.. I feel like we might as well focus on the ones that there's hope for.
To say that hope is absent for a 10-year-old child is a particularly telling 
comment on how dramatically the accountability system has altered the realm 
of imagined possibility in the classroom. Now, with an unforgiving bottom 
line for which to strive, teachers can retain hope only for those perceived 
as potential passers. To assert that students below the bubble are just too 
low-performing to help establishes that the only worthwhile improvement in 
this brave new world is one that converts a nonpasser to a passer.

The problem is that those students who arrive at school as the most 
disadvantaged are often the lowest scoring. And since the focus on the 
bubble kids at Beck Elementary begins not in the third grade -- the first 
year that students take state tests -- but the moment students enter 
kindergarten, they are branded as "hopeless cases" from the very first days 
of their schooling.

An important shift occurs in a system focused on the percentage of students 
above a particular threshold. When a low-performing student enters a 
teacher's classroom, he or she is seen as a liability rather than as an 
opportunity to promote individual student growth. As Michael Apple 
trenchantly wrote, the emphasis changes "from student needs to student 
performance, and from what the school does for the student to what the 
student does for the school."9

Certainly one can imagine uses of data that could turn attention to the 
individual needs of each and every student. However, the current monolithic 
discourse on data-driven decision making begs for a discussion of unintended 
consequences. Data can be used to target some students at the expense of 
others, and it is happening today.
When we blindly defer to "the data," we abdicate responsibility for tough 
decisions, all the while claiming neutrality. But data are not actors and 
cannot do anything by themselves. Data do not make decisions; people make 
decisions that can be informed by data. Decisions about resource allocation 
are ethical decisions with which educators and communities must grapple and 
for which they must ultimately take responsibility.
What we need above all is a sustained discussion among educators and the 
broader polity about the very real tradeoffs involved in schools' responses 
to accountability systems. If schools adopt the practices of educational 
triage in response to NCLB, the consequence may be suboptimal outcomes for 
students "below the bubble," as well as for their peers who are mid-level 
and high-achieving students. And all of these unintended consequences can 
happen while official pass rates increase.

Dilemma 2. It is unfair to hold schools accountable for new students or for 
subgroups that are too small to yield statistically reliable estimates of a 
school's effectiveness; however, the consequence of excluding some students 
may be to deny them access to scarce educational resources. Educational 
triage does not end with the diversion of resources to the "bubble kids." 
Because of the fine print in NCLB, all students are not equally valuable to 
a school's test scores. Subgroups are not disaggregated if the number of 
test-takers does not meet a minimum size requirement, and students are not 
counted at all in a school's scores if they are not enrolled in a school for 
a full academic year. For example, in Texas, the scores of students who 
arrive at the school after the end of October do not count toward schools' 
scores. Such a definition is logical, for it attempts to isolate the impact 
of schools on students. Including students wh! o have not attended the 
school for a reasonable period of time might bias estimates of the school's 
quality and unfairly penalize schools serving more mobile students.

However, if resources flow only toward those students who affect a school's 
outcomes, students who do not "count" may be denied access to scarce 
educational resources. I found that another pithy term, "the accountables" 
-- those students who count toward a school's scores -- was incorporated 
into the lexicon of Beck educators. Teachers engaged in a second kind of 
educational triage by focusing resources on the "accountables," to the 
virtual exclusion of students who "did not count." In accountability's 
ultimate contradiction, the protean word "accountable" retained only a 
semblance of its intended meaning -- taking responsibility for each and 
every student.

How many students are affected by the mobility provisions of NCLB? Take the 
Houston Independent School District as an illustrative example. Serving 
211,157 students, this district is the largest in Texas and the seventh 
largest in the nation. The average Houston school excludes 8% of its 
students from its "accountables."10 Almost one-third of Houston schools 
(31%) exclude more than 10% of their students from scores used for 
accountability. By any measure, this is not an insignificant number of 
students. Moreover, because mobility is not uniformly distributed across the 
population, some demographic groups have much higher numbers of mobile -- 
and thus unaccountable -- students. In Houston, an average of 16% of special 
education students and 11% of African American students are not counted in 
schools' scores because they have not been enrolled in a school for a full 
aca! demic year. Ironically, the very students NCLB was designed to target 
are often those least likely to be counted.

A second way that students may "not count" stems from states' definitions of 
the subgroup size required for disaggregation. If states define subgroup 
size expediently, the scores of various subgroups will continue to be buried 
in schoolwide averages. Again, Texas is a good example of artful definition 
of subgroup size. Under the Texas state accountability system, subgroups 
must include at least 30 students and account for at least 10% of all 
students -- or include 50 or more students -- to be evaluated. Under Texas' 
NCLB implementation plan, subgroups must include at least 50 students and 
make up at least 10% of all students -- or include 200 or more students -- 
to be evaluated. Under the state system, 82% of Houston schools with African 
American test-takers disaggregate scores for African American students, 
while for the purposes of NCLB, only 66% do.
Though Texas does not include a special education subgroup in its state 
system, the impact of using the 50 and 10% or greater than 200 definition 
rather than the lower threshold is significant. Shifting the definition 
upward reduces the percentage of Houston schools that disaggregate scores 
for special education from 55% to 24%. Other states have similarly gamed the 
subgroup-size provision of the law. In 2005, the U.S.

Department of Education allowed Florida to change its minimum subgroup size 
to 30 students who also make up 15% of test-takers. Because special 
education students rarely account for more than 15% of a school's 
population, very few schools in Florida will be required to disaggregate 
scores for these students.

There is an irreconcilable tension between accurately measuring school 
effects and forestalling the potential negative consequences of excluding 
some students from accountability calculations. If accuracy of measurement 
is privileged, some students will necessarily be excluded from 
accountability calculations. In order to best estimate school effects, a 
school should not be responsible for students who attend it for a short 
period of time. Similarly, small subgroups may yield statistically 
unreliable estimates of the school's efficacy with a particular group of 
students. Moreover, mainstream state tests may be inappropriate measures for 
some English-language learners or special education students. In other 
words, there are valid reasons, from a measurement perspective, for 
excluding students from schools' scores. On the other hand, the consequence 
of excluding these students may be to deny them access to scarce educational 
resources.
Better Choices?

So Mrs. Dewey can choose to teach all of her students, regardless of their 
potential contribution to her school's bottom line, or she can participate 
in educational triage. If she refuses to focus her time and attention on 
those students most likely to raise the school's scores, she risks not only 
the school's survival but her professional reputation as a good teacher and, 
potentially, her job.

Mrs. Dewey should not be asked to make such choices, and it is 
unconscionable to question her ethics when she does what she has little 
choice but to do. Systems of public policy cannot be designed solely for 
those with the moral certitude to qualify them for sainthood.

Educators will respond to systemic incentives, and NCLB's current incentives 
structurally induce behaviors that are inimical to broader notions of equity 
and fairness. In many cases, these perverse incentives turn educators' 
attention away from NCLB's intended beneficiaries. Until these issues are 
addressed, we can expect to see educational triage practices flourish across 
the country.


--------------------------------------------------------------------------------

1. My use of the phrase "educational triage," as well as the title of this 
article, draws on the work of David Gillborn and Deborah Youdell, Rationing 
Education: Policy, Practice, Reform, and Equity (Buckingham, U.K.: Open 
University Press, 2000).

2. Like Ted Sizer's Horace Smith, Mrs. Dewey is not one informant whom I 
encountered during an ethnographic study of an urban elementary school in 
Texas. Instead, she is a representative amalgam of the school's teachers. My 
study included 71 interviews -- 34 with teachers and administrators and 37 
with students -- in addition to 180 hours of participant-observation. Some 
of the findings discussed here were initially reported in Jennifer 
Booher-Jennings, "Below the Bubble: 'Educational Triage' and the Texas 
Accountability System," American Educational Research Journal, vol. 42, 
2005, pp. 231-68.

3. Julie B. Cullen and Randall Rebeck, "Tinkering Towards Accolades: School 
Gaming  Under a Performance Accountability System," Working Paper, 
University of California, San Diego, 2006; David N. Figlio and Lawrence S. 
Getzler, "Accountability, Ability, and Disability: Gaming the System," 
Working Paper 9307, National Bureau of Economic Research, 2002, 
www.nber.org/papers/w9307; and Brian A. Jacob, "Accountability, Incentives, 
and Behavior: The Impact of High-Stakes Testing in the Chicago Public 
Schools," Working Paper 8968, National Bureau of Economic Research, 2002, 
www.nber.org/papers/w8968.

4. Walt Haney, "The Myth of the Texas Miracle in Education," Education 
Policy Analysis Archives, 2000, epaa.asu.edu/epaa/v8n41; Linda M. McNeil, 
"Faking Equity: High-Stakes Testing and the Education of Latino Youth," in 
Angela Valenzuela, ed., Leaving Children Behind: How "Texas-Style" 
Accountability Fails Latino Youth (Albany, N.Y.: SUNY Press, 2005), pp. 
57-112.

5. Linda M. McNeil and Angela Valenzuela, "The Harmful Impact of TAAS 
Testing in Texas: Beneath the Accountability Rhetoric," in Gary Orfield and 
Mindy L. Kornhaber, Raising Standards or Raising Barriers? Inequality and 
High-Stakes Testing in Public Education (New York: Century Foundation, 
2001), pp. 127-50.
6. Linda M. McNeil, Contradictions of School Reform: The Educational Costs 
of Standardized Testing (London: Routledge, 2000).

7. Brian A. Jacob and Steven Levitt, "Rotten Apples: An Investigation of the 
Prevalence and Predictors of Teacher Cheating," Quarterly Journal of 
Economics, vol. 118, 2003, pp. 843-77.

8. Booher-Jennings, op. cit.; Gillborn and Youdell, op. cit.; "Making AYP: 
Cause to Celebrate?," Philadelphia Public School Notebook, Winter 2004, 
www.thenotebook.org/editions/2004/winter/editorial.htm; Joel Rubin, "Are 
Schools Cheating Poor Learners?," Los Angeles Times, 28 November 2004, p. 
B-1; Daniel White, Dara Wexler, and Juliette Heinz, "How Practitioners 
Interpret and Link Data to Instruction: Research Findings on New York City 
Schools' Implementation of the Grow Network," paper presented at the annual 
meeting of the American Educational Research Association, San Diego, 2004; 
and Katie Weitz White and James Rosenbaum, "Inside the Black Box: 
Sociological Mechanisms Affecting Professional Deviance, Student 
Classification, and School Culture," in Allan R. Sadovnik et al., ed! s., No 
Child Left Behind and the Reduction of the Achievement Gap: Sociological 
Perspectives on Federal Education Policy (New York: Routledge, forthcoming).

9. Michael W. Apple, Educating the "Right" Way: Markets, Standards, God, and 
Inequality (London: Routledge, 2001), p. 71.

10. Jennifer Booher-Jennings and Andrew A. Beveridge, "Who Counts for 
Accountability? High-Stakes Test Exemption in a Large Urban School 
District," in Sadovnik et al., op. cit. All analyses of Houston data 
mentioned in this article derive from this paper.


--------------------------------------------------------------------------------

JENNIFER BOOHER-JENNINGS is a doctoral candidate in the Department of 
Sociology at Columbia University, New York, N.Y. She would like to thank 
Andy Beveridge, Jason Booher-Jennings, Herb Gans, Toni Molnar, and Uri Shwed 
for their helpful comments and suggestions.




Priscilla Gutierrez
Outreach Specialist
New Mexico School for the Deaf

....change is inevitable, growth is optional...