Current accountability systems fail to take into account what the public really wants from schools.
Educational accountability has been relentlessly criticized for its imperfections over the past 15 years. These criticisms generally take aim at the narrowness of measurement systems, which rely chiefly on standardized test scores, as well as at the punitive sanctions associated with underperformance. Without a doubt, these are serious matters. But relatively little concern has been directed at a related problem: the failure of accountability systems to meaningfully engage the public.
The problem is not that public outreach is entirely absent. After all, states publish school-performance data and summative ratings of schools and districts for the explicit purpose of informing the public about how well their schools are doing. This, however, is an impoverished way of imagining public accountability. These systems ignore the actual interests and concerns of the public, focusing instead on a limited set of instrumental aims, such as raising student test scores. They sidestep the need for deliberation, instead employing formula-driven ranking and rating methods. And they do little to build capacity for planning and action, focusing instead on sanction and control. In sum, existing state accountability systems are public only in the most superficial sense.
Putting the public back into public accountability systems won’t be easy. But we believe that truly democratic systems, though perhaps less efficient, represent our best hope for successful and sustainable school improvement.
The present system: Three shortcomings
As policy rhetoric and existing measurement systems indicate, the state’s chief interest in educational accountability is instrumental in nature. That is, to the state, good schools matter insofar as they produce industrious workers and competent citizens. But for members of the public, schools also serve a number of noninstrumental purposes — purposes worth pursuing for their own sake. Advocacy for arts education, for instance, or for unstructured play is rooted in the belief that such things matter, even if they produce no tangible or predictable returns to the state. Similarly, parents tend to support smaller classes whether or not they have a measurable impact on learning. That’s because they value the nature of the learning experience as much as the outcome. Additionally, while the state is unitary in its instrumental aims, the public is plural. Parents care about school quality for different reasons than other adults in a school’s community do. And different school communities, embedded as they are in different local ecologies, vary in their priorities and concerns. Existing accountability systems fall short in advancing the noninstrumental and highly plural aims of the American public (for more, see Schneider, 2017).
A second shortcoming in present accountability systems is the fact that states can’t “see” (Scott, 1998). Unlike members of the public, who gauge school quality holistically, states rely on standard, highly simplified information that can be compiled and combined. This results from the inherent challenge in trying to see and judge from afar, but it is exacerbated by the state’s unitary and instrumental aims, which lead the state to focus on a narrow range of performance metrics. Thus, because measurement systems tend to include only the data in which states are interested — test scores, graduation rates, and the like — they overlook any additional aims that schools serve. Consequently, states are blind to the actual strengths and weaknesses of schools, at least as they are valued by stakeholders. If the state can’t see schools from multiple perspectives, how can it know their holistic quality?
A third shortcoming in present systems is their inflexibility, especially in prescribing consequences or solutions for lower-performing schools. This inflexibility, we argue, is directly related to the lack of local involvement in holding schools accountable. Working with limited information to advance a narrow set of aims, the state is ill-equipped to respond appropriately to the various factors shaping school performance. Additionally, state offices of education have limited staffing, which restricts their capacity for differentiated involvement. Further, states have a mandate to act impartially and consistently toward localities, which tends to manifest in generic, one-size-fits-all policies. Because of all this, state interventions can look glaringly mistaken or misguided to local constituents. Putative “fixes” for underperforming schools — such as firing teachers, converting schools to charters, or closing schools entirely — are the equivalent of prescribing a broad-spectrum antibiotic to address all manner of symptoms, whatever the differences in the underlying illnesses. Through the lenses the state uses, schools with records of underperformance are all functionally the same, and they are dealt with in the same limited set of ways.
Key features of a truly public system
For an accountability system to truly serve the public, it must first recognize the plural aims of education, many of which are noninstrumental. In other words, it must answer the questions that stakeholders actually have about their schools. Do students feel safe and cared for? Is school programming rich and diverse? Are young people developing valued character traits and civic dispositions?
To date, policy leaders have tried to address the shortcomings of existing accountability systems by adopting a variety of additional metrics, but these often fail to reflect noninstrumental or plural aims. Consider how states have updated their accountability frameworks under the Every Student Succeeds Act (ESSA). In response to the mandate to include at least one additional metric, most have opted for student attendance on the grounds that it is a significant predictor of outcomes like test scores and graduation rates. Instead of providing new kinds of information, then, they have offered new indicators for the same goals. Even laudable exceptions, like the CORE districts in California (https://coredistricts.org/our-data-research/improvement-measures), generally treat measures like social-emotional learning as side dishes to the main course of test scores and graduation rates.
If states seek to build truly public accountability systems, they must give stakeholders a voice in what a given school or district is accountable for — beyond instrumental metrics like academic proficiency and graduation rates. Presently, states often invite the public to participate in formal comment periods, but these usually only seek commentary about specific criteria employed in their otherwise inflexible systems. Accountability can and ought to be more democratic than that. Whether through polling, focus groups, or civic deliberation, the public must have greater power over the way that school quality is conceptualized and tracked.
In addition to enumerating further criteria by which schools will be judged, the public must also contribute to the actual evaluation of particular schools. Presently, most measurement and accountability systems rely on mechanistic rating systems, which generally rank schools against each other on a narrow set of measures. Done in the name of public accountability, such an approach falls short in two ways. First, it disregards the fact that communities may prize some aspects of school quality above others, meaning that plural public expectations will not conform to uniform state rating systems. Second, it ignores the fact that school quality is not a zero-sum game. All schools can succeed, even if some are stronger in particular areas than others, and the public must have a say in determining what constitutes success and failure.
Public involvement in conducting evaluations would draw upon local knowledge as a resource rather than a hindrance. Local stakeholders, who are attuned to their community’s needs, are best positioned to apply what they see as the most salient criteria of school quality. Such an approach would improve the substantive accuracy of school evaluations and ensure that the weaknesses and strengths of each school are fully accounted for on their own terms.
Lastly, the public needs to have a say in charting the course of a given school in light of its annual evaluation. Given the breadth of the public’s interests in school quality, the results of an evaluation might imply a vast number of possible improvements. Which of those should be pursued, and in what order, cannot and should not be settled by market mechanisms or automatic state sanctions. Far from simply consuming and making personal choices based on test data, and then simply watching as the state purges an underperforming school’s faculty or shuts the school down entirely, the public ought to play a more active role in holding schools accountable for the aims those schools are meant to serve.
What does this look like in practice?
The first step in building a truly public accountability system is to engage stakeholders in defining a broad set of educational aims — aims more plural, and less instrumental, than graduation rates or student standardized test scores. One model of such work can be seen in the Massachusetts Consortium for Innovative Education Assessment (MCIEA), where coauthor Jack Schneider serves as director of research. Through focus groups with hundreds of students, parents, teachers, principals, and community members, MCIEA created a School Quality Framework that consists of 16 constructs — from “school leadership” to “relationships” to “civic engagement.” The framework is then used to guide measurement that captures information aligned with what stakeholders value. Relying on member districts to provide administrative data, employing comprehensive surveys, and using performance assessment tools, the consortium provides stakeholders with rich information about student learning and school progress. Though MCIEA’s work is concentrated in seven Massachusetts districts, the research team is also supporting similar efforts elsewhere.
The second step in building a truly public accountability system is to engage stakeholders in conversations about what constitutes acceptable performance. Rather than ranking schools against each other and then automatically sanctioning the lowest quartile or quintile of performers, accountability systems should reflect the expectations and desires of the public. What is their yardstick for variables like “student sense of belonging” or “participation in creative and performing arts”? What are their standards when it comes to dimensions of school performance like “college-going and persistence” or “social and emotional health”? At MCIEA, stakeholders have played a significant role in shaping the performance benchmarks used to interpret each school’s data. The consortium plans to take such work a step further by bringing different stakeholder groups together to deliberate in person. Empowered with broad and inclusive performance data, school communities will, we hope, meet regularly to discuss school quality and identify their schools’ strengths and weaknesses.
Finally, in mapping out a future course for schools, algorithms should be tossed aside in favor of public convenings. After all, if accountability is to be truly public, it must be authentically determined by those with a stake in schools, not by a unitary and mechanistic statewide system. MCIEA is outlining a model for such convenings, which would bring together community members, state officials, data analysts, school administrators, teachers, parents, and students. Having already established their schools’ strengths and weaknesses, and possessing a shared understanding of the community context, such groups would discuss and outline the road ahead for their schools or districts. This kind of deliberative model would be time-consuming, certainly. And it would have to sidestep the bureaucratic nature of school board meetings, while avoiding the unrepresentative nature of voluntary groups. Additionally, it would require significant capacity building. Ultimately, however, it might also make school improvement efforts more likely to be accepted and to succeed.
In response to potential objections
Obvious objections to this sort of proposal may immediately arise. Won’t allowing significant local control in accountability undercut the state’s ability to meet its responsibility for ensuring equity among schools throughout the state? Isn’t judging each school differently the very definition of unfairness? Won’t localities be able to game the system and adjust criteria to make their schools look good?
The possibility of local control undercutting state oversight responsibilities is an important worry. But involving local stakeholders in making decisions does not exclude the state’s official concerns. The involvement of state officials in these deliberative bodies means that the state’s interest in equity and quality is represented and, indeed, enforced. The goal here is not to omit or minimize the instrumental interests that the state serves; it is to include the various needs and concerns of local stakeholders, even when these are noninstrumental.
As for the risks associated with treating schools differently, we have a substantially more robust version of fairness in mind than the sort expressed through present accountability systems. Current practices ensure fairness through a mechanical version of objectivity in which schools are evaluated through an algorithmic combination of a handful of measures, without significant public input or engagement. We believe that deliberations are fairer than algorithms and that a more substantive and less procedural version of fairness will be both more just and more democratic. Because a school’s real quality depends on the values of its community, a school must be evaluated holistically and multidimensionally. Putting local representatives in conversation with one another and with state officials maximizes the likelihood of doing justice to the quality of a particular school while still responding to the concerns of the state.
With respect to the possibility of gaming the system, the very objection is based on a mistake. Gaming the system is only desirable in an environment where accountability is limited to doling out rewards and punishments on the basis of performance against a set of measures that the community does not necessarily care about. In other words, gaming the system is only a threat under current accountability systems — ones that privilege accountability to the state. But schools belong to the public they serve, and local people are the ones most likely to take an interest in the quality of their schools, just as they are most likely to take an interest in the quality of their roads or the safety of their neighborhoods. Under a genuinely public accountability regime, there is no impersonal system to game. The state, instead, channels its efforts into ensuring that data are comprehensive and transparent and that the public has the support it needs in assessing school performance.
Finally, some may object to how unwieldy this whole process may become. Such a concern is answerable in two ways. First, current practices are far from simple, despite their widespread acceptance. Present accountability systems require massive expenditures of money and time that might be spent more profitably in other ways. Though the wheels of accountability now churn along seemingly without effort, we must remember that these systems have consumed tremendous resources over the past two decades. A second response to the objection of unwieldiness is to reiterate the local nature of these deliberative groups. In each case, we imagine cohorts of 10-20 people, some of whom — namely district and state officials — will be extremely practiced, serving on these groups multiple times per year. This process is only chaotic to the same extent that jury deliberations are chaotic. Such processes are enshrined in our public practices because they are internally related to the ideal of democratic governance. Deliberations would not be simple or easy, but the alternative is fundamentally unjust.
Our public schools operate within a pluralistic democratic society. In our national mythos, this has been a source of unique strength, even as it has presented challenges. But present school accountability systems seem to wish away this complexity — obscuring the challenges of pluralistic democracy behind a system that will smooth over the vagaries of difference among us. Our collective delusion is that we can build an accountability machine, set it in motion, and let the unimpeachable results roll in, free from the responsibilities of situational judgment. This is an ignoble wish. While present accountability systems might appear to fulfill a responsibility to the public, they actually represent a negation of democracy.
Can members of the public agree on a set of relevant educational aims? Can they take on the responsibility of assessing school quality? We believe so. Democratic processes are not fail-safe or foolproof, but mechanistic accountability systems are an inadequate alternative. If human judgment is imperfect or frail, the solution is not to eliminate the possibility of judgment, but rather to put diverse judgments together in serious conversation. Democratic justice has always required the judgments of one’s peers and of one’s community. And, though it may not come easy, the importance of educational accountability merits an equally high standard.
Schneider, J. (2017). Beyond test scores: A better way to measure school quality. Cambridge, MA: Harvard University Press.
Scott, J.C. (1998). Seeing like a state. New Haven, CT: Yale University Press.
Citation: Gottlieb, D. & Schneider, J. (2018). Putting the public back into public accountability. Phi Delta Kappan, 100 (3), 29-32.