When evaluating an education initiative, look beyond the average to consider its effects on all students.
In the same way that different people can have different responses to the same medical treatment because of individual variability in genes, environment, and lifestyle, students can have different reactions to the same educational treatment because of their individual conditions. In other words, what works for one student can hurt another. Because of this, interventions in education, be they policy or practice, can have both positive, intended, and desirable effects and, at the same time, adverse and unwanted side effects (Zhao, 2018). In the December 2018 Kappan, Rick Hess and Michael McShane describe such side effects as “happy (and not so happy) accidents” and consider how they played out in the education reforms of the Bush-Obama era. I believe the idea can also apply to the debate about school vouchers.
In the 1950s, the free-market economist Milton Friedman, a Nobel winner in his field, proposed turning public education into a free market through school choice (Fowler, 2002; Friedman, 1955; Greene, 2011). The debate has become increasingly fierce over the years as states have implemented various forms of school choice, such as public charter schools, magnet schools, and cross-district choices, resulting in greater levels of privatization and marketization (Ravitch, 2013). The election of Donald Trump as U.S. president in 2016 and his appointment of Betsy DeVos as secretary of education have raised the stakes because, to many, their strong support for vouchers amounts to using public funds to support private, and even religious, schools (Willingham, 2017).
Friedman, sometimes called the grandfather of vouchers, and his wife Rose Friedman launched the Friedman Foundation for Educational Choice (now EdChoice, where, as it happens, McShane is the director of national research) in 1996 to “promote school choice as the most effective and equitable way to improve the quality of K-12 education in America” (Forster, 2016). Opponents, however, argue that, among other things, school choice negatively affects public schools and results in a host of negative consequences such as distortion of the purpose of education, decreased civic engagement and commitment to the common good, and greater social segregation (Abowitz & Stitzlein, 2018; Labaree, 2018; Ravitch, 2013).
Reviewing the evidence
Setting aside the philosophical objections to and defenses of choice, as important as they may be, let’s consider whether school choice even leads to better achievement for participating students. Since the 1990s, the two sides have used the same imperfect body of empirical evidence to argue for and against choice. The central point of disagreement is the size of the effect of choice on student achievement, with proponents finding more significant positive effects than independent researchers (Carnoy, 2001; Dynarski, 2016; Rouse & Barrow, 2009).
For example, the May 2016 EdChoice report A Win-Win Solution: The Empirical Evidence on School Choice (4th ed.) systematically reviews publicly available empirical studies that used the “gold standard” of research methods — random assignments of participants. The report’s author, Greg Forster (2016), concludes that “the empirical evidence shows that choice improves academic outcomes for participants and public schools, saves taxpayer money, moves students into more integrated classrooms, and strengthens the shared civic values and practices essential to American democracy” (p. 1). Even the most controversial form of school choice, vouchers to send children to private schools, is a “win-win” solution for everyone, he concludes. Similarly, voucher advocates at the University of Arkansas have found a positive effect of vouchers after “a meta-analytic and systematic review” of “the participant effects of private vouchers across the globe” (Shakeel et al., 2016, p. 1).
Other researchers found much less significant effect. For example, Cecilia Rouse and Lisa Barrow (2009) reviewed the empirical evidence and concluded that the achievement gains for students in voucher programs are relatively small, with most not statistically different from zero. The significant effects reported in both reports have also been challenged. Christopher Lubienski pointed out that the reported significant effects of voucher programs in the two reports resulted from ideological bias and flawed research: “Both reports are marred by a number of serious problems and errors, including misrepresentations of the research literature, a failure to acknowledge the limitations of their approaches, not addressing the shortcomings of the theoretical underpinnings of vouchers, and the use of methods that bias the selections of the studies they utilize” (Lubienski, 2016, p. 1).
In any case, this debate has focused to date only on the average effect of school choice on the students who participate in it. More helpful, however, would be a discussion of the varied effects of choice-based initiatives. In other words, even if such programs have a positive effect on average (which is under dispute), we still need to ask whether and to what extent they might also be harming certain students. And in fact, while there may be good reason to challenge Forster’s selection and analysis of previous research studies, it’s worth noting that even his report, though written by an advocate for school choice, offers clear evidence of such negative side effects.
Different students, different effects
Forster’s claim about the academic benefits of choice for participants is based on 18 studies using random-assignment methods. He concludes that “[T]his body of evidence shows that school choice benefits students” because:
Fourteen studies find positive effects on school choice participants: six find choice had a positive effect across all students participating and another eight find choice had a positive effect on some student groups and no visible impact on other students. Two studies find no visible effect from choice. Two studies on Louisiana’s voucher program find that it had a negative effect. (Forster, 2016, p. 10)
But Forster’s own data do not support his sweeping conclusion. Only six out of 18 studies found a positive effect across all students. Eight other studies Forster counts as having a positive effect actually had mixed results: positive for some and no visible effect on others. Further, two studies found an outright negative effect.
Likewise, the University of Arkansas report finds that the effect sizes of voucher programs varied a great deal across countries and locations. Overall results “indicate that school vouchers have positive effects in both reading and math, but that these impacts are largest in programs outside of the U.S.” (Shakeel et al., 2016, p. 33). Further, they note that studies found no effect on reading in the U.S. but large positive effects in other countries, with the largest in Bogota, Colombia. Judging from the evidence, then, the effects of school choice, like all education interventions, is dependent on individual characteristics and contexts.
If school choice expands its reach, we must carefully study its effects and side effects on students with a wide variety of characteristics.
The contexts in which a treatment is implemented can moderate its effect or even cause adverse side effects. To date, there has been little empirical research into the ways in which contextual factors interact with school choice programs, but there has been no lack of speculation. Forster, for example, suggests that the significant negative impact of the Louisiana voucher program found by two studies (Abdulkadiroglu et al., 2015; Mills & Wolf, 2016) is the result of “poor program design and fear of future action from hostile regulators” that led to the low participation of private schools (Forster, 2016, p. 12), with participating private schools being more likely to be lower performers. Mark Dynarski (2016) argues that the negative effect may be due to the decline of quality of private schools and the narrowing gap in quality between public and private schools in Louisiana. The University of Arkansas report (Shakeel et al., 2016) found that publicly funded vouchers delivered larger effect sizes than privately funded ones, but the difference is largely influenced by the outsized effect of one publicly funded program in Bogota, Colombia.
There are certainly other contextual variables — such as population density, transportation, culture, and the race, ethnicity, and religion of participating students — that can affect the degree to which school choice has a positive or negative effect on students. There is no reason, then, to conclude that school choice is a panacea. If it helps some students, it may hurt others. Parents and policy makers should neither embrace nor reject school choice without understanding its full range of effects and side effects.
Tankers and leapers
The effect of attending private schools supported by school choice varies a great deal among participating students and across cohorts. For instance, Martin Carnoy (2001) found that the standardized test performance of some students, termed “tankers,” fell more than 1.5 standard deviations when they switched to a private school, while the score of “leapers” rose by more than two standard deviations. Different sites saw inconsistent patterns of effects across cohorts, with the Washington, D.C., program being the only site that showed relatively consistent achievement gains across grades (Carnoy, 2001).
What led to the decline among “tankers” and the improvements among “leapers”? Why was there so much inconsistency across cohorts? There is evidence that individual variability played a role.
So which groups are most likely to experience negative side effects in a voucher program? Researchers from the University of California at Irvine found that a voucher experiment in New York City “did little to influence student achievement” but had “a small negative effect for a small group of high-performing students after the first two years of the program” (Domina & Penner, 2013, p. 24). A similar finding was also reported by another group of researchers (Barnard et al., 2003). The New York City experiment was also found to have a negative effect on non-Black students’ math achievement (Krueger & Zhu, 2004). Similarly, research into a voucher program in Charlotte, N.C., showed that Black students with a mother who dropped out of high school experienced negative effects, while the effect was positive for those whose mothers graduated from college and who lived in a two-parent home (Cowen, 2008).
There may be adverse effects on students with other characteristics, but these studies do not systematically report them, perhaps because the researchers did not collect the relevant individual data or because they did not look for these patterns with the data they had. Because many voucher programs were designed to serve low-income, minority students in urban areas, the student population studied is more homogeneous than American schools as a whole. Thus, if school choice expands its reach, we must carefully study its effects and side effects on students with a wide variety of characteristics.
Parent characteristics and choice
Even if school choice leads to positive effects for all students who participate, not all parents are willing or able to participate. Certain characteristics of parents have been found to affect their decision to accept vouchers and send their children to private schools, further complicating our understanding of the effects of vouchers.
In his analysis of the Milwaukee Parental Choice Program, John Witte (2001) noted that Black and Latinx parents are more likely than White parents to apply to voucher programs like Milwaukee’s, female parents are more likely to apply than males, lower-income families are more likely to apply than those of higher incomes, and mothers with higher levels of education are more likely to apply than less educated mothers. William Howell and Paul Peterson (2006) made both similar and different observations in The Education Gap: Vouchers and Urban Schools. In their studies, families with lower incomes, families of Black students, and mothers with less education were more likely to decline vouchers. And a recent study of the voucher program in Charlotte found that parents who attended some college and those living in two-parent homes were more likely to participate, while Black parents were less likely to participate (Cowen, 2008).
In sum, although the nature of the effect is unclear and further study is needed to uncover a clear pattern, race, gender, education level, and family structure all appear to influence student participation in choice initiatives. Thus, certain children are more likely than others to experience the effects of choice, whether positive or negative.
Side effects include . . .
As much as we might want to seek a perfect solution for all students, one student’s medicine may very well be another one’s poison. As students’ characteristics and education treatments interact, negative side effects may occur. Funding private schools with public dollars probably does not affect all students positively in a uniform fashion. To date, studies of school voucher programs have found their effects to vary among different populations of students.
Moreover, besides the side effects resulting from the interactions between students’ characteristics and education treatments, side effects also occur because of the broad range of desirable and potentially competing education outcomes. So far, evidence of the effects of voucher programs has been limited to a narrow set of outcomes such as academic achievement. Little, if any, empirical evidence has been collected concerning other equally important outcomes of schooling, such as preparing students for civic engagement and betterment of a shared society (Abowitz & Stitzlein, 2018; Labaree, 2018). Thus, we do not know their effects, negative or positive, on other important outcomes. It is, however, reasonable to believe that voucher programs and other forms of privatization of education can have negative side effects on individual students, the public school system, and the society (Labaree, 2018).
Abdulkadiroglu, A., Pathak, P.A., & Walters, C.R. (2015). School vouchers and student achievement: First-year evidence from the Louisiana Scholarship Program. Cambridge, MA: National Bureau of Economic Research.
Abowitz, K.K. & Stitzlein, S.M. (2018). Public schools, public goods, and public work. Phi Delta Kappan, 100 (3), 33-37.
Barnard, J., Frangakis, C.E., Hill, J.L., & Rubin, D.B. (2003). Principal stratification approach to broken randomized experiments: A case study of school choice vouchers in New York City. Journal of the American Statistical Association, 98 (462), 299-323.
Carnoy, M. (2001). School vouchers: Examining the evidence. Washington, DC: Economic Policy Institute.
Cowen, J.M. (2008). School choice as a latent variable: Estimating the “complier average causal effect” of vouchers in Charlotte. Policy Studies Journal, 36 (2), 301-315.
Domina, T. & Penner, E.K. (2013). Distributional effects of a school voucher program: Evidence from New York City. http://emilykpenner.com/epkwp/wp-content/uploads/2013/07/Bitler_Domina_Penner_Hoynes_2013_NYC_Voucher.pdf.
Dynarski, M. (2016). On negative effects of vouchers. Washington, DC: Brookings. www.brookings.edu/research/on-negative-effects-of-vouchers.
Forster, G. (2016). A win-win solution: The empirical evidence on school choice. Indianapolis, IN: Friedman Foundation for Educational Choice.
Fowler, F.C. (2002). Introduction: The great school choice debate. The Clearing House, 76 (1), 4-7.
Friedman, M. (1955). The role of government in education. New Brunswick, NJ: Rutgers University Press.
Greene, J.P. (2011). Why America needs school choice. New York, NY: Encounter Books.
Hess, F.M. & McShane, M.Q. (2018). The happy (and not so happy) accidents of Bush-Obama school reform. Phi Delta Kappan, 100 (4), 31-35.
Howell, W.G. & Peterson, P.E. (2006). The education gap: Vouchers and urban schools. Washington, DC: Brookings Institution Press.
Krueger, A.B. & Zhu, P. (2004). Another look at the New York City school voucher experiment. American Behavioral Scientist, 47 (5), 658-698.
Labaree, D.F. (2018). Public schools for private gain: The declining American commitment to serving the public good. Phi Delta Kappan, 100 (3), 8-13.
Lubienski, C. (2016). Review of A Win-Win Solution and The Participant Effects of Private School Vouchers across the Globe. Boulder, CO: National Education Policy Center.
Mills, J.N. & Wolf, P.J. (2016). The effects of the Louisiana Scholarship Program on student achievement after two years. New Orleans, LA: Education Research Alliance for New Orleans.
Ravitch, D. (2013). Reign of error: The hoax of the privatization movement and the danger to America’s public schools. New York, NY: Knopf.
Rouse, C.E. & Barrow, L. (2009). School vouchers and student achievement: Recent evidence and remaining questions. Annual Review of Economics, 1 (1), 17-42.
Shakeel, M.D., Anderson, K.P., & Wolf, P.J. (2016). The participant effects of private school vouchers across the globe: A meta-analytic and systematic review. Little Rock, AR: University of Arkansas Department of Educational Research.
Willingham, A.J. (2017, May 24). How to make sense of the school choice debate. CNN. www.cnn.com/2017/05/24/us/school-choice-debate-betsy-devos/index.html.
Witte, J.F. (2001). The market approach to education: An analysis of America’s first voucher program. Princeton, NJ: Princeton University Press.
Zhao, Y. (2018). What works may hurt: Side effects in education. New York, NY: Teachers College Press.
NOTE: This article was adapted from What Works May Hurt: Side Effects in Education by Yong Zhao (Teachers College Press, 2018).
Citation: Zhao, Y. (2019). Side effects in education: Winners and losers in school voucher programs. Phi Delta Kappan, 100 (5), 63-66.
On Feb. 6, 2019, Patrick Wolf posted a reply to Yong Zhao on EducationNext:
Here’s Zhao’s response:
In response to my recent Kappan article Side Effects in Education: Winners and Losers in School Voucher Programs, Dr. Patrick J. Wolf, professor and 21st Century Chair in School Choice in the Department of Education Reform at the University of Arkansas College of Education and Health Professions, offers me three pieces of “friendly advice:”
- First, don’t call people names.
- Second, get your facts right.
- Third, know the subject of which you speak.
The advice is sound and should be heeded by everyone who wishes to engage in productive discussions of any topic. Unfortunately, though, Wolf does not follow his own advice and misses the point of my article. Permit me to pass his advice back to him.
First, don’t call people names.
According to Wolf, I am a name-caller because I label “school choice researchers as either ‘proponents’ (a.k.a., ‘advocates’) or ‘independent researchers.’” However, I don’t understand why identifying researchers in these ways amounts to name-calling. An advocate (or proponent) is commonly understood to be someone who supports, defends, or argues in favor of a cause. It is not a derogative, insulting, or offensive word, unless of course the cause itself is morally questionable, objectionable, or repugnant. Since it does not appear that Wolf believes school choice to be a repugnant cause, describing researchers as advocates and proponents hardly equates to insulting them.
(As an aside, the American Federation of Children, which explicitly describes itself as an “organization that does lobbying and grassroots advocacy work” in support of school choice, has referred to Wolf as an “advocate” for its cause. I have to wonder whether he has accused that organization of “name-calling.”)
Further, because I refer to some researchers as advocates and others as independent, Wolf accuses me of attempting to judge researchers’ intentions: “Zhao would need to possess the capacity to look inside of the hearts of his fellow human beings and therefore observe their intentions.” Actually, though, my intention, which Wolf appears to misread, is simply to characterize a pattern I have observed in the findings presented by researchers. It might be nice to have the power to peek inside people’s hearts, but for my purposes, it’s sufficient to look at their publications.
While we’re on the topic of name-calling, let me note also that Wolf chooses to label me as an ideologically biased “doubter”:
“So long as doubters like Professor Zhao continue to ignore the wealth of published evidence to the contrary, empirical research on school choice and civic values will continue. Many commentators, however, insist on trusting their ideological preferences on the matter of school choice and civic values instead of their lying eyes.”
Further, not only does Wolf presume to have seen what lies within my heart (ideological bias, he thinks) but also to have surveyed the whole of my academic expertise. “School choice,” he writes, “is new academic territory for Zhao, a Distinguished Professor who specializes in education technology and virtual learning.” Apparently, Wolf believes that if researchers are new to a topic, then their findings can be dismissed out of hand. In contrast, he touts his long history with the subject of school choice as evidence of his superior insight. He doesn’t just offer me “friendly advice” but he does so “as someone who has studied the topic for over 20 years.”
I find it odd for Wolf to suggest that I must be wrong because I’m newer to this topic than he is. As Charlotte Bronte once remarked: “I do not think, sir, you have any right to command me, merely because you are older than I, or because you have seen more of the world than I have; your claim to superiority depends on the use you have made of your time and experience.”
Moreover, Wolf is simply wrong about my background. My specialization in educational technology hardly precludes me from studying additional subjects, and I have in fact studied and written extensively about a broad range of issues in education, which Wolf could easily have learned by reading some of my recent writings or browsing my website.
Second, get your facts right.
Again, very sound advice that we all should follow. But here, too, Professor Wolf misses the point. The primary purpose of my Kappan article — adapted from a chapter in my book What Works May Hurt: Side Effects in Education (Teachers College Press, 2019) — is to call attention to the complex nature of educational interventions such as voucher programs, which can have both positive and negative effects. My point is that scholarly and policy-focused debates about these programs have focused mainly on their average effect on students. It is also important, I argue, to consider those programs’ differing effects on various groups of students, with attention to a broad range of outcomes.
Wolf, however, misreads my argument. He accuses me of criticizing “the body of research on the achievement effects of school choice” on the grounds that it focuses exclusively on average effects. Then, as if to prove me wrong, he points to studies that have examined variations of outcomes across different subgroups of students.
But my article does indeed point to research findings, including Wolf’s, that show negative effects of school vouchers on some students. My point isn’t that the body of research has excluded such findings but, rather, that these negative effects have been under-reported or ignored in publications that favor voucher programs, on the grounds that, as Wolf says, “A program has a single, general effect on participants unless its effects on different subgroups are, themselves, significantly different from each other based on statistical tests.”
What Wolf misses here is that individual children are not statistical probabilities and no matter how small the number of children who experience negative effects, those effects should be given serious attention. By way of comparison, a medical procedure may have benefits for the majority of people, but if it causes harm to even a very small number of patients, those adverse side effects should be discussed, not ignored.
I suspect that Wolf has a problem with my article not because I get the facts wrong but because I highlight facts that are right but which he may not want to discuss. Specifically, I have called attention to facts that challenge a simplistic and overly friendly interpretation of the evidence supporting voucher programs.
Third, know the subject of which you speak.
I couldn’t agree more. But again, I would ask Wolf to give that advice to himself, too, particularly when it comes to the broader study of K-12 education.
As he sees it, I don’t have a solid grasp of the research on school choice: “Zhao concludes his essay with the claim that, ‘Little, if any, empirical evidence has been collected concerning other equally important outcomes of schooling, such as preparing students for civic engagement and betterment of a shared society.’” But, he adds, “Actually, there is a deep and broad research literature on the mostly positive effects of school choice in general and private schooling in particular on civic values such as political tolerance, volunteering in one’s community, political knowledge, political engagement, social capital, and patriotism.”
However, rather than proving that I don’t know the subject at hand, Wolf’s accusation only exposes his own limited understanding of the value of public education for a democratic society. In fact, those benefits go well beyond “political tolerance, volunteering in one’s community, political knowledge, political engagement, social capital, and patriotism.” (For more, see the two articles that I reference in my piece: Abowitz & Stitzlein, 2018, and Labaree, 2018). For example — and this is hardly an exhaustive list — important student outcomes include creativity, curiosity, psychological well-being, physical health, and the ability to communicate effectively across varied contexts. Even if we accept Wolf’s statement that there is “a deep and broad research literature” on the effects of school choice on civic values, it remains true that choice’s influence on many other important educational outcomes have yet to be assessed carefully.
Moreover, while my article focuses mainly on the relative lack of debate about the negative effects of school voucher programs, Wolf points to research about “school choice in general and private schooling in particular” as evidence of my ignorance of the field.
Again, it strikes me that Wolf has largely missed the point of my Kappan article. Educational interventions, like medical treatments, often have adverse side effects that accompany the intended benefits. Because education has many competing outcomes, interventions that have a positive impact on one outcome may come at the cost of other potential outcomes. And because students have differing characteristics, an intervention that helps some students may hurt others. This applies to all educational interventions, and voucher programs are no exception.
Abowitz, K.K., & Stitzlein, S.M. (2018). Public schools, public goods, and public work. Phi Delta Kappan, 100 (3), 33-37.
Labaree, D.F. (2018). Public schools for private gain: The declining American commitment to serving the public good. Phi Delta Kappan, 100 (3), 8-13.