Under No Child Left Behind Act (2001) , schools must show improvement in learning outcomes.

1. A “PAT” example of feedback challenges • Under No Child Left Behind Act (2001), schools must show improvement in learning outcomes. • Goal: to have all students proficient in reading and math by 2014. Math scores are gradually improving up, but reading scores remain flat. • Some criticisms: LESS SCIENCE AND HISTORY ARE TAUGHT – • Because state assessment tests focus on reading and math, other subjects get squeezed out. A study looked at how elementary-school teachers apportion their time each week. Weekly hours/instructional: Grades 1-6 - • Reading [Up] 40 min. '99-'04 • Math [Down] 17 min. '99-'04 • Science [Down] 23 min. '99-'04 • History [Down] 17 min. '99-'04

2. A “PAT” example (con.) • LOWER STATE STANDARDS: Federal law requires students be tested annually to determine reading and math skills – up to states to devise exam. Result, critics say, is that some states make their tests easier so it appears that their students are doing well. • The evidence: huge gaps between state results and scores on national standardized tests. Anecdotal evidence suggests when time-to-teach increases, so do scores. • By its own count, Mississippi is tied for the best score in the country. But on a U.S. test called the National Assessment of Educational progress, the state drops to 50th place—a whopping 71 points lower • Sources: National Center for Education Statistics; the Education Trust; Testing, Learning and Teaching by Martin West, Brown University

3. Conclusion – information and implementation • Governments often fail to acknowledge need to synthesize what experts and policy analysts know with local , indigenous, or “street-level” knowledge. • Over-reliance on experts can lead to efforts to intimidate or indoctrinate public (e.g., efforts to compel immigrants to abandon their folkways; efforts to get public to accept the ‘safety’ of nuclear power – Stone, 2001). • We need to translate expert knowledge into information useable by local populations through sensitivity to local cultural practices, norms, and traditions. • Experts must incorporate community perspectives into professional judgments –to gather all the information needed for making good decisions (Blood, 2003; Frank Fischer, 2001 (e.g., citizen participation and environmental risk, community expectations about schools, local public health and nutrition).

4. Evaluating policy – outcomes and evaluators • Evaluation is often termed the “final” stage of policy process. • Viewed as scientific and “detached” – but is usually neither: it is agenda-driven, valuational, and political. • Two issues: 1) the range of organizations doing evaluation, and the approaches they employ; and 2) the divergent ways problems are defined and evaluated – by policy makers and analysts. • Who are the “evaluators”? • Numerous organizations: some issue-based, others philosophy-based, have different sources of funding, clients, products. • Evaluation has become a “boutique” industry – you may choose your analysis – and evaluation based on your goals, policy orientation, analytic approach, major products, stage of process you want evaluated.

5. Evaluation as “final” stage of policy AGENDA SETTING POLICY MAKING PHASE POLICY IMPLEMENTATION PHASE Demands – FOR POLICY CHANGE, REFORM, RE-DIRECTION Outputs – laws, rules, regulations Supports – THROUGH CONTINUED FUNDING, REVENUE STREAMS, FAVORABLE OPINIONS. Policy formulation/ law-making by legislatures Policy execution – application/ enforcement by bureaucracy • Feedback – evaluation and policy impact assessment – ARE POLICIES MEETING GOALS & EXPECTATIONS? • Every stage of policy features interaction among competing interests. • Diversity of evaluators and their approaches/philosophies/expectations reflects these interests. • This explains why – at some level, ALL policies are seen as “failures;” no outcomes will ever be viewed as favorable by all interests. Failure is in the eyes of the beholder/evaluator. • Assume, moreover that IF a policy outcome is viewed as widely successful – supporters will want the policy extended to other groups or issues – thus, it must be shown that outcomes “fall short” in some way (e.g., national parklands, public works programs, cancer research).

6. An overview of policy evaluation • If evaluation is an interest-laden component of the policy process, then how it is conducted (and whether something is viewed as a failure or success) is subject to divergent perspectives: • Time bias and disposition: • Temporal data for evaluation – short vs. long; accuracy of projections and interpretations. • Changes in public attitudes – if a policy “regime” persists for a long-period of time, public attitudes toward policy may change – sometimes due to success! • Time paradox – if a policy is successful it will be continued beyond it’s original anticipated duration (e.g., block grants for urban re-development, energy price supports) - increasing likelihood of some failure!

7. Spatial and perceptual failures • Facility-siting issues: e.g., “NIMBY, NIABY, AND LULUs.” (energy issues, waste management problems, homes and facilities for special populations (e.g., mentally challenged, ex-convicts). • Jurisdictional extension-ism issues: federalizing a state policy; internationalizing a national program. • Social expectation issues: when a program and cultural values are “out of sync.” (e.g., civil rights, public housing, public transit). • Public perception or “unanticipated consequences” bind – e.g., land use, housing, transportation, many environmental decisions • All these “failures” are inter-connected and systemic – i.e., failures of both outcome and process or, political not just program failures!

8. Success, failure, “successful failure” – Project Apollo (1961 – 1972) Apollo 11 – first landing of humans on the moon (July 1969) Apollo 13 – oxygen tank explosion in mid-flight forced mission to be aborted, astronauts safely returned to earth (April 1970)

9. Manned lunar program and policy evaluation • Goal – to land an American on the moon and bring him/her back safely by “end of the decade” (April 1961, President Kennedy – succeeded in 1969). • Cost = $10 billion; divided into three sub-programs to devise different spacecraft, missions, and technical feats – Mercury, Gemini, Apollo. • Run by NASA – one of most popular Cold War agencies; aim was partly scientific, partly political (restore national prestige and land on another world). • Initially successful and popular (e.g., “the Right Stuff” – astronauts were folk heroes). Other benefits: • Huge economic multiplier effects: California, Florida, other sun-belt states; for every $1 of public expenditure, $5-7 in other investment/spending/jobs. • Huge technological spinoffs: telecommunications, personal computers and digital technology, materials science, aerospace, bio-metrics, education (space grant program).

10. Failure as success • Changes in public mood: Americans became more pessimistic , cynical by end of 1960s; Vietnam War, racial divide, persistent poverty and urban problems. New questions were posed in the framework of Congressional oversight : • Should we continue to spend money on this? • If we achieved the goal, then is it appropriate to continue to do this? • If the goal is “science;” can’t we meet it more economically with un-manned probes? • If we can “land a man on the moon” why can’t we fix our cities? • Program succeeded beyond expectations – while there was tragedy (e.g., Apollo 1 in 1967), goal was achieved and scientific/political gains were significant. This played out in context of program evaluation: • Success bred contempt – NASA made it look too easy – perhaps new goals are needed! • In 1970 – when near-tragedy averted (Apollo 13) reaction was: is the risk worth it? • Given the need for new goals, and desire to avoid risk – maybe we could better spend a space program budget on other things?

11. Feedback as a new policy agenda • By 1972, Congress forced NASA into “choice:” • Continue moon landings or have a space shuttle program and space station. • A re-useable spacecraft that could perform various missions & build “near earth” capacity (i.e., shuttle would help build space station). • Such a program would – serve many constituencies: scientific, educational, and even commercial (space policy as distributive policy). • This option would also permit other monies to be spent on un-manned planetary probes. • Changes in temporal, perceptual & cultural assessments led to policy transformation! – (John Logsdon – space policy expert, others).

12. MEDIA REACTION TO APOLLO PROGRAM TERMINATION – 1972: William Hines, syndicated Chicago Sun-Timescolumnist, who had opposed the project from the start: "And now, thank God, the whole crazy business is over.“ The Christian Science Monitor cautioned that “Such technological feats as going to the moon do not absolve people of responsibilities on earth.” Time magazine – “after the magnificent effort to develop the machines and the techniques to go to the moon, Americans lost the will and the vision to press on. Apollo's detractors … were prisoners of limited vision who cannot comprehend, or do not care, that Neil Armstrong's step in the lunar dust will be well remembered when most of today's burning issues have become mere footnotes to history.”

13. Pruitt-Igoe federal housing project • Modernist housing project designed in 1951; high-rise “designed for interaction;” seen as a multi-benefit solution to problems of urban renewal: • combine housing & services • park-like environmental amenities • facilitate sense of community • Completed 1956; thirty-three, eleven story buildings on a 35 acre site just north of downtown St. Louis, MO.

14. City officials wanted to build a spatially-dense “Manhattan-style” development as a model for middle- as well as low-income families; cheaper to build and maintain.

15. Pruitt-Igoe as multi-modal policy failure • Social expectations –public perceptions – it was anticipated as an integrated development. • Whites were unwilling to live in close proximity to blacks (de facto segregation). • “(Thus) the entire … project soon had only black residents." (Alexander von Hoffman, Joint Center for Housing Studies, Harvard University). • Once it was populated by impoverished residents, the project’s outcomes became out-of-synch with original decision-maker goals of becoming a high-density model of residential development. • Initially viewed as a social experiment, over time project became seen by the public as a LULU. It was poorly maintained and neglected, largely due to inadequate continued funding (perceptual/temporal change).

16. Multi-modal policy failure (con.) • Unanticipated consequences: from failure-to-failure – • "The problems were endless: Elevators stopped on only the fourth, seventh and 10th floors. Tenants complained of mice and roaches. Children were exposed to crime and drug use, despite the attempts of their parents to provide a positive environment. No one felt ownership of the green spaces that were designed as recreational areas, so no one took care of them. A mini-city of 10,000 people was stacked into an environment of despair.” – “A federally built and supported slum.” Lee Rainwater, 1970 • Problems effectively “cascaded” from one failure to another: neither blame nor responsibility could be ascribed clearly. • Once perceived as failure, policy-makers literally (as well as figuratively) abandoned it. • Goal-based failure – planners and policy-makers assumed that a rationally-constructed project based on objectivist models of human behavior (i.e., poverty as understood by social scientists) would provide a tenable solution. A classic “principal-agent” problem.

17. See - http://www.defensiblespace.com/book/illustrations.htm Razed: 1972- 1976

18. So …. Why do policies fail? • Bad timing, poor information – not drawing on adequate information about future conditions; especially in making investment decisions, predicting societal attitudes. • Poor administrative capacity for reversing flawed decisions – centralized bureaucracies are immune to countervailing pressures; accountability-resistant cultures – Ingram and Fraser, 2005). Seen in many public works-type decisions, as well as social welfare programs. • Vague rights and responsibilities of policy actors (i.e., who’s responsible for what aspects of a project? Who gets “accountable” if something goes wrong? (Ascher, 1999) • Bias and agenda-driven behavior – decisions driven by an agency’s or organization’s ideology or political objectives, rather than in response to a clearly defined problem. • Conflicts of interest and poor distinction between implementation and evaluation - leading to ‘boondoggles’ or failures in systems that are supposed to be fool-proof and resistant to accident (e.g., Gulf oil-”blow-out;” space shuttle accidents). • Failure to seek and acquire public acceptance of decisions – this includes input into alternative models of choice and incorporation of public values in program design.

19. Can we ensure that policies succeed? • No …. However, design them to be technically credible, scientifically defensible, and politically effective if they are adaptively designed. • Assume that public policies will fail to fully embrace temporal, spatial, perceptual, goal based failures. • incorporate an “error-provocative design” within the policy – i.e., assume that flaws are inevitable and politically-unavoidable given the interest group pressures in their design (Ackerman, 1980; Robison, 1994). • Prepare for these flaws by providing checks and balances in their implementation – at the first sign of a problem, even a “street-level bureaucrat” responsible for some policy “mode” can veto a decision; demand that some problem be inspected, audited, examined, evaluated, or revisited. • Permit input from a wide range of stakeholders & provide capacity to modify the policy or program when new information is acquired.