Panel 3 Response and Investigation of Alleged and/or Actual Irregularities in Academic Testing

Panel 3Response and Investigation of Alleged and/or Actual Irregularities in Academic Testing Panelists Bob Wilson: Wilson, Morton & Downs, LLC Tisha S. Edwards: Baltimore City Public Schools Steve Ferrara: Pearson Lou Fabrizio: North Carolina Department of Public Instruction

Robert (Bob) E. WilsonAttorneyWilson, Morton & Downs, LLC One of the Special Investigators Appointed by the Governor of Georgia to Investigate the Atlanta Public School System

Atlanta Investigation Investigative Team – over 60 state agents, attorneys and paralegals 10 month investigation 2200 interviews 56 schools investigated Cheating found in 44 schools In 30 schools, 82 principals and teachers confessed Over 180 educators involved 38 principals involved For System: time consuming/off mission

Atlanta Created Targets Beyond NCLB + Imposed Pressure to Achieve (Both Positive and Negative) = DISASTER

You have three years to meet targets or I will find someone who will. There will be no exceptions and no excuses.

Warnings Significant increases in scores Significant decreases in scores Unusual answer patterns High wrong-to-right erasures Report of suspicious conduct Report of cheating

Report To School Leadership System Leadership System Legal Counsel State Education Authorities State Licensing Authority Resource Officer/Law Enforcement

Fundamentals Ignore nothing – all concerns are important Vigorously inquire Interview everyone who might know anything: - Administrators - Teachers - Staff - Parents - Students Consult experts Re-Interviews Polygraphs

The test should be about children, not adults. Image is not more important than children. Nothing is more important than creating a culture of integrity with an honor system to report suspicious activity.

Testing Integrity Symposium Baltimore City Public Schools Andrés A. Alonso, Ed.D.CEO, Baltimore City Public Schools February 28, 2012 Baltimore City Public Schools

Objectives Share a lesson learned from Baltimore City Public Schools’ Test Monitoring and Investigation Process Provide Baltimore City Schools’ best practices on its Test Monitoring and Investigation Process Share recommendations for restoring credibility Baltimore City Public Schools

Best Practice Story Issue: Result: Lessons Learned A Baltimore City School was alleged to have altered its scores from the Maryland State Test in order to qualify for Blue Ribbon Status. There had been previous allegations concerning this school that were found to be unsubstantiated. Duration of case: 18 months City Schools experienced challenges establishing culpability as a result of: people not talking; there was over-sensitivity in the reporting of allegations; there was a back tracking of allegations once staff and parents were interviewed. By conducting multiple analyses (communication analyses, erasure analyses, etc) City Schools established a very clear case that there was likely systematic cheating occurring at the school. Analysis is never enough to establish culpability. Unless comprehensive investigations occur, the credibility of districts and schools suffer. While the organization under investigation will take an initial hit by the investigation, the price is worth it Strengthening all aspects of our internal controls. Conducting multiple types of analyses in order to establish culpability with certainty. Continuing to adhere to the Code of Maryland Statutes for conducting testing investigations. Baltimore City Public Schools

City Schools’ Test Monitoring and Investigation Processes PREVENTING POTENTIAL VIOLATIONS DETECTING POTENTIAL VIOLATIONS INVESTIGATING POTENTIAL VIOLATION PREPARE INVESTIGATIVE REPORT MAINTAINING CREDIBILITY Baltimore City Public Schools

PREVENTING POTENTIAL VIOLATIONS • Establish and implement internal controls that will signal potential problems • Complete statistical forensic analyses that will pick up unusual patterns or trends in the data. City Schools analyses include: • Growth Analysisto examine annual results to identify significant gains/decreases • Benchmark Analysisto examine student performance on Quarterly Benchmarks in comparison to their performance on State Assessments • Entrant and LeaverAnalysisto examine how students perform as they transfer in or out of schools • CohortAnalysisto examine the results of a particular group of students over a period of time • Develop a detailed monitoring plan and monitor all high stakes assessments annually • In SY 2009-10 City Schools dispatched 120 central office administrators to monitor all aspects (distribution and collection of materials, classroom observations, security protocols, secure location etc.) of the MSA test administration to 13 schools • In SY 2010-11 City Schools hired over 200 temporary employees to monitor all aspects of the MSA test administration to 157 schools • Independent monitors assigned to each school increases the integrity of the results and the public’s trust of the results • Enhance the education and awareness for all stakeholders • Provide a mandatory training for all persons participating in the administration • Disseminate video presentation from the CEO stressing the importance of implementing the assessments with fidelity and consequences for violators • All personnel participations in the administration sign a Non-Disclosure form • Communicate monitoring plan to all stakeholders in collaboration with the state Baltimore City Public Schools

DETECTING POTENTIAL VIOLATIONS • Establish well defined protocols to detect potential concerns • Conduct forensic data analysis after each administration to identify potential concerns • Flag potential concerns with support from stakeholders • Work in collaboration with the state and other stakeholders to flag potential concerns. • For example, the state provides City Schools with Scoring Alerts which identify potential concerns for City Schools to review. City Schools also collaborates with the state to complete Erasure Analysis where necessary. Baltimore City Public Schools

INVESTIGATING POTENTIAL VIOLATIONS • Establish lines of communication between City Schools and the State • Primary responsibility for conducting test security investigations belongs to City Schools • City Schools works in collaboration with the state to complete our investigations • Coordinator of Security Investigations in City Schools assigned to directly collaborate with the state’s Test Security Officer on all testing investigations. • Establish well defined protocols for conducting investigations in accordance with state statutes • Adhere to the state statures for conducting security investigations • Increase investigative protocols by hiring an external law firm to review and codify protocols • Implement a process to ensure that ALL allegations are investigated • Plan for the investigation • Have a plan in place if a test security violation or test administration procedural deviation occurs • Collect strong qualitative evidence • A barrier is establishing open communication and getting individuals to talk openly about the allegations. • City Schools has protocols in place to ensure confidentiality of all individuals interviewed as a part of the investigation. • All staff are interviewed to protect the source of the information. • All person interviewed are provided with a summation of the district’s anti-retaliation policy. Baltimore City Public Schools

PREPARE INVESTIGATIVE REPORT • Create a strong investigative report • Conclusions are based on solid data utilizing both quantitative and qualitative • Where possible, utilize methods to triangulate the data • Evidence supports findings • Present findings to proper internal authorities • Plan for litigation • Throughout the investigation, the Coordinator of Security Investigations works with: • City Schools’ Legal Office on potential findings and to prepare for case hearings • City Schools’ Labor Relations in preparation for personnel discipline hearings Baltimore City Public Schools

MAINTAINING CREDIBILITY How do you maintain credibility in the face of accusations? • Administrators should address the allegations publicly. • Administrators should also buffer schools during the investigation, making sure that they do not suffer a decline in credibility. • Effective communication is very important as it is difficult to establish causation in terms of cheating, especially if schools are not open to discussing the allegation. • City Schools’ CEO actively takes the lead on communicating potential issues with the board, media, politicians etc.) • In the best practice story shared, City Schools’ CEO met with the staff, parents, and the newly assigned principal to share his expectations and how the school was going to move forward. • The Coordinator of Security Investigations works in conjunction with City Schools’ Office of Parent and Community Engagement to meet with community organizations and media to share the Testing Integrity Plan. Baltimore City Public Schools

RESTORING CREDIBILITY How do you restore credibility of a school system tarnished by alleged or actual wrong doing? • Fortunately for City Schools, our reputation was not tarnished as a result of testing investigations. • A reason for this was CEO’s proactive position to openly and honestly communicate the districts’ efforts to eliminate these issues. • For example, the CEO held a press conference with the State Superintendent to publically announce that based on City Schools’ investigations and internal controls, he was referring several schools to the state for violating testing protocols. • Recommendations to School Districts: • Maintain open and honest communication acknowledging the problem and how it is being addressed. • Firmly communicate that these types of actions will not be accepted as they undermined the great work being done in the district. • Provide a general overview of what the district is doing to prevent potential violations from reoccurring. Baltimore City Public Schools

Investigation and Response: Experiences and Reflections of a Former State Assessment Director Steve Ferrara Pearson Center for Performance Assessment NCES Testing Integrity Symposium Panel III: Response and Investigation of Alleged and/or Actual Irregularities in Academic Testing February 28, 2012

Overview • Background regarding investigations • Testing integrity is broader than test security • Mutually supportive test security system and culture of professional ethics • How the system and culture can support investigations • Investigation vignettes • Investigation considerations

Testing integrity is broader than test security • Conceptual distinction • Test security • Security of test content and materials • Appropriateness of test administrations • Testing integrity • Requires test security • Requires professional ethical behavior • Yields trustworthy data • Avoids inappropriate messages to children 22

Mutually supporting test security system and culture of professional ethics • Security system • Rules, guidelines, requirements, and procedures—including training—to protect security, avoid breeches, and enforce security • Culture1 of professional, ethical thinking and behavior • You don’t have to persuade people to buy into the culture; you do have to get them to abide by it. 1 The set of shared attitudes, values, goals, and practices that characterizes an institution, organization, or group (from http://en.wikipedia.org/wiki/Culture)

How the system and culture can support investigations • Three groups of people () • Culture of professional ethics • Encourages (supports, exerts pressure) professionally ethical thinking and behavior • Encourages reporting of irregularities and cooperation with investigations when irregularities may have occurred

Three (hypothetical) groups of people

Brief vignettes • What often happens after an investigation of a school? • Behavior typically changes, typically in a positive direction • Concern: misbehavior becomes less brazen, goes further underground • State Teacher of the Year in a state’s Blue Ribbon School • Teacher dictating responses to a testing group • During task pilot testing • In an operational administration

Considerations for investigations • Typically, test development and delivery contractors are not involved in investigation, are involved in protecting security, may be involved in detection • Investigation principles and procedures • Must be fair • Must withstand potential challenges in administrative law courts • Manuals, training programs, and experts in virtually every field • How do you get that expertise into schools, LEAs, and states? • Cheating risks with new devices • Currently, we can lock down online testing; tablets have task swapping and screen shot capabilities • Investigation challenges if personally owned devices have been allowed • It is necessary to investigate, enforce, and sanction rigorously and consistently

Thanks! Steve Ferrara Pearson Center for Performance Assessment steve.ferrara@pearson.com 612-581-6453

Responses to Alleged Testing Irregularities Lou Fabrizio, Ph.D. Director of Data, Research and Federal Policy NC Department of Public Instruction

State Board of Education(Rules & Regulations/Administrative Code) • Testing Code of Ethics • (16 NCAC 06D .0306) • License Suspension and Revocation • (16 NCAC 06C .0312) • Standards of Professional Conduct • (16 NCAC 06C .0602)

Record & Monitor Irregularities NC uses an online system Suspected and actual irregularities must be reported Local investigation required Assistance, as to process, is provided by Regional Accountability Coordinators and others from the SEA Serious irregularities trigger required check-lists

Steps in Serious Allegations/Actual Irregularities Once LEA completes a local report, it is reviewed by Accountability Services staff in the SEA Reports of serious irregularities are then forwarded to the State Board of Education (SBE) Attorney In collaboration with the State Attorney General’s (AG) office, there can be a further investigation by the SBE and/or AG-assigned attorney

Barriers Limited resources – There are no staff with sole assignment for investigations and no specific budget line item for investigations Follow-up investigations can take longer than desired because of the above Some investigations come before the State Superintendent’s Ethics Advisory Committee for Teacher Licensure Some investigations result in civil action and/or loss of license

Other Thoughts Serious allegations or actual instances of irregularities should be communicated to appropriate leadership in the SEA, including the Communications & Information Division Prevention is the best solution Need to advocate for a culture of honesty

Panel 3 Response and Investigation of Alleged and/or Actual Irregularities in Academic Testing