Investing in Innovation (i3) Reviewer Orientation

1. Investing in Innovation (i3) Reviewer Orientation July 2010 Development Grant Competition Tier 2 Note: These slides are intended as guidance only. Pleaserefer to the official documents published in the Federal Register.

2. Schedule for Today

3. What is the basic i3 review process? • The Department will use independent peer reviewers from various backgrounds and professions who have been thoroughly screened for conflicts of interest • Applications will be assigned to panels by one of the four absolute priorities where possible • Evidence and evaluation experts will score the selection criteria (B and D) focused on evidence and evaluation • Peer reviewers will determine whether any competitive preference priority points should be added For all three grant types… • Development applications will be reviewed in a two tier process • In Tier 1, all eligible applicants will be reviewed and scored against Selection Criteria A, C, E, F and G by three peer reviewers. Competitive Preference Points will also be added as appropriate by these Tier 1 peer reviewers. • Only those highest rated in Tier 1 will advance to Tier 2, where Selection Criteria B and D will be scored by two peer reviewers who are evidence and evaluation experts. Development Only…

4. What is expected of a Peer Reviewer? • Review the entire i3 application package (even though you are scoring only 2 selection criteria) and FAQs • Review and become thoroughly familiar with the selection criteria and factors • Familiarize yourself with the e-Reader system – used for inputting scores and comments • Participate in all scheduled conference calls • Provide appropriate comments that justify the score awarded and are helpful to the applicant • Revise comments as suggested by your panel monitor • Return all forms as required to ensure payment and completion of review process • Be available the entire review process • Draw upon your expertise • Maintain confidentiality throughout the review process Preparation Process Behavior

5. What must a Peer Reviewer really do? • Receive applications and panel assignments from Synergy • Identify any Conflicts of Interest, and notify Synergy or panel monitor immediately • Participate in Orientation webinar • Register in and become acquainted with e-Reader • Read applications (It may be helpful to you to take notes!) • Write draft comments, assign scores, and submit drafts in e-Reader before panel discussion (We recommend cutting and pasting from Word) • Receive feedback on comments from Panel Monitor • Revise draft comments based on feedback as appropriate • When comments/scores are acceptable to Panel Monitor, submit final comments/scores • Sign and return Technical Review Signature Forms to Synergy • Remember to submit All Necessary Forms to Synergy Prior to Panel Discussion After Panel Discussion

6. Schedule for Today

7. i3 Is One Part of UnprecedentedDirect Federal Investment in Education Race To The Top and Other Grants($9.7B in FY2009 Funding) FormulaGrants $26B • $4.35B - Race to the Top Fund, including $350MM for development of assessments • $3.5B* - School Improvement Grants • $650MM - Investing in Innovation Fund • $650MM -Education Technology • $300MM* - Teacher Incentive Fund • $250MM - Statewide LongitudinalData Systems SFSF $48.6B ARRA K-12 InvestmentAligned with Four Assurances * Includes regular FY 09 appropriations

8. Investing in Innovation (i3) Fund Summary • To provide competitive grants to applicants with a record of improving student achievement and attainment in order to expand the implementation of, and investment in, innovative practices that are demonstrated to have an impact on: • Improving student achievement or student growth, closing achievement gaps, decreasing dropout rates, increasing high school graduation rates, or increasing college enrollment and completion rates Purpose Funding $650 million to be obligated by September 30, 2010 • Eligible applicants are: • Local educational agencies (LEAs) • Nonprofit organizations in partnership with (a) one or more LEAs or (b) a consortium of schools Applicants

9. Types of Awards Available Under i3 i3 Development Validation Scale-up

10. Applicants Eligible Applicants can be: A local educational agency (LEA) A nonprofit organization in partnership with (a) one or more LEAs or (b) a consortium of schools

11. Key Definition: Partners Why It Is Important Official partnermeans any of the entities required to be part of a partnership under section 14007(a)(1)(B) of the ARRA (i.e., a non-profit organization, an LEA, or a consortium of schools). In the case of an eligible applicant that is a partnership between a nonprofit organization and (1) one or more LEAs or (2) a consortium of schools, the partner that was the applicant, and became the grantee when the partnership was selected to receive an award, may make subgrants to one or more of the official partners Other partnermeans any entity, other than the applicant and any official partner, that may be involved in a proposed project.

12. All Eligible Applicants Must Implement Practices, Strategies, or Programs for High-Need Students High-need studentmeans a student at risk of educational failure, or otherwise in need of special assistance and support, such as students who are living in poverty, who attend high-minority schools, who are far below grade level, who are over-age and under-credited, who have left school before receiving a regular high school diploma, who are at risk of not graduating with a regular high school diploma on time, who are homeless, who are in foster care, who have been incarcerated, who have disabilities, or who are limited English proficient. MUST MUST

13. i3 Absolute Priorities Teacher and Principal Effectiveness Early Learning (0 or 1 point) Improved Use of Data Systems College Access and Success (0 or 1 point) Improve Achievementfor High-Need Students College- and Career-ready Standards and High Quality Assessments Serving Students with Disabilities and Limited English Proficient Students (0 or 1 point) Improving Achievement in Persistently Low-performing Schools Serving Students in Rural LEAs (0, 1, or 2 points) Required forall applications Must select one(Absolute Priority) May select one or more(Competitive Preference)

14. Notes on Absolute Priority 1Innovations that Support Effective Teachers and Principals • “…increase the number or percentages of teachers or principals who are highly effective teachers or principals or reducethe number or percentages of teachers or principals who are ineffective, especially for teachers of high-need students…” • “…by identifying, recruiting, developing, placing, rewarding, and retaining highly effective teachers or principals (or removing ineffective teachers or principals).” • “…teacher or principal effectiveness should be determined through an evaluation system that is rigorous, transparent, and fair; performance should be differentiated using multiple rating categories of effectiveness; multiple measures of effectiveness should be taken into account, with data on student growth as a significant factor, and the measures should be designed and developed with teacher and principal involvement.” Two Possible Routes Multiple Methods Multiple Measures of Effectiveness 14

15. Notes on Absolute Priority 2 Innovations that Improve the Use of Data Two Possible Areas of Focus • “…(a) encourage and facilitate the evaluation, analysis, and useof student achievement or student growth data by educators, families, and other stakeholders in order to inform decision-making and improve student achievement, student growth, or teacher, principal, school, or LEA performance and productivity; or(b) enable data aggregation, analysis, and research” • “…data must be disaggregated using the student subgroups described in section 1111(b)(3)(C)(xiii) of the ESEA…” Data Disaggregation 15

16. Notes on Absolute Priority 3 Innovations that Complement the Implementation of High Standards and High-Quality Assessments • “…standards and assessments that measure students’ progress toward college and career-readiness…” • “…may include, but are not limited to, … • increase the success of underrepresented student populations in academically rigorous courses and programs…; • increase the development and use of formative assessments or interim assessments, or other performance-based tools and “metrics” that are aligned with high student content and academic achievement standards; or • translate the standards and information from assessments into classroom practices that meet the needs of all students, including high-need students. • “…eligible applicant must propose a project that is based on standards that are at least as rigorous as its State’s standards…” Focus on College & Career Readiness Range of Allowable Projects RigorousStandards 16

17. Notes on Absolute Priority 4 Innovations that Turn Around Persistently Low-Performing Schools • “Whole-school reform, including, but not limited to, comprehensive interventions to assist, augment, or replace Investing in Innovation Fund Absolute Priority 4 schools, including the school turnaround, restart, closure, and transformation models of intervention … or …” • “Targeted approaches to reform, including, but not limited to: • Providing more time for students to learn core academic content by expanding or augmenting the school day, school week, or school year, or by increasing instructional time for core academic subjects • integrating ‘‘student supports’’ into the school model to address non-academic barriers to student achievement • creating multiple pathways for students to earn regular high school diplomas” Projects May Choose Either Approach 17

18. i3 Priority 4 Schools • Under Absolute Priority 4, the Department provides funding to support strategies, practices, or programs that are designed to turn around schools that are in any of the following categories: • persistently lowest-achieving schools (as defined in the final requirements for the School Improvement Grants program, see http://www2.ed.gov/programs/sif/faq.html); • Title I schools that are in corrective action or restructuring under section 1116 of the Elementary and Secondary Education Act (ESEA); or • secondary schools (both middle and high schools) eligible for but not receiving Title I funds that, if receiving Title I funds, would be in corrective action or restructuring under section 1116 of the ESEA. 18

19. i3 Competitive Preference Priorities Teacher and Principal Effectiveness Early Learning (0 or 1 point) Improved Use of Data Systems College Access and Success (0 or 1 point) Improve Achievementfor High-Need Students College- and Career-ready Standards and High Quality Assessments Serving Students with Disabilities and Limited English Proficient Students (0 or 1 point) Improving Achievement in Persistently Low-performing Schools Serving Students in Rural LEAs (0, 1, or 2 points) Required forall applications Must select one(Absolute Priority) May select one or more(Competitive Preference)

20. Notes on Competitive Preference Priority 5 Innovations for Improving Early Learning Outcomes • “…improve educational outcomes for high-need students who are young children (birth through 3rd grade) by enhancing the quality of early learning programs” Focus on High-need Children • “…(a) improving young children’s school readiness (including social, emotional, and cognitive readiness) so that children are prepared for success in core academic subjects (as defined in section 9101(11) of the ESEA); (b) improving developmental milestones and standards and aligning them with appropriate outcome measures; and (c) improving alignment, collaboration, and transitions between early learning programs that serve children from birth to age three, in preschools, and in kindergarten through third grade.” Projects Must Address All 3 20

21. Notes on Competitive Preference Priority 6 Innovations that Support College Access and Success • “… enable kindergarten through grade 12 (K–12) students, particularly high school students, to successfully prepare for, enter, and graduate from a two- or four-year college…” • “…(a) address students’ preparedness and expectations related to college; (b) help students understand issues of college affordability and the financial aid and college application processes; and(c) provide support to students from peers and knowledgeable adults.” Focus on College Graduation Projects Must Address All 3 21

22. Notes on Competitive Preference Priority 7Innovations to Address the Unique Learning Needs of Students with Disabilities and Limited English Proficient Students • “…address the unique learning needs of students with disabilities, including those who are assessed based on alternate academic achievement standards, or the linguistic and academic needs of limited English proficient students.” • “…must provide for the implementation of particular practices, strategies, or programs that are designed to improve academic outcomes, close achievement gaps, and increase college- and career-readiness, including increasing high school graduation rates (as defined in this notice), for students with disabilities or limited English proficient students.” Focus on Either Student Population Projects That Improve Specific Outcomes 22

23. Notes on Competitive Preference Priority 8Innovations that Serve Schools in Rural LEAs • “…focus on the unique challenges of high-need students in schools within a rural LEA…” • “…must include practices, strategies, or programs that are designed to improve student achievement or student growth, close achievement gaps, decrease dropout rates, increase high school graduation rates, or improve teacher and principal effectiveness in one or more rural LEAs.” Focus on Specific Locations Projects May Address Range of Outcomes 23

24. Schedule for Today

25. Schedule for Today

26. i3 Selection Criteria and Points * Development grants will be judged in two tiers: all eligible applications will be scored on Criteria A, C, E, F, and G and the competitive preference priorities; then high-scoring applications will be scored on Criteria B and D by a different panel of reviewers.

27. Evidence and Evaluation Reviewers In Tier 2, Evidence and Evaluation Reviewers will Score Criteria B and D only

28. Development GrantsB. Strength of Research, Significance of Effect, and Magnitude of Effect (up to 10 points) The Secretary considers the strength of the existing research evidence, including reported practice, theoretical considerations, and the significance and magnitude of any effects reported in prior research on whether the proposed project will improve student achievement or student growth, close achievement gaps, decrease dropout rates, increase high school graduation rates, or increase college enrollment and completion rates. Eligible applicants may also demonstrate success through an intermediate variable that is strongly correlated with improving these outcomes, such as teacher or principal effectiveness. In determining the strength of the existing research evidence, the Secretary considers the following factors: The extent to which the eligible applicant demonstrates that there are research-based findings or reasonable hypotheses that support the proposed project, including related research in education and other sectors. The extent to which the proposed project has been attempted previously, albeit on a limited scale or in a limited setting, with promising results that suggest that more formal and systematic study is warranted. The extent to which the eligible applicant demonstrates that, if funded, the proposed project likely will have a positive impact, as measured by the importance or magnitude of the effect, on improving student achievement or student growth, closing achievement gaps, decreasing dropout rates, increasing high school graduation rates, or increasing college enrollment and completion rates.

29. Reasonable Hypothesis: Development

30. Development Grants D. Quality of the Project Evaluation (up to 15 points) The Secretary considers the quality of the evaluation to be conducted of the proposed project. In determining the quality of the evaluation, the Secretary considers the following factors: The extent to which the methods of evaluation are appropriate to the size and scope of the proposed project. The extent to which the methods of evaluation will provide high-quality implementation data and performance feedback, and permit periodic assessment of progress toward achieving intended outcomes. The extent to which the evaluation will provide sufficient information about the key elements and approach of the project so as to facilitate replication or testing in other settings. The extent to which the proposed project plan includes sufficient resources to carry out the project evaluation effectively.

31. Subject Matter Reviewers Subject Matter Reviewers will Score Criteria A, C, E, F, and G (not including the Evidence and Evaluation Criteria)

32. Development GrantsA. Need for the Project and Quality of the Project Design (up to 25 points) The Secretary considers the need for the project and quality of the design of the proposed project. In determining the need for the project and quality of the design of the proposed project, the Secretary considers the following factors: The extent to which the proposed project represents an exceptional approach to the priorities the eligible applicant is seeking to meet (i.e., addresses a largely unmet need, particularly for high-need students, and is a practice, strategy, or program that has not already been widely adopted). The extent to which the proposed project has a clear set of goals and an explicit strategy, with the goals, objectives, and outcomes to be achieved by the proposed project clearly specified and measurable and linked to the priorities the eligible applicant is seeking to meet.

33. Development Grants C. Experience of the Eligible Applicant (up to 25 points) The Secretary considers the experience of the eligible applicant in implementing the proposed project. In determining the experience of the eligible applicant, the Secretary considers the following factors: The past performance of the eligible applicant in implementing projects of the size and scope proposed by the eligible applicant. The extent to which an eligible applicant provides information and data demonstrating that— (a) In the case of an eligible applicant that is an LEA, the LEA has— (i) Significantly closed the achievement gaps between groups of students described in section 1111(b)(2) of the ESEA, or significantly increased student achievement for all groups of students described in such section; and (ii) Made significant improvements in other areas, such as graduation rates or increased recruitment and placement of high-quality teachers and principals, as demonstrated with meaningful data; or (b) In the case of an eligible applicant that includes a nonprofit organization, the nonprofit organization has significantly improved student achievement, attainment, or retention through its record of work with an LEA or schools.

34. Development GrantsE. Strategy and Capacity to Further Develop and Bring to Scale (up to 5 points) The Secretary considers the quality of the eligible applicant’s strategy and capacity to further develop and bring to scale the proposed project. In determining the quality of the strategy and capacity to further develop and bring to scale the proposed project, the Secretary considers: The number of students proposed to be reached by the proposed project and the capacity of the eligible applicant and any other partners to reach the proposed number of students during the course of the grant period. The eligible applicant’s capacity (e.g., in terms of qualified personnel, financial resources, or management capacity) to further develop and bring to scale the proposed practice, strategy, or program, or to work with others (including other partners) to ensure that the proposed practice, strategy, or program can be further developed and brought to scale, based on the findings of the proposed project. The feasibility of the proposed project to be replicated successfully, if positive results are obtained, in a variety of settings and with a variety of student populations. Evidence of this ability includes the availability of resources and expertise required for implementing the project with fidelity, and the proposed project’s evidence of relative ease of use or user satisfaction. The eligible applicant’s estimate of the cost of the proposed project, which includes the start-up and operating costs per student per year (including indirect costs) for reaching the total number of students proposed to be served by the project. The eligible applicant must include an estimate of the costs for the eligible applicant or others (including other partners) to reach 100,000, 250,000, and 500,000 students. The mechanisms the eligible applicant will use to broadly disseminate information on its project so as to support further development or replication.

35. Development Grants F. Sustainability (up to 10 points) The Secretary considers the adequacy of resources to continue the proposed project after the grant period ends. In determining the adequacy of resources for the proposed project, the Secretary considers the following factors: The extent to which the eligible applicant demonstrates that it has the resources, as well as the support of stakeholders (e.g., State educational agencies, teachers' unions), to operate the project beyond the Development grant. The potential and planning for the incorporation of project purposes, activities, or benefits into the ongoing work of the eligible applicant and any other partners at the end of the Development grant.

36. Development Grants G. Quality of the Management Plan and Personnel (up to 10 points) The Secretary considers the quality of the management plan and personnel for the proposed project. In determining the quality of the management plan and personnel for the proposed project, the Secretary considers: The adequacy of the management plan to achieve the objectives of the proposed project on time and within budget, including clearly defined responsibilities, timelines, and milestones for accomplishing project tasks. The qualifications, including relevant training and experience, of the project director and key project personnel, especially in managing projects of the size and scope of the proposed project.

37. Schedule for Today

38. Schedule for Today

39. What is the basic i3 review process? • The Department will use independent peer reviewers from various backgrounds and professions who have been thoroughly screened for conflicts of interest • Applications will be assigned to panels by absolute priority where possible • Evidence and evaluation experts will score the selection criteria (B and D) focused on evidence and evaluation • Peer reviewers will determine whether any competitive preference priority points should be added For all three grant types… • Development applications will be reviewed in a two tier process • In Tier 1, all eligible applicants will be reviewed and scored against Selection Criteria A, C, E, F and G by three peer reviewers. Competitive Preference Points will also be added as appropriate by peer reviewers. • Only those highest rated in Tier 1 will advance to Tier 2, where Selection Criteria B and D will be scored by two peer reviewers who are evidence and evaluation experts. Development Only…

40. What is expected of a Peer Reviewer? • Review the entire i3 application package and FAQs • Review and become thoroughly familiar with the selection criteria and factors • Familiarize yourself with the e-Reader system – used for inputting scores and comments • Participate in all scheduled conference calls • Provide appropriate comments that justify the score awarded and are helpful to the applicant • Revise comments as suggested by your panel monitor • Return all forms as required to ensure payment and completion of review process • Be available the entire review process • Draw upon your expertise • Maintain confidentiality throughout the review process Preparation Process Behavior

41. What must a Peer Reviewer really do? • Receive applications and panel assignments from Synergy • Identify any Conflicts of Interest, and notify Synergy or panel monitor immediately • Participate in Orientation webinar • Register in and become acquainted with e-Reader • Read applications (It may be helpful to you to take notes!) • Write draft comments, assign scores, and submit drafts in e-Reader before panel discussion (We recommend cutting and pasting from Word) • Receive feedback on comments from Panel Monitor • Revise draft comments based on feedback as appropriate • When comments/scores are acceptable to Panel Monitor, submit final comments/scores • Sign and return Technical Review Signature Forms to Synergy • Remember to submit All Necessary Forms to Synergy Prior to Panel Discussion After Panel Discussion

42. Conflict of Interest: Direct vs. Indirect Direct Conflict of Interest A peer reviewer has a DIRECT conflict of interest if: • The individual’s financial interests were affected by the outcome of the Investing in Innovation competition; • An individual helped prepare an Investing in Innovation application, even if he or she has no financial interest in the outcome of that application; • An individual has agreed to serve as an employee or consultant, or otherwise provide assistance or advice, on any project for which funding is being sought in any Investing in Innovation application, or has been offered the opportunity to do so, and has not yet accepted or declined. A peer reviewer has an INDIRECT conflict of interest if any of the following individuals or organizations has a personal financial interest in the outcome of the competition: •  The reviewer’s spouse, his or her child, a member of his or her household, or any relative with whom he or she has a close relationship; • Any employer the reviewer has served within the last 12 months; a business partner; an organization the reviewer has served as an officer, director, or trustee within the last 12 months; or an organization that he or she serves as an active volunteer; • Any person or organization with whom the reviewer is negotiating for, or has an arrangement concerning, future employment; or • Any professional associate – including any colleague, scientific mentor, or student – with whom the reviewer is currently conducting research or other professional activities, or with whom the reviewer has conducted such activities within the last 12 months. Indirect Conflict of Interest

43. Conflict of Interest: What must a Peer Reviewer do? • Complete the conflict of interest questions that were provided by Synergy. • Review the applications and contact your panel monitor immediately if, while reading the applications, you identify an area in which you may have a direct or indirect conflict of interest. • Specifically, a peer reviewer should review the applicant, official and other partners, and, if applicable, sources for the private-sector match to ensure that he or she does not have a conflict of interest with any of the entities associated with each application. Before Receiving Applications After Receiving Applications

44. Other than the peer reviewer, what are the key roles within the peer review process? • Manages relationship with Contractor (SEI/Synergy) • Coordinates across all panel monitors • Supports all panel monitors • Contact reviewers to schedule panel meetings • Assist reviewers with any questions • Identify any conflicts of interest • Make sure reviewers enter comments/scores in e-Reader promptly • Keep panel discussions on track • Promptly send feedback on written comments/scores to reviewers and notify reviewers when scores and be submitted in final form • Assign peer reviewers to panels, and re-assign as required • Mail applications to peer reviewers • Processes paperwork and compensates peer reviewers • Provide peer reviewers resources on their website Competition Manager Panel Monitor Contractor (SEI/ Synergy)

45. Schedule for Today

46. Scoring Applications • Numeric scores indicate how well the applicant responded to the selection criteria • Use the entire range of points for each criterion. • Make sure that your comments are consistent with your numeric scores • If full points are awarded to a criterion, all comments should be in “strengths” section and “no weaknesses found” should be entered in the “weaknesses” section. • If partial points are awarded to a criterion, there should be points and comments under both the “strengths” and “weaknesses” sections. • You should award zero (0) points only if all reviewers agree that the information needed to assess the criterion is missing.

47. Writing Comment - “Do’s & Don’ts”

48. Sample Comment – #1 “The application proposes to create a professional development program for teachers that will instruct them how to use performance data from class assignments and to use it to improve their teaching. It’s an excellent idea.” • Problems: The comment simply paraphrases the application, and it does not explain why the reviewer likes the idea. • Revision: “One of the strengths of the proposed professional development program is that it gives teachers direct practice in using the newly created online repository of student assignments that can be used “in the moment” when classroom assessments identify a learning deficiency. As the application demonstrates on p. 17, this technology-based approach aligns with research showing that learning deficits can be remedied if addressed immediately.” • Why is it Better?: It explains more clearly why the reviewer believes this is a quality program, and it includes specific references to the proposal.

49. Sample Comments – #2 “This management plan seems thorough and thoughtful, with appropriate LEA representation on the advisory council. The allocation of leadership and responsibilities seems appropriate to the tasks, and the schedule laid out in the chart seems reasonable.” • Problems: At first glance, these comments seem fine, but they are not as helpful as they could be. More detail is necessary to explain what about the management plan is “thorough and thoughtful.” What characteristics of the schedule or the allocation of responsibilities is the reviewer referencing? • Partial Revision: “…It makes sense in this case to split administrative leadership of the project from the coordination of the each of the school-based implementation sites and to assign these duties to different individuals…” • Why is it Better?: The revision makes more clear what specific aspects of the leadership plan are “appropriate” and “reasonable.”

50. Sample Comments – #3 (Weakness) “The administrative employee is on contract and is not a full-time employee. Is that advisable?” • Problem: It may appear that points have been deducted based on an issue of uncertainty, and the applicant cannot respond. It’s generally best to avoid questions. No weakness has been specifically identified. • Revision: Because the administrative employee is contracted, rather than serving as a full-time employee, it is unclear whether the applicant would be able to absorb this individual’s critical functions after the grant period is over. • Why is it Better?: The revision refrains from asking a question and more clearly explains why there might be a weakness in the way this employee has been budgeted.