Research Outcomes from GK-12

1. Research Outcomes from GK-12 Presented by Jenifer Helms and Linda Fredericks GK-12 Rocky Mountain Regional Conference October 8, 2010

2. Agenda • Introductions • Program Design • Evaluation Overview • Outcomes • How Evaluation Has Changed Over Time • Evaluation Challenges • Supporting Project Improvement • Communicating Results • Q & A

3. Welcome and Introductions • Facilitators • Participants—name and project • Questions about evaluation/research

4. Program Design: University of Colorado at Boulder Participants • 9 Fellows from Ecology & Evolutionary Biology and Computer Science departments • 3 Faculty Co-Pis plus Asst. Superintendent of BVSD • Jessica Feld, Project Manager

5. Program Design: University of Colorado at Boulder Main Project Components • Content focus on extreme environments (field component at Thorne Institute salt marsh and the CU Mountain Research Station) with environmental sensors • 2 year fellowship, alternative 4th or 5th grade and middle school • Elementary focus on FOSS kits and extreme environments • Middle school has afterschool science club component • 1.5 days orientation during summer and first few weeks of school • 5 seminars with teachers and fellows • Biweekly fellows meetings • Monthly PI meetings • International Supplement – last year 4 fellows and a teacher went to Antarctica

6. Program Design, University of Colorado at Denver: “Transforming Experiences” Participants: • Graduate Fellows:Four mathematics fellows and four science fellows are chosen each year of the project. Fellows are called “resident mathematicians” and “resident scientists.” Fellows can re-apply and serve for more than one year if their applications are approved. • Middle School Teachers:Each fellow collaborates with one lead teacher from a local middle school. Every school has two fellows, one in mathematics and one in science.

7. Program Design from UCD, cont’d • Faculty Mentors:In the first two years of the grant, each fellow was assigned a faculty mentor, for a total of eight faculty mentors. Beginning this year, there is one faculty mentor working with the four mathematics fellows, and one faculty mentor working with the four science fellows. • Management Team: The PI,three co-PIs, and project coordinator meet on a monthly basis to discuss the project and plan events. The evaluator also attends the monthly meetings when available.

8. Program Design from UCD, cont’d Main Project Components: • Four-day orientation held in early summer for fellows and lead teachers to meet each other, work together, and socialize. Fellow-teacher matchups are made at this time. • In addition, all fellows attend a 5-day seminar in summer to engage in team-building activities, enhance communication skills, and learn about interdisciplinary learning, classroom culture, inquiry-based instruction, assessment, and international education. • Fellows attend bimonthly meetings to share experiences, discuss concerns, and learn new skills

9. Program Design from UCD, cont’d • All project participants meet together four times a year. • Teams of teachers and fellows from each school meet periodically to plan interdisciplinary activities. • There is an international component in collaboration with several universities in China. In May of 2011, a team of fellows, faculty members, lead teachers, and management team members will be traveling to China to conduct research activities, meet with peers, and observe middle school classrooms.

10. Evaluation Overview, UC Boulder • Pre- and Post-interviews with Fellows • Focus group in the winter • Pre-and Post interviews with Fellows • Survey of teachers • Interviews with teachers who attend MRS field trip • Surveys of students who attend MRS field trip • Interviews with graduate advisors • Fellows keep weekly logs of progress and issues

11. Evaluation Overview, UCD • Professional development evaluations • Pre/post assessments with fellows and lead teachers that measure attitudes and knowledge for teaching mathematics and science • Focus groups with fellows conducted at the end of fall and spring semesters • Interviews conducted at the end of fall and spring semesters with lead teachers, faculty mentors, and management team members • Classroom observations conducted in spring semester • Focus groups with students conducted in the spring semester • Interviews with former fellows and teachers conducted in the fall

12. Selected Outcomes, UC Boulder • Rocky first year, by second year found their rhythm, new PI • MRS field trip (day and overnight) big highlight • Teachers and Fellows cite numerous benefits • Fellows see value in what they are learning for their future career • There have been challenges for Fellows who work with multiple teachers • Successfully developed curriculum that incorporates sensors in the extreme environments

13. Selected Outcomes, UCD • There was strong agreement among respondents that middle school teachers and students benefited from project involvement. • Teachers rated graduate fellows excellent for respect for teacher ideas, subject matter knowledge, and enthusiasm for the subject. • Middle school teachers ratings of preparedness to teach mathematics and science topics and to use various teaching practices showed increases. • Project goals for training and mentoring fellows were more fully achieved at the end of Year 2. • There were concerns expressed about time requirements for the GK-12 program interfering with fellows’ research and other academic commitments.

14. How Evaluation Has Changed Over Time Evaluation needs to reflect and measure significant changes in program operation Examples: • At UCD, new questions werewritten into all qualitative instruments to measure the impact of the meetings that included fellows, teachers, and faculty mentors. • At UCD and CU Boulder, first year evaluation revealed a significant gap between participants’ expectations and actual experiences, so questions were added to instruments to gauge how well expectations for the project were explained and met. • At UCD and CU Boulder, first year focused heavily on identifying and remedying weak aspects of the project design • At CU Boulder added survey of student and teacher experience at MRS • At CU Boulder continuing to figure out best plan for classroom observations

15. Evaluation Challenges • Experiencing uncertainty about how to best help client • Obtaining a high response rate with a relatively small number of participants • Maintaining confidentiality when there are few respondents • Obtaining meaningful student impact data • Other?

16. Supporting Project Improvement • At UCD, first year reports of participants’ uncertainty about expectations of their own and others’ roles led to the development of an in-depth orientation manual as well as focusing some of the content of the orientation on explaining different roles. • At UCD, some issues with fellow/teachers matches are being addressed by a more extensive interview process with prospective teachers and greater time for fellows and teachers to collaborate . • At UCD, the lack of the fellow’s impact outside of one classroom has been addressed so that fellows are providing outreach to more teachers and classes and conducting interdisciplinary lessons with the other fellow at the school.

17. Supporting Improvement, cont’d • At CU Boulder, leadership issues in the first year lead to some tension in the relationship with the district. These issues were resolved in the second year. • At CU Boulder, concerns raised by fellows and teachers about expectations continue to be addressed. • Concerns about the overwhelming administrative issues required of the project manager the first year were addressed so that her work now focuses more on the fellows and teachers.

18. Communicating Results • At UCD, reports are prepared twice a year: at the end of the fall semester, and at the end of the academic year. A mid-year report provides useful feedback on the experiences of all project participants after the first semester, and allows for necessary changes to be made. The evaluator distributes written copies of reports to all Management Team members and provides an oral overview of the findings at Management Team meetings. In-depth findings and recommendations are always included. • At UCD, the evaluator attends monthly Management Team meetings to update members on evaluation efforts and raise any concerns that arise in the gathering of data.

19. Communicating Results, cont’d • At CU Boulder, evaluator attends most PI meetings to update the group on any issues identified through evaluation efforts • Periodic short reports/memos are distributed following data collection • Final reports are submitted in the spring along with the annual report

20. Questions? • Any questions? Thank you!

21. Contact Information Jenifer V. Helms Linda Fredericks Inverness Research Research Associate 556 Clayton Street RMC Research Corporation Denver, Colorado  80206 633 17th Street, Suite 2100 cell:  303.888.0432 Denver, CO 80202 home:  303.333.2701 Telephone: 303-825-3636 fax:  303.355.5235 Toll-free: 800-922-3636 mailto:jhelms@inverness-research.orgFax: 303-825-1626 www.inverness-research.orgEmail: fredericks@rmcdenver.com www.rmcdenver.com