California Community College Accountability

1. California Community College Accountability/Performance Framework Project Patrick Perry System Vice Chancellor of Technology, Research, & Information Systems Willard Hom Director of Research, System Office

2. Who Are We? California Community College System: largest in nation 109 campuses 2.5 million students System Office: Sacramento Unit and enrollment-level data collection since 1992 Greetings from Arnold!

3. Prior Efforts �Partnership for Excellence� (PFE, 98-04) used 5 metrics to measure system and college-level performance PFE was tied to money for improvement Money doled out by FTE, based on system performance had system not �performed�, money would have been doled out by college performance

4. PFE Metrics: Annual volume of transfers to CSU/UC Annual volume of awards/certificates Rate of successful course completions (all courses) Annual volume of Voc. Ed. Course completions Annual volume of basic skills improvements

5. PFE: In Memoriam When the budget got tight, PFE money was used to pay CA�s electric bill PFE �volume� metrics were not good metrics to do institutional comparisons with Would have been a disaster to base allocations on! 4 of 5 PFE metrics were �annual volumes� (only 1 rate)

6. Arnold says: �Community Colleges are eemportant parts of de economic reee-covery of Callefornya, and I attended one of deez colleges myself. I will geeev you deez new monies to help all of Callefornya, but de seestem weeel be accountable for dair expenditure.� The result: a bill, AB1417.

7. AB1417? The bill charged our BOG to recommend the structure of a college-level performance evaluation system We got to make it up...and pump it up Bill passed October 04; framework & metrics due March 05 Oh and by the way, get everyone�s input and approval We literally had 90 days to do it

8. AB1417-Accountability CCC RP Group hired as contractor External Panel of Experts reviewed model Trudy Bers, Peter Ewell, Joseph Burke, Jim Jacobs, Andrew Gill Nov. 04-Mar. 05�creation, approval from Consultation, BOG, Finance, Leg Analyst�and Legislature Was placed into budget act for 05-06 (enacted it) First report is due March 2007 In the end...Arnold said:

9. AB1417 The Model: Measures 4 areas: Student Progress & Achievement-Degree/Certificate/Transfer Student Progress & Achievement-Vocational/Occupational/Workforce Dev. Pre-collegiate improvement/basic skills/ESL Participation

10. AB1417 The Model: Has two levels of reporting: College Core Indicators Metrics reported on for all colleges All are RATES System Indicators Will report system-level metrics only; not broken out by college Most are VOLUMES

11. Student Prog. & Achievement: Degree/Cert/Xfer District Core: Student Progress & Achievement Rate-**NEW**(page 2A) 1st year to 2nd year (Fall-Fall) persistence rate System: Annual volume of transfers In-state private/out of State institutions **NEW** (page 2B) Transfer Rate for 6-year cohort of FTF�s (page 2C) Annual % of BA/BS grads at CSU/UC who attended a CCC

12. Student Prog. & Achievement: Voc/Occ/Wkforce Dev College Core: Successful Course Completion rate: vocational courses System: Annual volume of degrees/certificates by program Increase in total personal income as a result of receiving degree/certificate (p. 3A, 3B) Broken out by program

13. Wage Gain Research A data matching project linking labor agency wage records to students and their educational histories in the state CC system. Analyses showing pre-exit wages and post-exit wages Analyses by occupational area

14. Precollegiate Improvement/Basic Skills/ESL College Core: Successful Course Completion rate: basic skills courses ESL Improvement Rate Basic Skills Improvement Rate System: Annual volume of basic skills improvements

15. Participation College Core: None. System: Statewide Participation Rate (by demographic)

16. AB1417: Performance Evaluation No carrots or sticks-not tied to $. Colleges will not be ranked. College performance will be against that Colleges� past performance; you will see 3-yr running total. All parties desired the need for some type of meaningful comparison, however. But�we didn�t want to make it too easy�

17. Disclosure Minefields We have experienced 3 �traps� related to metric disclosure: Legislature�s desire to �rank� and �punish� low performers The Media and their short attention span The Enemy is US� �We�re Number One!-itis� The Goal: Make it useful but not too simple

18. AB1417: Peer Grouping �Peer grouping� summaries will be available as report appendices. For each college metric, System Office will perform regressions to find exogenous factors that affect rates. Peer groupings unique to each metric will group multiple logical clusters so you can �eyeball� your performance within each exogenous peer group.

19. Peer Group Formation

20. Exogenous Variables For example, the following affect a college�s Progress & Achievement Rate: (p. 5A) Preparedness of Incoming Students Economic Service Area Index (income) Distance to nearest public 4-yr institution Percentage of students >30 yrs old at college We have derived many of these

21. College Peer Group Each college will have one peer group for each indicator Colleges will be in different peer groups throughout the report Peer groups will come from a cluster analysis (p. 5B)

22. Economic Service Area Index A data match linking Census (household income) per zip code to the students attending each community college Provides a more precise measure of the background of students enrolled at a CC than county-level economic data Cut by college (page 5A)

23. Preparedness Index Data match with CA K-12 system to get SAT-9 and CAT-6 test scores (11th grade) Matched with incoming FTF cohorts at the CCC 2-3 years later Cut by college to create �index of academic preparedness� for incoming FTF cohorts (page 5A) Explains a huge amount of variability for many CCC metrics

24. Conclusions We feel that we are making advancements in research from a mandated (and usually resisted) report Performance indicator equity will be improved (even though the focus is not institutional comparison)

25. Hasta la Vista Baby! I�LL BE BACK (in Kansas City).