Assessing & Evaluating Impact

1. Assessing & Evaluating Impact NSF ADVANCE National Conference April 20, 2004 Dr. Cathy A. Trower

2. Overview • What is transformation? • What is culture? • What about data? • So now what?

3. “The progress of this institution…will be directly proportional to the death rate of the faculty.”

4. William T. Foster (1879-1950) President, Reed College – 1911 There were 46 students and 5 faculty members.

5. Law #1: Inertia “The status quo is the only solution that cannot be vetoed.” Clark Kerr (1911-2003) Chancellor, UC Berkeley President, UC Comment made 1982

6. Academic Transformation: An Oxymoron? • Alters the culture of the institution by changing select underlying assumptions and institutional behaviors, processes, and products; • Is deep and pervasive, affecting the entire institution; • Is intentional; and, • Occurs over time.

7. Organization Culture “A pattern of shared basic assumptions that the group learned as it solved problems…that has worked well enough to be considered valid and, therefore, to be taught to new members as the correct way to perceive, think, and feel in relation to those problems” (p. 12). (Schein, 1992, Organizational Culture and Leadership, Jossey-Bass).

8. Organization Culture The ethos of a place – the behavioral norms, espoused values, language, customs, rituals, and the modus operandi of an organization (Bolman and Deal, 1997; Schein, 1992). Three layers of culture • Artifacts: visible structures and practices • Espoused values: what people say they believe • Underlying assumptions: unconscious, taken-for-granted beliefs, thoughts, and feelings – ultimate source of values and actions

9. Why we don’t change • We are working primarily at the artifact level. • Academics have the lingo down pat – the espoused values are not a problem. ________________________________________ A. We don’t get to the underlying assumptions. --AND-- B. We don’t change the reward structure; we don’t reward what we say we value.

10. We don’t examine our underlying assumptions • The academy is a meritocracy • Those who perform get rewarded • There is an “ideal worker”* who… • Receives terminal degree in late 20s • Completes postdoc training • Takes a full-time tenure-track position • Performs a modicum of service, builds a teaching dossier, and generates research resulting in publications. * Penn State/Sloan “Faculty & Families” Drago, Crouter, Wardell, and Willits

11. We don’t examine our underlying assumptions • If he performs as an “ideal” worker, he gets tenure and promotion at the end of the 6th year. • He will then strive to generate a superior record to be promoted to full professor. • T&P are the most obvious rewards, but there is also: pay, course scheduling, resource allocation, accolades, citations, awards, and leadership positions.

12. Discrimination avoidance • Caregiving signals that you are not an “ideal” worker and are, therefore, a substandard academic. • Policy reform • No meetings before 8 or after 5 • Quality, onsite childcare • Part-time tenure tracks • Emergency leave • Stop-the-tenure-clock • Flexible schedules

13. Espoused values v. reward structures We say we value… • Teaching and service, but research earns you tenure. • Interdisciplinary work, but discipline-specific work brings in grants/prestige. • Diversity (we need you on committees), but it won’t earn you tenure. • Students, but spending time with them will hurt your tenure bid. • Innovation, but reward the status quo. • Collaboration, but reward competition (solo work). • Academic freedom, but your work better reinforce the prevailing norms rather than rock the boat. • New thinking/entrepreneurialism, but uniformity and conformity win. • Liberalism, but reward conservatism.

14. Multiple cultures • National System of Higher Education • Academic Profession • Discipline • Institution • Department

15. Catalyze Compare Identify/Signal Illuminate/Enlighten Influence Inform Monitor Socialize Substantiate Symbolize Roles that Data Play

16. Why No Direct Link? • Assuming rationality may be erroneous. • Data are used selectively for political and symbolic purposes that may or may not be tied directly to decisions. Links between data and decisions get lost in chaos and “organized anarchy.” • We tend to assume that all participants have all the data at the same time. • But this rarely happens. Decisions are, in part, a function of the availability of data at any given decision moment.

17. Why No Direct Link? • The amount of data that people can or choose to consume differs. Data consumption depends on the person and the issue. • Sometimes a sample of one is all people need to draw a conclusion or make a case. e.g., A senior woman in chemistry; a woman dean…becomes iconic

18. Why No Direct Link? • Data appear in different places at different times and people carry data from one arena to another and use it in ways not originally intended. • Difficult to get the “right” data into play in the “right” venue.

19. Why No Direct Link? • Data “use” is an ambiguous concept. • Information is sought from numerous sources in a non-linear way. • Often, decision-makers are not sure what data, if any, they are using and how they are using it. • People gather data but use other means to make decisions. • Decisions are often made before seeing data and the data are then used to justify the decision. • Decision-makers rarely admit to “gut-feel” decisions so pretend to use data.

20. Academe Even More Complicated • Faculty work, the workplace, and the culture are complex and not easily quantified. • Faculty autonomy is high at the best institutions -- a weak internal market for performance measures. • Even when quantifiable, the data are subject to multiple interpretations. • Data do not address visceral concerns. How do you place academic freedom, peer review, tenure and promotion, into the realm of data?

21. Academe Even More Complicated • When the issue concerns beliefs and values, data have less sway. • Difficult, if not impossible, the locate the locus for many decisions in academe given shared governance processes. “Decisions happen…”

22. Henry Rosovsky (1990) • “Never underestimate the difficulty of changing false beliefs with facts.” • “When given the opportunity--in the absence of incontrovertible scientific proof, and sometimes even then--people believe what they wish, and empirical evidence does not lead to quick altering of cherished positions.” The University: An Owner’s Manual

23. Framework for Assessing Change • Clarity about intended outcomes • Consideration of unintended outcome • Comparisons to baseline data, measured over time • What activities, processes, practices, outcomes, expectations, structures, experiences, language, and symbols are different as a result of the intervention? • Are we working at all three levels? • Are we working on the multiple cultures?

24. “The Master’s tools will never dismantle the Master’s house.” --Audre Lourde