Doing More with Less

1. Doing More with Less More or Less Victor M. H. Borden, Ph.D. Professor, Educational Leadership and Policy Studies Indiana University Bloomington vborden@indiana.edu

2. Conclusions • IR has two basic purposes • Represent, protect and defend the institution • Inform and facilitate improvements • Do something worthwhile or else • You may end up doing nothing with nothing

3. The Purposes of IR • What Saupe said… • research conducted within an institution of higher education to provide information which supports institutional planning, policy formation and decision making. • What everyone but IR people just heard • Blah, blah, blah

4. The Purposes of IR Providing information and analysis for • Accountability and • Improvement

5. Information & Analysis for… • Accountability • Answering to a legitimate authority • Telling the institution’s story to “the public” • Demonstrating quality • Improvement • Figuring out what you are trying to achieve in measurable terms • Determining the degree to which you have made progress • Figuring out better ways to make progress

6. Quality of What? • Three factors distinguish top international universities from their competitors. The first: a high concentration of talented teachers, researchers and students…The second factor…[is] their sizable budgets…The third factor…is a combination of freedom, autonomy, and leadership…[T]heir status is conferred by the outside world on the basis of international recognition. JamilSalmi, Tertiary Education Coordinator, World Bank

7. Quality of What? • Learning—that is, the knowledge, skills and competencies a student gains by taking a college course or program—really needs to be recognized as the primary measure of quality in higher education. Right now, that is simply not the case. • Jamie P. Merisotis, President, LuminaFoundation

8. Quality of What? • [I]t should be obvious that quality is about content and intellectual innovation. If we are serious about having a high quality higher education system, then we have to start asking questions about content, avoiding the risk of suggesting that there is one standard way of measuring this • Ferdinand von Prondzynski, President, Dublin City University

9. Quality of What? • St. Ignatius Loyola… thought that the real test of higher education was what happened to the students — intellectually, socially, morally, and spiritually — under Jesuit tutelage. A university that measures its "greatness" by application numbers and endowment rather than by the character of its graduates is a school with a decidedly secular notion of greatness. • George Weigel, Senior Fellow of the Ethics of Public Policy Center

10. Quality • I know it when I see it • U.S. Supreme Court Justice Potter Stewart on obscenity, Jacobellis v Ohio , 1964

11. Providing information to an appropriate party that satisfies a legitimate interest about whether and how well something is being done Accountability

12. The Accountability Dilemma • Professionals have a responsibility to uphold standards of competence and morality (Schön) • There are legitimate external interests • Professionally-driven assessments for improvement do not satisfy external demands • External accountability demands require less palatable approaches

13. Egregious Example 1 • The dreaded “P” word • NCHEMS/Delta Cost Project/Lumina • Degrees “valued” according to the median income of recipients • Bachelor’s STEM > PhD Humanities • No distinction about cost to deliver • No accommodating value beyond personal income • Teachers, social workers not worth much?

14. Egregious Example 2 • Value-added learning measure in VSA pilot • Psychometrically unsound • Little between institution variance to account for • Ceiling effect results in most elite institutions having lowest gain scores

15. Ewell’s Assessment Paradigms

16. The Accountability Dilemma • We wouldn’t do what we should do if they weren’t forcing us to do so • But what they are forcing us to do isn’t what we should be doing

17. Paradoxical Tensions • Andreas Wagner (2008). Paradoxical Life • Dualisms • Self/other • Matter/meaning • And many more…

18. Self-Other • I tripped but you fell • Inside out: • Problems are due to insufficiency of materials with which we have to work • Student, facilities, finance, equipment • Outside in: • Problems are with the people doing the work and the processes they use to get their work done

19. Meaning/Matter • Positivist v. constructivist epistomologies • Outside in: • There are simple facts about things we can now across settings • Graduation rates, student/faculty ratios, instructional expenditures per FTE • Inside out: • You can’t compare us on common measures because we have different contexts • Selective v. open-access institutions

20. Living with Paradoxical Tension • Neither party to a dispute sees that its entrenched position is just one side of a coin. It raises itself over other, whole over part, body over mind, matter over meaning, or vice versa and denies the other side legitimacy. Nothing may be wrong, except the belief that one of them is the truth.

21. Principles for Effective Accountability

22. Dimensions of Quality - What • Student learning • Student character development • Scholarship • Boyer – discovery; integration; application; teaching (and learning) • Civic engagement • <insert other important stuff here>

23. Accommodating Diverse Learners – For Whom • (relatively) Homogeneous • Selective • Historically black; Hispanic-serving • Specialized institutions • Heterogeneous • From remedial to doctoral • Structurally diverse • Low to high SES

24. Connecting Performance and Outcomes - How • Performance • Programs and processes, the traditional quality assurance view • Transformations • Learning and other near-term outcomes • Impacts • Longer term consequences • Improving the quality of life for learners and other individual and communal beneficiaries

25. Validity for Purpose – Proof How

26. Improving accountability • If external accountability pressure is the primary driver of assessment of institutional effectiveness, then the only way to improve institutional improvement is to make accountability more accountable

27. How do you actually make something better? Improvement

28. How to Improve • Traditional IR answer • Through data-driven decision-making • Alternative approach • Collaborative organizational learning

29. I realize that I will not succeed in answering all of your questions. Indeed, I will not answer any of them completely. The answers I provide will only serve to raise a whole new set of questions that lead to more problems, some of which you weren’t aware of in the first place. When my work is complete, you will be as confused as ever, but hopefully, you will be confused on a higher level and about more important things The Institutional Research Credo

30. Rational Decision-Making Ethos • Rational, data-driven decision making processes yield more informed and successful decisions • Improved access to well-managed data sources provides decision makers with the tools they need to inform decision processes • To be useful, data must be turned into information, and information into knowledge • Data, information, and knowledge can all be managed • The primary function of research and information support is to provide information needed by decision-makers and decision-making groups

31. Herbert Simon on Bounded Rationality • …the complexity of the world in which we live, the incompleteness and inadequacy of human knowledge, the inconsistencies of individual preference and belief, the conflicts of value among people and groups of people, and the inadequacy of the computations we can carry out, even with the aid of the most powerful computers

32. Other Aspects of Non-Rationality • Primacy, recency and other cognitive biases (Kahneman & Tversky) • Coordination problems in group dynamics (Steiner; Thibaut & Kelley) • Heightening of emotion as related to competing perspectives and agendas (Mazur; Louis, Taylor & Douglas)

33. The Higher Education Context • The professional bureaucracy • Composite of hierarchical/bureaucratic and collegial • Professional (faculty) maintain superior authority to decide major goals • Administrators limited to deciding means and setting performance standards • Highly decentralized, loosely coupled authority structures

34. shift the paradigm

35. Collaborative Organizational Learning • Collaborative • Buy-in and transfer of learning through participation • Collective intelligence and diverse perspectives • Greater likelihood gained knowledge is useful • Organizational • Shared and/or aligned purposes and objectives • Accommodating interdependencies • It’s where we work • Learning • Incorporates uncertainty, ambiguity, and multiple styles • Individual and organizational learning are compatible concepts • It’s mission critical; it’s what we do

36. What’s important is not what the data say, It’s what we say and do about the data

37. Action Research • Continuous cycle of data collection  data analysis  data feedback  action plans  data collection • Stakeholder empowerment through active and on-going participation • Data feedback meetings promote collaboration, dialogue, and collective analysis • Active learning and discovery fostered by critical reflection process • Data-driven action plans developed = research linked to action

38. Linking Research and Action • Who does what? • Decides what actions are taken? • Is responsible for effective implementation? • Can devise appropriate evaluation protocols? • Has access to or can collect appropriate evidence? • Reviews the results and decides what to do? • What can be done to get these people to work together and in concert?

39. Doing More with Less • Stop doing all that %$#& that’s not really worth much and start doing more important things • Use your talents and skills to… • Contribute to making accountability more accountable • Work with your colleagues to learn how to do things better • Start charging on a per decision basis…

40. Scenarios in the Spreadsheet of My Heart What if…I received a shiny penny for identifying any at-risk student What if…I felt the tickle of a nickel for decisions I supported that were prudent (CHORUS) I know I won’t be getting very wealthy in this line of work I knew that from the very start But I can’t stop from modeling scenarios in the spreadsheet of my heart

41. Scenarios in the Spreadsheet of My Heart What if…I had a dime for every time someone thanked me for submitting IPEDS promptly What if…I received a brand new quarter when the data warehouse was a fait accompli (CHORUS) I know I won’t be getting very wealthy in this line of work I knew that from the very start But I can’t stop from modeling scenarios in the spreadsheet of my heart

42. Scenarios in the Spreadsheet of My Heart What if…Fifty cents were mine for every time our web site met the needs of ad hoc queries What if…I got a dollar for each scholar who supported all our methods and our theories (CHORUS) I know I won’t be getting very wealthy in this line of work I knew that from the very start But I can’t stop from modeling scenarios in the spreadsheet of my heart

43. Scenarios in the Spreadsheet of My Heart What if…The provost finally recognized the value added by my never-ending labors What if… My mother finally understood the work I do and could explain it to her neighbors (CHORUS) I know I won’t be getting very wealthy in this line of work I knew that from the very start But I can’t stop from modeling scenarios in the spreadsheet of my heart

44. References • Argyris, C., Putnam, R., & McLain Smith, D. (1985) Action science: concepts, methods, and skills for research and intervention.San Francisco: Jossey-Bass. • Borden, V. M. H. & Kezar, A. (in press). Institutional research as collaborative organizational learning. To appear in R. Howard, G. McLaughlin & W. Knight (eds). The handbook of institutional research. San Francisco: Jossey-Bass. • Borden, V. M. H. (2010). The accountability/improvement paradox. Inside Higher Education, Views, April 30. http://www.insidehighered.com/views/2010/04/30/borden. • Borden, V. M. H. & Pike, G. R.(2008). Assessing and accounting for student learning: Beyond the Spellings Commission. New Directions for Institutional Research, No. S1. • St. John, E.P. & Wilkerson, M. (2006). Reframing persistence research to improve academic success. New Directions for Institutional Research, 130. San Francisco: Jossey-Bass. • Schön, D. A. (1983) The reflective practitioner: How professionals think in action. London: Temple Smith • Wagner, A. (2009). Paradoxical life: Meaning, matter, and the power of human choice. New Haven, CT: Yale University Press.