Managing for organisational performance: What information do state sector managers actually rely on?

1. Managing for organisational performance: What information do state sector managers actually rely on? Joint Seminar School of Government, Victoria University Institute of Public Administration of New Zealand 15 & 21 May 2009 Derek Gill and Rob Laking A first report on results from the Managing for Performance (M4P) project in the Chief Executives’ Emerging Issues Programme (EIP)

2. Aim of the session • Present results from a survey of state sector managers on use of information for organisational performance management: • Headline results of interest • Preliminary analysis of drivers of use of information • Supplement analysis with observations from interview-based case studies • Invite questions and comments

3. Historical context • We have had output based accrual budgeting and reporting for twenty years • Output budgeting is resilient but: • A lot of non–financial performance measures are still “crap” <C&AG 2008> after nearly 20 years; • Managing for Outcomes has been a “failure” (along with KRAs/SRAs and outcome reporting in the old Public Finance Act)

4. Project to date • Goals • Positive - find what information do politicians managers and front line staff actually use to decide what to do and what, if any, use is made of the current performance information that they receive. • Normative –develop a proposal for a requisite system that better aligns key users’ wants and experiences and make recommendations for the direction of change • Progress • Completed fieldwork and analysis for seven case studies: two networks, five agencies; now writing up results • Completed questionnaire phase and most analysis of results of survey of 2500 state sector managers

5. Survey sample • Invitations to 2,500 managers in twelve Public Service departments (covering 60% of Public Service) and five Crown Entity Agents; • 75% started the survey and nearly 70% completed it; • No respondent bias evident in location, tier, non-completers or early vs. late completers;

6. Organisation parameters: Legal form Staff and budget SSC “type” General demand for information by purpose (Q14) Respondent parameters: Management tier (Q1) Location (Q2) Reporting staff (Q3) Work unit tasks (Q4) Dependencies Demand for numerical information by purpose (Q15) Demand for organisational information by purpose (Q17) Environment factors: Daily work influences (Q6) Motivation (Q8) Definition and value of “performance” (Q9, Q11) Quality of information (Q10) External climate (Q13) Survey structure Explanatory variables Dependent variables

7. Survey parameters • Went live in February 2009 • Emailed invitations to work addresses from lists supplied by HR departments • Chief Executives endorsed completion • Survey completed on-line • Respondents assured that their identities are confidential to core project group • Results given to organisations only in aggregate

8. Survey: headline results Management level of respondents

9. Survey: headline results Size of respondent’s work unit

10. Survey: headline results Location of respondents *Mainly corporate services units separate from head office

11. Survey: headline results Work unit main tasks *Managers reporting spent “a lot or nearly all” of their time (>2 tasks / mgr)

12. Survey: headline results Q9.1: Clear idea from management about organisation’s objectives.

13. Survey: headline results Q9.2: “know what is expected of us”

14. Survey: headline results Q9.3: “Mostly judged against specific performance targets”

15. Survey: headline results Q.8: Key motivators *Agree / strongly agree as percent of total sample

16. Survey: headline results Q.6: Professional and empowered or routinised and rule-driven? • Managers say they are empowered: • 96% of managers agree they “rely a lot on applying professional training or knowledge” • 75% agree they “have a lot of discretion in how we organise and prioritise our work” • But: • 80% agree they are “mostly guided by established rules or procedures” • 76% agree “we have a work plan and we stick to it” • Only 34% agree “we have a lot of freedom in how we allocated budget and staff”

17. Survey: headline results A public service of rules and control? • Managers are not ‘muddling through’ or mainly working in horizontal informal networks … • …they are mainly managing activities to plan, following rules; so it looks like … • … the formal system based on a rational control model focussed on control and budgeting is deeply embedded in most agencies. • There is less variation in these results than we expected.

18. Survey: headline results The dog that didn’t bark? What we expected and what we found • A former senior Minister: no-one in their right mind would rely on government performance information • We expected that managers: • Would think the information they got from the organisation was of poor quality • Would substitute informal, unstructured information for formal, structured information (the more so the closer they were to the front line) • We found that managers: • Have mixed views about the quality of information they get, but … • … make extensive use of formal organisational and numerical information for performance management (the more so the closer they are to the front line)

19. Survey: headline results Unstructured and local or structured and organisational? • Local: own groups and work contacts • Unstructured: conversations, reading and observation • Organisational: “from your organisation” • Structural: could be categorised and stored in a database; not necessarily numerical but we settled for “numerical” as a descriptor

20. Survey: headline results Use of numerical information *Validity of index: Cronbach’s Alpha = 0.917

21. Survey: headline results Use of organisational information Mean = 21.5 *Validity of index: Cronbach’s Alpha = 0.909

22. Survey: headline results Which managers use formal, structured information the most? • We expected: the closer to the front-line, the less managers use formal organisational and numerical information and the more they use tacit information and direct experience • What we found was important was more complex: • Task*: some work unit tasks seem to require more formal, structured information • Purpose*: similarly with some purposes for use of information • Location: the further from head office the more managers use organisational and numerical information • Tier: Tier 1 (top) and Tier 5 (bottom) use organisational and numerical information the most *Discussed further later on

23. Survey: headline results Regression results: what matters? • So far in analysis, three main factors identified: • Organisation: matters a lot - independent of legal form or type or task, specific organisational factors independently influence use of formal and numerical information • Task: the more time spent on service delivery and law enforcement and the less time spent on policy, the more use is made of numerical or organisational data • Manager location: the further from Wellington national office, the more use is made of numerical or organisational data • But variance (particularly across organisations) is still large. Environment, political salience, institutions, leadership, culture? • Using exploratory data analysis to investigate further.

24. Survey: headline results Information quality • We expected managers to say: They have better information on outputs (Q10.1, Q10.5) than outcomes (Q10.2, Q10.3). And they did! • But what was surprising was the relatively high dissatisfaction with information on quality and performance after nearly 20 years of output based budgeting.

25. Survey: headline results Quality of information *Percent respondents who agree or strongly agree

26. Survey: headline results “Lies damned lies and statistics… ….The reporting mechanisms are neither precise enough or encompassing enough for an accurate application to performance” (Text response on use of information (Q16) by tier 4 manager in mid-sized policy and delivery agency)

27. Survey: external-internal coupling Does external accountability drive internal performance management? • Testing central agency view that “it is critical that the same body of data that is used for internal decision-making be used for any external reporting.” (Treasury-SSC 2008). • Inferred from: • Information use by work units engaged in direct final services (strongest direct relationship to EA) • Work units using information on outputs, outcomes, process, inputs (basis for EA) • Use of specific performance targets (assumes meeting organisational requirements) • Extent of reliance on organisational information (assumes organisation enacts EA requirements internally)

28. Survey: external-internal coupling Relative high uses of numerical and organisational information

29. Survey: external-internal coupling Numerical information: high use on outputs and outcomes

30. Survey: external-internal coupling Numerical information: high use on inputs and processes

31. Survey: external-internal coupling Organisational information: high use on outputs and outcomes

32. Survey: external-internal coupling Organisational information: high use on inputs and processes

33. Survey: external-internal coupling Use of specific performance targets

34. Survey: external-internal coupling Cautious conclusions • For direct services to the public and direct enforcement of law and regulations, numerical and organisational data is relatively important; • For direct services to Ministers and managing joint projects, non-numerical and non-organisational data is relatively important. • External accountability does have a significant impact on managers’ demand for information to manage performance, particularly for direct services to the public or direct enforcement activities. • Overall, • local and unstructured information is demanded for all tasks • there is still a large unexplained variance.

35. Summary of research themes • Public sector reforms have shifted the locus of control from public service wide to agency level. But control still dominates • Managers tactically manage inputs through processes to deliver activities/outputs. Little evidence of “managing for outcomes.” • But ‘managing for outcomes’ is not ‘dead in the water’ – just under the radar

36. Managing for organisational performance (M4P) - Next steps • Sense making of survey results, analysis of survey text responses, case studies, literature review • Finalise review of formal system • Develop directions for reform (IPANZ/SOG seminar of 15 July) • IPS Book Publication (end August 2009)

37. Thank you and question time Further detail at http://ips.ac.nz/events/Ongoing_research/ Or email: derek.gill@vuw.ac.nz rob.laking@vuw.ac.nz