1 / 16

Transport Canada Transports Canada

Transport Canada Transports Canada. Program Evaluation, Transport Canada May, 2000. INTEGRATING PERFORMANCE MEASUREMENT AND POLICY/PROGRAM EVALUATION IN TRANSPORT CANADA. Presentation for CES - Conference 2000 Presented by Jennifer Birch-Jones , Evaluation Manager

rrembert
Download Presentation

Transport Canada Transports Canada

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Transport CanadaTransports Canada Program Evaluation, Transport Canada May, 2000 INTEGRATING PERFORMANCE MEASUREMENT AND POLICY/PROGRAM EVALUATION IN TRANSPORT CANADA Presentation for CES - Conference 2000 Presented by Jennifer Birch-Jones, Evaluation Manager Gail Young, Evaluation Manager Program Evaluation Transport Canada May 16, 2000

  2. Objective • Share Transport Canada’s (TC’s) evolving approach to integrating performance measurement and policy and program evaluation • spheres of influence • attribution • on-going performance monitoring • research on performance challenges Page 2

  3. Transport Canada’s Approach: Performance is… • Not just staying within budgets and meeting revenue targets • Not just being very busy, all the time • Not just producing a lot of high quality goods and services • Not just having satisfied “customers” • but making a difference in the real world...changing attitudes and behaviours, social and economic conditions, the state of the environment, the sustainability of natural resources, the state of the world Page 3

  4. Transport Canada’s Approach: Add Value • Our evaluation unit is part of the policy shop - distinct from audit and consulting functions that reside in corporate services • We look at program and policy impacts • We have been designated as the centre of functional expertise for performance measurement, as well as evaluation • Our strategic objectives include positioning ourselves to contribute to strategic decision making Page 4

  5. Transport Canada’s Approach: Work with Managers When it comes to performance, managers and evaluators both want to • Understand the policy or program logic • Keep track of what’s happening and make changes where needed • Figure out how policy/program interventions are making things happen - cause and effect • Report performance - monthly, yearly, every 5 years... Page 5

  6. State Behavioral Change Behavioral Change > Changes in individuals and organizations in - awareness - acceptance - ability/capacity - action/adoption > the “what” and the “who” > direct outcomes > Changes in individuals and organizations in - awareness - acceptance - ability/capacity - action/adoption > the “what” and the “who” > direct outcomes > the vision - the ideal world > benefits to the target population, the Canadian public, the industrial sector, the resource > the “why” > long-term impacts Operational Operational Operational > internal operations > the “how” > activities & outputs > internal operations > the “how” > activities & outputs > internal operations > the “how” > activities & outputs Direct Control Direct Control Direct Control Indirect Influence Direct Influence Direct Influence Performance Story Logic - Spheres of Influence With many thanks to Steve Montague, PMN Page 6

  7. Behavioral Change Behavioral Change State Target Audience > Rail Industry > General Public > NGOs > Passengers > OGDs > TSB Target Audience > Rail Industry > General Public > NGOs > Passengers > OGDs > TSB Vision > Best possible rail transportation system Safety Results> Protection of life, property and environment > Public confidence enhanced Operation Operation Operation Operation Operation > Legislate / Regulate > Inspect / Enforce > Subsidize > Educate > Legislate / Regulate > Inspect / Enforce > Subsidize > Educate > Legislate / Regulate > Inspect / Enforce > Subsidize > Educate > Legislate / Regulate > Inspect / Enforce > Subsidize > Educate > Legislate / Regulate > Inspect / Enforce > Subsidize > Educate Results > Awareness > Acceptance> Ability> Adoption / Action Results > Awareness > Acceptance> Ability> Adoption / Action Direct Control Direct Control Direct Control Direct Control Direct Control Direct Influence Direct Influence Indirect Influence Rail Safety’s Performance Story Page 7

  8. NaturalEnvironment Target Audience > Rail Industry > Vehicle Drivers > NGOs > Other Road Users > Provinces / Municipalities Safety Benefits > Reduction in Rail Crossing Accidents / Fatalities / Injuries > Public Confidence Enhanced Social / Economic Factors Results > Increased Road User Alertness (Awareness )> Improved Crossing Infrastructure (Ability)> Adoption of Safe Practices by Rail Industry and Road Users (Action) Direct Control Media Indirect Influence Safety Performance Story for Rail Crossings State Operation > Regulate > Educate > Incentives> Enforce Direct Influence Page 8

  9. NaturalEnvironment Target Audience > Provinces / Municipalities Safety Benefits > Reduction in Rail Crossing Accidents / Fatalities / Injuries > Public Confidence Enhanced Social / Economic Factors Results > Improved Crossing Infrastructure (Ability) Direct Control Media Indirect Influence Safety Performance Story for Rail Crossing Contributions State Operation > Financial Contribution Direct Influence Page 9

  10. Examples of Measures for Rail Safety Page 10 (See Attachment 1 for a more complete list)

  11. Application to Policy • A similar approach can be used to tell the performance story for major policies • Can lead to some interesting observations about actual and potential government intervention to achieve policy objectives • For example…mapping the spheres of influence for government’s economic oversight of the air navigation system (ANS) raises questions about government’s role in influencing economic variables Page 11

  12. TC Minister > board appts > approvals TC Minister > board appts > approvals TC Minister > board appts > approvals Sector Behaviour State of Transportation • NAV CANADA • > financially viable • > compliant with • leg/regs • - respects int’l • agreements • - respects charging • principles • - respects level of • service conditions • > responsive to users on • - price & service issues • - capital investment in • new technology • > transparent • NAV CANADA • > financially viable • > compliant with • leg/regs • - respects int’l • agreements • - respects charging • principles • - respects level of • service conditions • > responsive to users on • - price & service issues • - capital investment in • new technology • > transparent • USERS • > satisfied with ANS • price & service • > satisfied with • openness of • NAV CANADA • USERS • > satisfied with ANS • price & service • > satisfied with • openness of • NAV CANADA • > ANS proportion of air transportation costs minimized TC TC TC Monitoring ? Monitoring ? Monitoring ? TC Controls CTA user-initiated appeals CTA user-initiated appeals CTA user-initiated appeals TC Influences TC Contributes to Spheres of Influence for ANS Economic Oversight NaturalEnvironment Social / Economic Factors Media Page 12

  13. Changes in variables: What happened? (PM) Success: What was our contribution to changes at the state level? (eval) Effectiveness and success: Did we make a difference? Was it the expected difference? (eval) Cost-effectiveness and alternatives: Is there a better way? (eval) Indirect Influence Direct Control Direct Influence Measurement by Sphere of Influence BEHAVIOURAL STATE Changes in variables: What happened? (PM) OPERATION efficiency (PM) productivity (PM) quality (PM) timeliness (PM) Is the intervention still appropriate? (eval) Page 13 (See Attachment 2 for more detail)

  14. Why it’s Working • Shows realistic accountabilities: managers are relieved to finally see it acknowledged that they don’t control or even directly influence everything - they readily accept the logic of spheres of influence • Responds to an immediate management need to monitor, review, improve, report • Creates a meaningful link between performance management and reporting to central agencies - no longer just feeding the beast Page 14

  15. Why it’s Working Continued • Helps managers do the job they know they should be doing: monitoring the performance of their own programs and policies • Lets evaluators do the job they do best • helping managers set up performance measurement systems • applying research expertise to the tough questions such as attribution • Builds a community of practice • learning • changing • sharing Page 15

  16. Challenges • Objectivity, independence, co-option…need we say more? • Hiring and training the “new” evaluator - we need facilitators with people skills who can apply shared leadership in a team setting…and be solid analysts as well • Manager has to be ready and evaluator has to be flexible and adaptable - may not always get the perfect framework, measures or data • Research to determine attribution is a methodological challenge that cannot always be met, given the resources available to evaluators Page 16

More Related