Demonstrating the impact of careers guidance
1 / 34

Demonstrating the Impact of Careers Guidance - PowerPoint PPT Presentation

  • Uploaded on

Demonstrating the Impact of Careers Guidance. Lester Oakes President IAEVG Karen Schober Vice-president IAEVG Bryan Hiebert Vice-president IAEVG . A Challenge from Policy Makers. You say you are providing effective services We believe you BUT Show us the evidence.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Demonstrating the Impact of Careers Guidance' - ira

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Demonstrating the impact of careers guidance l.jpg

Demonstrating the Impact of Careers Guidance

Lester Oakes

President IAEVG

Karen Schober

Vice-president IAEVG

Bryan Hiebert

Vice-president IAEVG

A challenge from policy makers l.jpg
A Challenge from Policy Makers

You say you are providing effective services

We believe you


Show us the evidence

Policy practice research dialogue l.jpg
Policy-Practice-Research Dialogue

  • 1999: First International Symposium on Career Development and Public Policy

  • Establishment of International Centre for Career Development and Public Policy

    • IAEVG has been an active supporter of the International Centre

  • International symposiums held in Canada, Australia, Scotland, New Zealand

  • 2009: Sixth International Symposium on Career Development and Public Policy

  • Predominating theme was Prove it Works

Overview l.jpg

IAEVG has been an active participant in past symposiums and outcome-focused, evidence-based, practice has been an important part of IAEVG strategic planning

  • Update from European Lifelong Guidance Policy Network

  • Update from Canadian Research working Group on Evidence-based Practice in Career Development

Slide5 l.jpg

Karen Schober

will update us on what is happening in the European Lifelong Guidance Policy Network

Outcome focused evidence based practice l.jpg
Outcome Focused Evidence-Based Practice


Framework developed by the Canadian Research Working Group on Evidence-Based Practice in Career Development

Outcome focused evidence based practice7 l.jpg
Outcome-Focused Evidence-Based Practice

InputProcess Outcome

  • Indicators of client change

  • Learning outcomes

    • Knowledge and skills linked to intervention

  • Personal attribute outcomes

    • Changes in attitudes,

    • Intrapersonal variables (self-esteem, motivation, independence)

  • Impact outcomes

    • Impact of #1 & #2 on client’s life, e.g., employment status, enrolled in training

    • Societal and relational impact

    • Economic impact

Outcome focused evidence based practice8 l.jpg
Outcome-Focused Evidence-Based Practice

Input ProcessOutcome

  • Activities that link to outputs or deliverables

  • Generic interventions

  • Working alliance, microskills, etc.

  • Specific interventions

  • Interventions used by service providers

    • Skills used by service providers

    • Home practice completed by clients

  • Programs offered by agency

  • Involvement by 3rd parties

Evidence based outcome focused practice l.jpg
Evidence-based Outcome-focused Practice


  • Resources available

  • Staff

    • Number of staff, level of training, type of training

  • Funding

    • Budget

  • Service guidelines

    • Agency mandate

  • Facilities

  • Infrastructure

  • Community resources

Outcome focused evidence based practice10 l.jpg
Outcome-Focused Evidence-Based Practice




Process + Outcome

What will I do? +How is it working?

Professional Practitioner

Evidence based practice l.jpg
Evidence-based Practice

  • Research trials (traditional way in psychology)

  • Professional Practitioner

    • Purposeful intervention

    • Data (= evidence) on what was done

    • Evidence on client change

    • Look for patterns in data linking intervention with outcome

    • Develop scientific attitude toward practice

      • Skepticism

      • Curiosity

      • Inquiry

Intervention planning intervention evaluation l.jpg
Intervention Planning & Intervention Evaluation

Intervention Planning Framework

Client Outcomes• Knowledge• Skills• Attributes• Impact

Context:Client Needs

Client Goals

Counsellor Strategy

Client Strategy





Intervention planning intervention evaluation13 l.jpg
Intervention Planning & Intervention Evaluation

Intervention Planning Framework

Client Outcomes• Knowledge• Skills• Attributes• Impact

Context:Client Needs

Client Goals

Counsellor Strategy

Client Strategy

Intervention Evaluation Framework




Quality of service delivery l.jpg


Regular hours

Extended hours

Physical accessibility

Resources in alternate format

Ease of access, who can access


% calls answered by 3rd ring

Wait time for appointment

Wait time in waiting room


Respect from staff

Courteous service

Clear communication

Overall satisfaction

% rating service good or excellent

% referrals from other clients

Quality of Service Delivery

Need to negotiate these with funders

Comprehensive service evaluation l.jpg
Comprehensive Service Evaluation

Quality Service framework

  • Service delivery

    • Client volumes

    • Client presenting problems

    • Number of sessions

  • Service standards

    • Staff credentials, competencies, resources

    • Efficiency (at client needs being met?)

  • System requirements

    • Adherence to mandate

    • Completion of paper work

    • Cost-effectiveness

  • Negotiated outcomes l.jpg
    Negotiated outcomes

    • If you are lucky, funders might identify personal attributes [client motivation, improved job satisfaction, increased self-confidence] or knowledge, or skills, as accountability indicators

    • BUT more likely funders will identify impact outcomes [employment status, enrolment in training, reduced # of sick days, increased productivity, etc.] or inputs [client flow, accessibility, timeliness of paper work, etc.]

    • So service providers need to identify the knowledge, skills, personal attributes that will produce the impacts and negotiate these as accountability indicators

      Be careful what you promise to deliver BUT

      deliver what you promisePromise small – DELIVER BIG

    Assessment as decision making vs judgement l.jpg
    Assessment as Decision Making (vs. Judgement)

    Please use a two-step process

    • Would you say that your level of mastery of the attribute under considerations is

    • Then assign the appropriate rating

      • 0 = really quite poor

      • 1 = just about OK, but not quite

      • 2 = OK, but just barely

      • 4 = really very good

      • 3 = in between barely OK and really good










    Problem with skill self assessment l.jpg
    Problem with skill self-assessment

    • Participants asked to rate their skill (or knowledge) before and after a program

    • Often, pre-workshop scores are high and post-workshop scores are lower

      • People find out as a result of the workshop that they knew less than they thought or had less skill than they thought

      • Based on the new awareness, post-scores are lower

    • People don’t know what they don’t know

    • How can we get around this problem?

    Assessing learning attribute outcomes l.jpg








    Assessing Learning & Attribute Outcomes

    Post-Pre Assessment

    We would like you to compare yourself now and before the workshop. Knowing what you know now, how would you rate yourself before the workshop, and how would you rate yourself now?

    Please use a two-step process:

    • Decide whether the characteristic in question is acceptable or unacceptable, then

    • assign the appropriate rating



    Crwg ongoing projects l.jpg
    CRWG: Ongoing Projects

    • Validate framework and approach

      • ACT evaluation

    • Field test interventions

      • Develop evaluation component as part of intervention

      • SME project

    • Field tests of practitioner use

      • LMI project

    • On the horizon

    Applied career transitions program l.jpg
    Applied Career Transitions Program

    Module 1

    Building Career


    Module 2

    Developing Career


    Module 3

    Getting Experience:

    The Internship

    • On-line Program Curriculum

    • Program Access Options

      • On-line only

      • Coached (four coaching appointments per module)

      • Coached with In-class Group Sessions (four coaching appointments and four group sessions per module)

    Results post pre assessment l.jpg
    Results: Post-Pre Assessment

    For Module 1

    • All together there were10 (items) x 29 (participants) = 290 ratings

    • Pre: 144 Unacceptable Ratings and 6 Exceptional Ratings

    • Post: 3 Unacceptable Ratings and 130 Exceptional Ratings

    • Exceptional Ratings increased from 2 to 44% of the participants

    • Pre: 50% Unacceptable Ratings; Post: 86% Acceptable Ratings

    Results impact outcomes l.jpg
    Results: Impact outcomes

    Module 1

    • 23 out of 29 had found a job

    • 10 of the jobs lined up well with career vision

      Module 2

    • 4 out of 6 had found a job

    • 3 of the jobs lined up well with career vision

    Attribution for change l.jpg
    Attribution for Change

    To what extent would you say that the changes depicted above were the result of completing Module 1 of the ACT program, and to what extent were they a function of other factors in your life?

    Wsi project career development in smes l.jpg
    WSI ProjectCareer Development in SMEs


    Career Development interventions for Small and Medium Enterprises to promote personal ownership for career planningand within-organization career mobility

    Three types of intervention

    • Minimal: Web-based self-help intervention

    • Moderate: On-the-job career conversations with managers, supervisors, colleagues

    • Intensive: Bilan des competences

    Wsi project l.jpg
    WSI Project


    General approach

    • Literature review

      • What is know already about the three areas of intervention

    • Formal needs assessment

    • Design intervention AND evaluation plan

    • Field test intervention

    • Consolidate evaluation evidence

    • Final report

    • Market results

    Wsi project27 l.jpg
    WSI Project


    General results

    • All 3 interventions worked

    • Different programs

      • designed for different purposes

      • requiring different resources

      • Chose the approach that best meets organizational needs

    • It’s better to do something than nothing

    Practitioner field tests l.jpg
    Practitioner Field Tests

    • Use and impact of Labour Market Information (LMI)

      • Isolate LMI from other interventions

    • Participant research approach

      • Normal clients seeking service

      • Agencies offering services

    • Use Post-Pre approach

    Research design l.jpg
    Research Design

    Job Search



    Time 2

    Time 1




    Lmi general results l.jpg
    LMI: General Results

    • All intervention-delivery combinations produced significant change

      • General ability to access and use LMI

      • Knowledge about how to use LMI

      • Skills for using LMI and taking action

      • Personal attributes, e.g., optimism, confidence

    • Assisted use produced greater change across time than independent use

    • 80% of clients attribute change to the program and not other factors

    On the horizon l.jpg
    On the Horizon

    Common Indicators Project

    • Identify 3-5 indicators of success that all agencies will collect

      • Focus groups to identify outcomes and data source

    • Aggregate results across agencies

    • Increase power of results

    Slide32 l.jpg

    Resources Available

    • Major report on evaluation practices

    • Special issue of Canadian Journal of Counselling

    • Sample tools and data gathering instruments

    • Evaluation workbook

    • Presentation notes, power point slides, etc.

    • Survey/needs assessment for self-help intervention

    To demonstrate value we need to develop l.jpg
    To demonstrate value, we need to develop

    Culture of evaluation:

    We need to reach the state where

    • Identification of outcomes is an integrated part of providing services

      • Without efficacy data, career services are vulnerable

      • It is in our best interest to gather evidence attesting to the value of the services we provide

    • Measuring and reporting outcomes is integrated into practice

    • Outcome assessment is a prominent part of counsellor education

    • Reporting outcomes is a policy priority

      This needs to be a priority in all sectors

    Demonstrating the impact of careers guidance34 l.jpg

    Demonstrating the Impact of Careers Guidance

    Questions and Comments

    Lester Oakes

    President IAEVG

    Karen Schober

    Vice-president IAEVG

    Bryan Hiebert

    Vice-president IAEVG

    [email protected]