Assessing the impact of lmi preliminary results of phase two
Download
1 / 52

Assessing the Impact of LMI: Preliminary Results of Phase Two - PowerPoint PPT Presentation


  • 91 Views
  • Uploaded on

Assessing the Impact of LMI: Preliminary Results of Phase Two. Research Team Canadian Research Working Group in Evidence-Based Practice (CRWG), Canadian Career Development Foundation (CCDF) In partnership with New Brunswick Post-Secondary Education & Labour ,

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Assessing the Impact of LMI: Preliminary Results of Phase Two' - neith


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Assessing the impact of lmi preliminary results of phase two

Assessing the Impact of LMI:Preliminary Results of Phase Two

Research Team

Canadian Research Working Group in Evidence-Based Practice (CRWG),

Canadian Career Development Foundation (CCDF)

In partnership with

New Brunswick Post-Secondary Education & Labour,

Saskatchewan Advanced Education, Employment & Immigration


Overview
Overview

  • Provide background on the project

  • Share results

  • Hear the perspectives of the research partners

  • Consider next research steps


Our research was an important step forward
Our Research was an Important Step Forward

  • Most LMI research focuses on usability of products

    • Readability

    • Accuracy of information

    • Easy to access

    • Amount of use

    • Most research is with students; very little with adults

  • Several questions remain unanswered

    • How do people use LMI?

    • What prompts them to make an action plan + implement it?

    • What (if any) assistance would be helpful?


  • There is very little literature about the direct contribution of LMI to employment/career decision-making

  • This research project directly addressed this question

    Savard & Michaud, The Impact of LMI on Career Decision-Making Process: Literature Review, FLMM, 2005


Research questions
Research Questions contribution of LMI to employment/career decision-making

  • If client needs are assessed and clients are given LMI consistent with their needs,

    • To what extent does assistance by a service provider enhance their effective use of LMI?

      OR

    • To what extent is independent self-help a sufficient process for clients to use LMI effectively?


Research questions1
Research Questions contribution of LMI to employment/career decision-making

  • What type of assistance in finding and using LMI, provided to clients with job-search and decision-making needs, leads to what kinds of outcomes?

  • How do clients process the LMI they access and how do clients use that information to create a plan for action?


Our approach was unique
Our Approach was Unique contribution of LMI to employment/career decision-making

  • We used real frontline counsellors with real “adult” clients in their customary settings – not counselling interns and students in an academic setting.

  • We based our experimental design process on the existing service delivery processes used by the participating agencies so the processes could be incorporated with little effort if research results proved positive.

  • We delivered interventions that isolated the effects of LMI on decision-making – separating it from other interventions such as career counselling or job search workshops that usually subsume LMI.


Method
Method contribution of LMI to employment/career decision-making

  • We reviewed current practices to determine:

    • The current process for identifying client service needs

    • “Favourite” LMI resources

    • Access to the Resource Centre


Favourite lmi resources
Favourite LMI Resources contribution of LMI to employment/career decision-making

  • We prepared “guided” LMI packages

    • Career Decision Making:

      • Know yourself

      • Know the Labour Market

      • Put it all Together

    • Job Search:

      • Check for “Fit”

      • Get Ready

      • Search for Work

      • Get a Job


D contribution of LMI to employment/career decision-making

I

M

E

N

S

I

O

N

S

E M P L O Y A B I L

I

T

Y

CCDF, 2010


Checklist for counsellor
Checklist for Counsellor contribution of LMI to employment/career decision-making


Another protocol to follow
Another protocol to follow contribution of LMI to employment/career decision-making


Even the client had a checklist
Even the Client had a Checklist contribution of LMI to employment/career decision-making

  • From the list of LMI resources they had in their LMI package, they had to check which they used and how many times. Example:

    • Career Cruising

    • Job Bank

    • Job Futures

  • And give names of any other sources they used


Intervention
Intervention contribution of LMI to employment/career decision-making

All participants in the study:

  • Received a needs assessment interview & completed an initial, pre-program survey

  • Received an LMI package specific to their identified employability need

  • Were randomly assigned to either a self- directed intervention delivery method or an assisted self-directed intervention delivery method.

  • Were given an orientation to the Resource Centre which they could freely use on their own

  • Completed a post-pre survey

  • Received a cash honorarium and certificate of participation


Intervention cont
Intervention (cont) contribution of LMI to employment/career decision-making

In addition, the assisted self-help clients received:

  • Two additional AIS interviews (20-30 minutes) focused on helping them understand, interpret and apply the LMI to their own situations and /or access additional LMI


Experimental conditions
Experimental Conditions contribution of LMI to employment/career decision-making

  • Intervention = a Career Decision-Making or a Job Search LMI Package

  • Delivery =

  • In two provinces: New Brunswick and Saskatchewan


Research design
Research Design contribution of LMI to employment/career decision-making

Job Search

Intervention

CDM

Time 2:

After

Time 1:

Before

Delivery

Independent

Assisted


Sample
Sample contribution of LMI to employment/career decision-making

  • 8 employment centres in two provinces, Saskatchewan and New Brunswick.

  • 169 participants began the study

    • 13 discontinued participation

    • 5 submitted incomplete data where one or more of the survey forms was missing

  • 151 provided complete data


  • Sample by province
    Sample by Province contribution of LMI to employment/career decision-making

    • There were no differences in pretest scores between provinces

      • Therefore we combined provinces for data analysis

      • This Increased the cell size + therefore increased the statistical power


    Sample continued
    Sample Continued contribution of LMI to employment/career decision-making

    • 74 males and 77 females

    • Age: 19 - 62 years (Mean age = 44 years)

    • 149 were Canadian citizens or landed immigrants (legally entitled to work in Canada)

    • 118 had not previously participated in employment services programs

    • 35 were working full or part-time

    • 6 said their work was a good fit for them

    • 115 participants were not employed


    Employment history
    Employment History contribution of LMI to employment/career decision-making

    • In the past 5 years

      • 30 of them had been in 5 or more jobs

      • 22 had been in 4 jobs

      • 25 had been in 3 jobs

      • 46 had been in 2 jobs

      • 24 had been in the same job for the past 5 years


    Unemployment history
    Unemployment History contribution of LMI to employment/career decision-making

    • In the past 5 years

      • 15 had been unemployed for 36-60 months

      • 15 had been unemployed for 18-30 months

      • 15 had been unemployed for 12-16 months

      • 19 had been unemployed for 6-11 months

      • 26 had been unemployed for 1-5 months

      • 42 had not been unemployed over the past 5 years


    Approach to evaluation
    Approach to Evaluation contribution of LMI to employment/career decision-making

    Outcome-Focused, Evidence-Based Practice

    Framework developed by CWRG

    InputProcessOutcome


    Outcome focused evidence based practice
    Outcome-Focused Evidence-Based Practice contribution of LMI to employment/career decision-making

    InputProcess Outcome

    • Indicators of client change

    • Learning outcomes

      • Knowledge and skills about using LMI

    • Personal attribute outcomes

      • Changes in attitudes, confidence, optimism, etc.

    • Impact outcomes

      • employment status


    Outcome focused evidence based practice1
    Outcome-Focused Evidence-Based Practice contribution of LMI to employment/career decision-making

    Input ProcessOutcome

    Activities that link to outputs or outcomes

    Tailored LMI Packages

    Protocol for assisted self-help

    Counsellor check lists

    Client checklists


    Outcome focused evidence based practice2
    Outcome-Focused Evidence-Based Practice contribution of LMI to employment/career decision-making

    InputProcessOutcome

    • Resources available

    • Staff time

    • Resource Centres

    • Self-selected on-line resources

    • Needs determination protocol


    What did we measure
    What Did We Measure? contribution of LMI to employment/career decision-making

    The Dependent Measures for the data analysis were:

    • General ability to use LMI

    • Knowledge

      • Clear vision of what I want in my career future

      • Knowledge of print and online resources

  • Skill

    • Have effective strategies for keeping myself motivated

    • Have a realistic action plan

  • Personal Attributes

    • Optimism about what lies ahead re meeting my career goals

    • Confidence in my ability to manage future career transitions


  • How did we measure problems with standard pre post design
    How Did We Measure? contribution of LMI to employment/career decision-makingProblems with standard Pre-Post Design

    • Pre: Before they experience a program, participants are asked to rate their skill (or knowledge)

      • Often, pre-program scores are higher because people don’t know what they don’t know

    • Post: After experiencing a program, participants are also asked to rate their skill

      • Often post-program scores are lower because people have found out that they knew less than they thought or had less skill than they thought

    • So, a comparison of difference in scores may not reflect the true difference.


    How can we get around this problem
    How can we get around this problem? contribution of LMI to employment/career decision-making

    Use a Post-Pre Assessment AFTER the program

    At the end of the intervention, we asked the participants to compare themselves now and before the intervention:

    “Knowing what you know now, how would you rate yourself before the research session, and how would you rate yourself now?”


    Descriptive results
    Descriptive Results contribution of LMI to employment/career decision-making

    • Looking at the 14 items in the survey instrument:

      • Before the intervention: between 30% and 58% of the respondents indicated that their level of competence on that item was “Not OK”;

      • After the intervention: 1% – 9% indicated that their level of competence on that item was “Not OK”.

    • Mean Scores:

      • Before the intervention: All responses (1 exception) were in the “Not OK” range;

      • After the intervention: All responses were greater than minimally OK (mean score 3 or greater) (2 exceptions where the mean scores were 2.19 and 2.96)


    Descriptive results cont
    Descriptive Results (cont) contribution of LMI to employment/career decision-making

    • Before the intervention: 45% (n=946) of responses were “not OK” (0 or 1);

    • After the intervention: 5% (n=95) of the responses were “not OK”.


    Descriptive results cont1
    Descriptive Results (cont) contribution of LMI to employment/career decision-making

    • Before the intervention: 5% (n=108) of the responses were “Exceptional” (4);

    • After the intervention: 39% (n=825) of the responses were “Exceptional”.

    • The amount of change was similar across all three dimensions of the survey: knowledge, skills, and personal attributes.

    • All 3 dimensions demonstrated about the same amount of change.


    Descriptive results cont2
    Descriptive Results (cont) contribution of LMI to employment/career decision-making

    • Of particular note are items that suggest increased ability to self-manage their careers, such as:

      • A clear understanding of what I need to do to move forward in my career.

      • A clear vision of what I want in my career future.

      • Knowledge of print and online resources that help me to research career/employment options.


    Descriptive results cont3
    Descriptive Results (cont) contribution of LMI to employment/career decision-making

    • More items that suggest increased ability to self-manage their careers:

      • The ability to access career resources that can help me implement my career vision.

      • Effective strategies for keeping myself motivated to achieve my career/employment goals.

      • A realistic action plan (or schedule) summarizing the main career/employment-related activities I want to pursue and the processes I am engaging in.

      • Confidence in your ability to manage future career transitions.

      • Confidence in my ability to research career, employment, and training options that are available


    Overall ability to use lmi
    Overall Ability to Use LMI contribution of LMI to employment/career decision-making

    • For group as a whole:

    • significant increase in skills for using LMI

    • neither intervention was more conducive to one manner of delivery compared to the other

    • Both CDM and JS groups had significant increases across time

    • Change in CDM group was significantly larger than in JS group

    • Participants in the JS group had higher scores than participants in the CDM group, likely indicating that JS participants were more familiar with using LMI before the project began.

    • Participants receiving assistance demonstrated greater change in skill at using LMI than did those in the independent mode


    Knowledge on how to use lmi
    Knowledge on How to Use LMI contribution of LMI to employment/career decision-making

    • For group as a whole:

    • significant increase in knowledge about how to use LMI

    • neither intervention was more conducive to one manner of delivery compared to the other

    • Both CDM and JS groups had significant increases across time

    • Change in CDM group was significantly larger than in JS group

    • Participants in the JS group had higher scores than participants in the CDM group, likely indicating that JS participants were more familiar with using LMI before the project began.

    • No difference in knowledge gain between those receiving assistance and those in the independent mode (p = .09)


    Skills in using lmi
    Skills in Using LMI contribution of LMI to employment/career decision-making

    • For group as a whole:

    • significant increase in skills for using LMI

    • neither intervention was more conducive to one manner of delivery compared to the other

    • Both CDM and JS groups had significant increases across time

    • Change in CDM group was significantly larger than in JS group

    • Participants in the JS group had higher scores than participants in the CDM group, likely indicating that JS participants were more familiar with using LMI before the project began.

    • Participants receiving assistance demonstrated greater change in skill at using LMI than did those in the independent mode


    Personal attributes related to using lmi
    Personal Attributes Related to using LMI contribution of LMI to employment/career decision-making

    • For group as a whole:

    • significant increase in personal attributes related to using LMI

    • neither intervention was more conducive to one manner of delivery compared to the other

    • Both CDM and JS groups had significant increases across time

    • Change in CDM group was significantly larger than in JS group

    • Participants in the JS group had higher scores than participants in the CDM group, but the gap was less at the end of the study (i.e., CDM group reported greater change).

    • Participants receiving assistance reported higher scores than did those in the independent mode


    Sample of learning outcomes
    Sample of learning outcomes contribution of LMI to employment/career decision-making

    • On these 6 items (K = 1, 2, 3, 5; S = 4; PA = 6)

    • Pre: 402 Unacceptable Ratings – Post: 35 Unacceptable Ratings

    • Unacceptable Ratings decreased from 45% to 4%

    • Pre: 56 Exceptional Ratings – Post: 343 Exceptional Ratings

    • Exceptional Ratings increased from 6% to 38% of the participants

    • Pre: 80% of means were Not OK – Post: all means were more than min OK


    Attribution for change
    Attribution for Change contribution of LMI to employment/career decision-making

    To what extent would you say that any changes in the ratings on the previous pages are a result of your participation in this research project, and to what extent were they a function of other factors in your life?

    Program


    Attribution for change1
    Attribution for Change contribution of LMI to employment/career decision-making

    • To what extent would you say that any changes in the ratings for the before and after survey questions are a result of your participation in this research project, and to what extent were they a function of other factors in your life?

      Of 148 respondents:

      • 47 said “somewhat this program”

      • 80 said “mostly this program”


    Impact outcomes 1
    Impact Outcomes-1 contribution of LMI to employment/career decision-making

    Employment status

    If you answered yes to the above question, to what extent does this work fit with your career vision?


    Impact outcomes 2
    Impact Outcomes-2 contribution of LMI to employment/career decision-making

    Action plan


    Impact outcomes 3
    Impact Outcomes-3 contribution of LMI to employment/career decision-making

    Desire help in creating an action plan


    Other noteworthy results
    Other Noteworthy Results contribution of LMI to employment/career decision-making

    • No significant gender differences

      • Women and men responded equally well to all intervention-delivery combinations

    • Other analyses in progress

      • Work history

      • Unemployment history

      • Age

      • Post-intervention interviews with selected clients (n=64) 1week and 4 months post the study


    Other noteworthy results cont
    Other Noteworthy Results (cont) contribution of LMI to employment/career decision-making

    • We also are analyzing the process data

      • The process checklists will provide evidence that the counsellors followed the game plan and that the clients were engaged with the LMI material

      • This will permit us to say that the program (Intervention + Delivery) is responsible for the change


    Lmi general summary of results
    LMI: General Summary of Results contribution of LMI to employment/career decision-making

    • All intervention-delivery combinations produced significant change in:

      • General ability to access and use LMI

      • Knowledge about how to use LMI

      • Skills for using LMI and taking action

      • Personal attributes, e.g., optimism, confidence, and by inference, motivation

    • Assisted use produced greater change across time than independent use

    • Over 80% of clients attribute change to the program and not other factors


    What have we learned
    What have we learned? contribution of LMI to employment/career decision-making

    Perspectives of the Research Partners


    What have we learned for clients in this study
    What have we learned? contribution of LMI to employment/career decision-making(for clients in this study)

    • “Guided” LMI, (i.e., LMI embedded in a learning process) results in knowledge and skill acquisition as well as the capacity for self-management

    • LMI appropriate for a client’s specific need (opposed to general LMI) appears to support engagement and action

    • A littleprofessional support can go a long way for many clients!

    • Structure, a roadmap, and timelines appear to motivate action and a sense of progress

    • Giving clients hands-on tools appears to motivate more than money!


    What helped make the project work
    What helped make the project work contribution of LMI to employment/career decision-making

    • The leadership of the provinces was exceptionally strong

    • The service providers caught the spirit of the research and went with it, even though it was not an exact fit for them

    • The front-end screening (service needs interview) was well done, and permitted a strong match between client need and intervention focus

    • The tailored intervention packages were superb

    • The attention to detail in data collection was very well done


    What needs to be done next a sample
    What needs to be done next? contribution of LMI to employment/career decision-makingA Sample…

    • This research was in English only; French to come (hopefully!)

    • There are 5 employability dimensions; only 2 have “guided” LMI

    • We do not know what the impact of “guided” LMI within an employment counselling process would be; nor do we know the impact of “guided” LMI packages self-selected by clients (no assessment) – these could be hugely informative for service delivery


    Thank you Research Partners, contribution of LMI to employment/career decision-making

    and

    Thank you for your attention today!


    ad