Beginning with the end in mind what we know about performance measurement for vmf programs
This presentation is the property of its rightful owner.
Sponsored Links
1 / 50

Beginning with the End in Mind: What We Know about Performance Measurement for VMF Programs PowerPoint PPT Presentation


  • 85 Views
  • Uploaded on
  • Presentation posted in: General

Beginning with the End in Mind: What We Know about Performance Measurement for VMF Programs. If you have not already done so, please connect to the audio portion of this Webinar: 1. Dial 1-877-668-4490 2. Enter the access code 622 071 435 #

Download Presentation

Beginning with the End in Mind: What We Know about Performance Measurement for VMF Programs

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Beginning with the end in mind what we know about performance measurement for vmf programs

Beginning with the End in Mind: What We Know about Performance Measurement for VMF Programs

If you have not already done so, please connect to the audio portion of this Webinar:

1. Dial 1-877-668-4490

2. Enter the access code 622 071 435 #

3. Enter your unique participant ID number displayed

on your screen

OR choose the option to let the program call you back!!!


Housekeeping

Housekeeping

  • Phones are muted

  • Ask questions by sending a chat message to “All Participants” using the chat panel on the right side of your screen


Beginning with the end in mind what we know about performance measurement for vmf programs

Who’s on the Webinar

175 Attendees

Representing Programs and Organizations from 44* States


Beginning with the end in mind what we know about performance measurement for vmf programs1

Beginning with the End in Mind: What We Know about Performance Measurement for VMF Programs

Outcomes

You will:

  • Learn about Performance Measurement for Veteran and Military Family Programs (VMFP)

  • Discover methods to measure outputs and outcomes with available tools

  • Learn about VMFP data sources to help measure outputs and outcomes

  • Explore best practice examples in measuring outcomes from Equal Justice Works


Hello and welcome from cncs

Hello and Welcome from CNCS!

  • Koby Langley

  • Corporation for National and Community Service

  • Webinar hosts: Education Northwest


Beginning with the end in mind what we know about performance measurement for vmf programs

The Serve America Act’s Influence on CNCS VMF Performance Measures

Employment

*Service to VMF

*Creating Service/ Volunteer Opportunities for VMF

Employment

Disaster Preparedness

Youth Mentoring

Access to Benefits

Education

Health

Transportation

Education and Certification

Wellness and Other

Support Services

* Overarching activities

of programs in SAA

*Community Coordination

Specific activities

Mentioned in SAA


Session presenters

Session Presenters

Dr. Chris Spera, CNCS Director

of Research and Evaluation

Kathryn Gravely, Program Manager, Federal Programs and Strategic Initiatives, Equal Justice Works

Kerry O’Brien, Director of Federal Programs and Strategic Initiatives

Equal Justice Works


Key national service saa issues areas by program

Key National Service SAA Issues Areas by Program


Wellness other supports and services by program n 59

Wellness/Other Supports and Services by Program (n=59)


Beginning with the end in mind what we know about performance measurement for vmf programs

Performance Measurement for Veteran and Military Family Programs

  • What is performance measurement?

    • A systematic process of tracking outputs and outcomes based on the goals of your program, as articulated in your program’s logic model

  • What is a logic model?

    • A logical chain of connectionsshowing what the program is to accomplish


  • Beginning with the end in mind what we know about performance measurement for vmf programs

    Why is Performance Measurement important for CNCS?

    • Assess the individual and collective results of programs and continue to enhance program effectiveness

    • Expand CNCS’s ability to account for the combined contributions of all organizations making use of our resources


    Beginning with the end in mind what we know about performance measurement for vmf programs

    Why is Performance Measurement important for VMF Programs?

    • Assesses the individual and collective results of VMF programs by tracking outputs and outcomes related to improving

      • Military family strength and resilience

      • Quality of life for veterans, i.e.,:

        • Transition to civilian life

        • Education attainment

        • Job/vocational skills

        • Behavioral health


    Performance measurement for vmfp collecting data on outputs and outcomes

    Performance Measurement for VMFP: Collecting Data on Outputs and Outcomes

    • Tools to measure outputs:

      • Intake forms, attendance and tracking sheets to keep count

  • Tools to measure outcomes:

    • Pre, mid-term and post program surveys designed to measure increases in specific areas over time, as a result of participation in the program, focus groups, interviews

  • Where to find instruments, tools and resources for performance measurement:

    • http://www.nationalservice.gov/resources/npm/home

      • search by focus area


  • Beginning with the end in mind what we know about performance measurement for vmf programs

    Performance Measurement for Veteran and Military Family Programs

    • Important point: Performance measurement is NOT evaluation

      • “Unlike performance measures, evaluations estimate the impacts of programs by comparing the outcomes for individuals receiving a service or participating in a program to the outcomes for similar individuals not receiving a service or not participating in a program.”

        • Code of Federal Regulations for CNCS

          http://www.gpo.gov/fdsys/pkg/CFR-2010-title45-vol4/pdf/CFR-2010-title45-vol4-part2522-subpartE-subjectgroup-id1784.pdf


    Beginning with the end in mind what we know about performance measurement for vmf programs

    Performance Measurement for Veteran and Military Family Programs

    • Performance measures track outputs and outcomes based on the goals of your program’s logic model

  • A logic model shows the chain of connections illustrating plans to meet community needs; the theory of change

  • Need: access to employment resources

    INPUTS

    OUTCOMES

    OUTPUTS

    Program investments

    Activities

    Participation

    Medium

    Long-term

    Short

    What we invest:

    People, time

    Service provided:

    # of in-take & referrals

    Service recipients:

    # of returning veterans receiving referrals

    What results/changes:

    Veterans placed in employment prep programs; then employed


    Beginning with the end in mind what we know about performance measurement for vmf programs

    Outputs for Veteran and Military Family Programs

    • Defining outputs

      • Amount of service provided (e.g., people served, products created, or programs developed) through the service intervention of your program

  • Common outputs for VMF programs

    • # of veterans or active Service members and families served

    • # of veterans or active Service members engaged in service provision through CNCS programs

  • Others?


  • Beginning with the end in mind what we know about performance measurement for vmf programs

    Outputs for Veteran and Military Family Programs – Program Examples

    • Veterans Helping Veterans Now

      • # of veterans referred to service providers that can meet their needs

  • Metro Community College, Omaha, Nebraska

    • # of people who received outreach from VISTAs regarding support needed by veterans returning to college

  • CA Conservation Corps

    • # of veterans employed as crew members


  • Beginning with the end in mind what we know about performance measurement for vmf programs

    Outcomes for Veteran and Military Family Programs

    • Defining outcomes

      • Reflect the changes in attitudes/beliefs, knowledge/skills, behavior, or conditions of individuals that occur in the organizations, communities, or environment as the result of the services provided

    • Common outcomes for VMF programs

      • Increase in educational attainment

      • Increase in vocational skills and credentials (e.g., professional licensure)

      • Increase in employment outcomes

    • Others?


    Beginning with the end in mind what we know about performance measurement for vmf programs

    Outcomes for Veteran and Military Family Programs– Program Examples

    • Veterans Helping Veterans Now

      • # of veterans receiving services as result of referral who then gain employment (for example) due to services received

  • Metro Community College, Omaha, Nebraska

    • # of veterans succeeding in college due to support of informed college staff and faculty

  • CA Conservation Corps

    • # of veterans who completed the program and gained key skills for future employment


  • Sources of data collection

    Sources of Data Collection

    • Self-reported data through national service members and volunteers (e.g., # of veterans served)

    • Self-reported data from veterans through surveys (e.g., percentage of veterans served that obtain employment post-service)

    • Public use and administrative data sources (e.g., data from local VA organizations)

    • Observation – outside observers making ratings, counting behaviors


    Self report data from service members

    Self-Report Data from Service Members

    • Key Question: Who are you serving?

    • Self-reported data through national service members and volunteers

    • Example: # of veterans served

    • Example: # of military families served

    • Important to know penetration rate (i.e., percent of veterans served in relation to total size of veteran population in your community)

    • VA data source, called VETPop, has publicly available data source on veteran population by zip code


    Self report data from beneficiaries

    Self-Report Data from Beneficiaries

    • Key Question: What outcomes are you achieving?

    • Potential Outcomes:

    • Increase in educational attainment

    • Increase in vocational skills and credentials (e.g., professional licensure)

    • Increase in employment outcomes

    • Data Sources:

      • Self-report survey data from veterans/military

      • Public use data sources


    Beginning with the end in mind what we know about performance measurement for vmf programs

    Public Use Data Sources of Interest

    • Demographic Data on Population

      • Veterans Population (VETPoP)

      • DMDC (DOD) – demographics report

  • Public Use Survey Data

    • Health (Wellness)

      • Survey of Army Families (SAF)

      • Air Force Community Assessment Survey

      • National Survey of Veterans


  • Beginning with the end in mind what we know about performance measurement for vmf programs

    Public Use Data Source of Interest

    • Public Use Survey Data (Cont.)

      • Health (Behavioral Health)

        • Survey of DoD Health Related Behaviors

        • MHAT Screenings

        • Sexual assault survey

      • Employment

        • VBA Vocational Rehab Survey

      • Homelessness

        • VA/DoL homeless counts


    Beginning with the end in mind what we know about performance measurement for vmf programs

    Administrative Data Sources of Interest

    • Administrative Data

      • Homelessness

        • DoL VETS data on HVRP data

      • Education

        • Data on GI bill usage (VA)

        • Vocational Rehabilitation data

      • Employment

        • Transition Assistance Program data

        • Local data sources from DVOPs, LVERs, etc.


    Tools and resources for performance measurement

    Tools and Resources for Performance Measurement

    http://www.nationalservice.gov/resources/npm/home


    Equal justice works

    Equal Justice Works

    Kathryn Gravely

    Program Manager

    Kerry O’Brien

    Director of Federal Programs


    Equal justice works overview

    Equal Justice Works Overview

    • Mission

      • Equal Justice Works is a 501(c)(3) nonprofit organization dedicated to creating a just society by mobilizing the next generation of lawyers committed to equal justice 

    • AmeriCorps Program

      • 40 full-time lawyer members in 10 states. Most members serve for two years

      • 360 minimum-time law student members in 48 states. Most members serve 300 hours in the summer

      • Placement sites are legal aid organizations and courts working with low-income residents, including veterans


    Equal justice works veterans theory of change

    Equal Justice Works: Veterans Theory of Change

    Summary

    Need: Veterans have complex legal problems that keep them homeless and in poverty

    Intervention: Lawyers and law students, in conjunction with veterans service infrastructure, resolve legal problems

    Evidence: 2012 Evaluation, Studies show that those with lawyers fare better than those without

    Outcome: Lives of veterans are improved, increased income, access to health care, better housing


    Equal justice works veterans theory of change1

    Equal Justice Works: Veterans Theory of Change

    • Need

      • Low-income and homeless veterans face complex legal problems related to disability benefits, fines, warrants, housing, employment, debt, and family issues

      • Lawyers are uniquely equipped to resolve such problems

    • Data Documenting Need

      • 2011 V.A. CHALENG report; Research articles on veterans & need for lawyers; HUD, HHA and Veterans Benefits Administration statistics; PTSD and TBI studies (e.g. 3 of top ten unmet needs of homeless vets are legal needs – CHALENG)


    Equal justice works veterans theory of change2

    Equal Justice Works: Veterans Theory of Change

    • Intervention

      • Lawyers and law students will resolve the legal problems of low-income and homeless veterans, leading to improved lives: higher incomes, better health, safe and secure housing and family stability

    • Intended Outcome

      • Improve the lives of veterans

    • Evidence-basis for Intervention

      • Impact of 2010-2012 Veterans Pilot Project

      • Studies that lawyers make a difference


    Equal justice works outputs vs outcomes

    Equal Justice Works: Outputs vs. Outcomes

    • Overarching Goal

      • Through legal assistance, improve the lives of veterans

    • Outputs: Amount of Service Provided

      • Example: Assist 30 veterans with benefits claims

    • Outcomes: Reflect the Changes that Occur

      • Example: 20 veterans win benefit cases, providing access to health care and $360,000 in additional yearly income


    Equal justice works outcomes

    Equal Justice Works: Outcomes


    Equal justice works data collection

    Equal Justice Works: Data Collection

    • What we could say in 2010

      • 1,500 veterans served

      • Compelling stories about individual cases

    • What we could not say in 2010

      • Outcomes of cases

      • Impact on veterans served

      • Specific information on veterans served (e.g., demographics, legal issues addressed, etc.)

      • Which sites were performing well/needed assistance to meet goals


    Equal justice works data collection1

    Equal Justice Works: Data Collection

    • Today, we can say:

      • 1,500 veterans served

      • Members completed 226 thorough case evaluations

      • Filed 91 disability benefits petitions or appeals

      • Thus far have won 29 of these cases, securing $642,668 in back benefits and $430,933 in yearly benefits for their veteran clients

      • 114 benefits cases are still pending

      • Successfully able to remove a barrier to employment for 127 veterans served

      • Demographic information specific to veterans served (e.g., era of service, age, gender)

      • Number of veterans served for each particular legal issues, both on macro and micro levels


    Equal justice works data collection2

    Equal Justice Works: Data Collection

    • Today, we can say:

      • Fines or other penalties waived or reduced for 78 veterans served, totaling $36,771

      • Assisted 19 veterans in getting their criminal matters dismissed or expunged

      • Assisted 118 veterans in repairing their credit

      • Of the 1,416 veterans served, 740 were homeless or at risk of becoming homeless, and 739 were veterans with a disability

      • 77% were male and 23% were female—meaning that Fellows served a high number of female veterans, as 10% of veterans in the U.S. are female


    Equal justice works data collection3

    Equal Justice Works: Data Collection

    • Today, we can say:

      • 20% of those receiving services served in the Vietnam era, and 23% served in Iraq and Afghanistan

      • The remaining veteran clients served in all other eras of service, including pre-Vietnam, the Gulf Wars and during peacetime

      • The majority of veterans served were either between 26 and 35 years of age or 55 and older (216 and 420 served, respectively)


    Equal justice works data collection4

    Equal Justice Works: Data Collection

    How?


    Equal justice works data collection5

    Equal Justice Works: Data Collection

    • 2011

      • Conducted interviews and focus groups about outcomes

    • 2012

      • Implemented new optional measures early 2012

      • Brought supervisors together in person – October

      • Outside evaluation completed – December

    • 2013

      • Data resulted in stronger application for funding – Feb

      • Implemented refined measures & made them mandatory in March 2013

      • Report submitted May 1 substantially better


    Equal justice works data collection6

    Equal Justice Works: Data Collection

    • Collection Format:

      • Was: Custom-built web application

        • Cost an hourly charge every time data needs changed

        • Was getting expensive

        • Output was Excel spreadsheet

      • Now: Google Drive - Forms

        • Program manager and coordinator designed, refined

        • Output is Excel spreadsheet

        • Drive.google.com – Create “Forms”


    Equal justice works data collection in google forms

    Equal Justice Works: Data Collection in Google Forms


    Equal justice works great stories

    Equal Justice Works: Great Stories

    • Equal Justice Works AmeriCorps Legal Fellow Rochelle Richardson

    • Homeless Persons Representation Project in Baltimore, MD

    • Many years experience as an employee of the VA

    • Reexamine the case of Mr. A, a homeless veteran with a service-connected mental disability whose claim for veteran's disability benefits had already been denied

    • Rochelle obtained records that had not previously been reviewed and petitioned to reopen the claim and submit further evidence

    • The petition was successful & the claim was granted in March 2013

    • The client will receive a retroactive one-time payment of $50,000 and $3,000 a month in benefits


    Equal justice works lessons learned

    Equal Justice Works: Lessons Learned

    • Define intended outcomes early

    • Align reporting with what funders require

    • Have performance measures and data collection tools ready at the beginning of program year

    • Have a “steering committee” of on the ground experts to consult

    • Keep refining and improving measures to more accurately reflect program impact


    Equal justice works next steps

    Equal Justice Works: Next Steps

    • Figure out how to track and capture long-term outcomes

      • Example: Did increased income result in better housing, better health, better outcomes for children of veterans served?

    • Comparison Evaluation

      • Compare veteran outcomes in our program with national statistics or other programs

    • Big Vision

      • Randomized control trial


    Performance measurement for veteran and military family programs re cap

    Performance Measurement for Veteran and Military Family Programs: Re-Cap

    • Performance Measurement:

      • Tracks outputs and outcomes based on the goals of your program to meet community needs

        • Outputs:

          • Amount of service provided (e.g., veterans) through the service intervention of your program

        • Outcomes:

          • Reflect the changes that occur as the result of the services provided (e.g., veterans employed)


    Questions

    Questions???

    Dr. Chris Spera

    CNCS

    Kathryn Gravely

    Equal Justice Works

    Kerry O’Brien

    Equal Justice Works


    To continue this discussion and others

    To Continue this Discussion…and Others

    Veterans and Military Families

    Knowledge Network

    www.nationalserviceresources.org/veterans

    http://bit.ly/VetsNet


    Join us for the next webinar in this series

    Join Us for the Next Webinar in this Series

    June 12, 2013

    • Making a Difference: What We Know about Using an Evidence-based Approach

    • Register: http://tinyurl.com/c8gp8ll


    Thank you

    Thank you!

    Please complete the

    post-webinar survey


  • Login