Inclusion Outcomes and Indicators of Success

1. Inclusion Outcomes and Indicators of Success Holly Matulewicz Institute for Community Inclusion, ICI (617) 287-7640 Holly.Matulewicz@umb.edu Lucy Bayard National Service Inclusion Project, ICI (617) 287-4355 Lucy.Bayard@umb.edu

3. Training and Technical Assistance

4. Objectives: To develop an understanding of: Indicators of successful inclusion Addressing program issues with data- driven management Measuring constructs Introduction to quantitative measure Introduction to qualitative measures

5. What is inclusion? Inclusion means that all people, regardless of their abilities, disabilities, or health care needs, have the right to: Be respected and appreciated as valuable members of their communities Serve as a member or volunteer in Senior Corps, AmeriCorps or Learn and Serve America programs Work at jobs in the community that pay a competitive wage and have careers that use their capacities to the fullest Participate in service learning opportunities with peers from elementary school through college and continuing education

6. What are indicators of successful inclusion?

7. What are indicators of successful inclusion? Buildings and programs are accessible Interviews, meetings, events and social gatherings are held in accessible locations Individuals are asked about their experience and satisfaction Individuals evaluate the effectiveness of products and strategies Materials available in alternative formats such as large print, electronic, Braille etc.

8. What does successful inclusion look like? Members in leadership roles: In addition to her service, Anna, an AmeriCorps member, serves in a leadership role on Nebraska’s InterCorps Council, which connects all AmeriCorps programs in the state. She conducts peer-grant reviews, and plans events and statewide service days. Justen has been an AmeriCorps*VISTA for two years and, in his third year, has become a VISTA Leader in his Corps. He provides support to 32 members and fulfills the goals of the organization.

9. Presentation Overview Addressing program issues with data-driven management Measuring constructs Introduction to quantitative measures administrative data surveys Introduction to qualitative measures 1:1 Interviews focus Groups field Observation

10. Part I. Addressing program issues with data-driven management

11. What is “data-driven” management Integration of data into your management practices Setting measurable goals & measure progress towards them Setting benchmarks or standards Using data in presentations to staff, funders Why is it useful? Make decision based on evidence not instinct, assumptions, or perceptions Have more information to use for analyzing issues / developing solutions Helps managers and funders see big picture, accountability for outcomes Helps identify trends over time Provides benchmarks for staff

12. Using data-driven management How am I already using this today? Do your goals have measurable outcomes? What data do you use to measure progress? How often do you review data for: Member/ volunteer level outcomes Agency-level outcomes State-level outcomes How often do you share these data with Members / volunteers, staff, agency upper management, funders How can I do more in the future?

13. How many programs can answer these questions? Using the data you collect now – can you report on: # of members / volunteers in each program # of partnerships made between National Service and Disability organizations # of applications for National Service of persons with disability # / Type of recruitment efforts to disability community Disability Organizations – type / volume of info shared about National Service opportunities Change in these numbers over time

14. Strategies for becoming “outcomes driven” Focus on data that matter to you Nurture the “inquisitive mind” Help others see the benefits of using data Build systems / procedures for enhancing data quality

15. Part II. Measuring constructs

16. Building Blocks of Data-driven Management Data is not as scary as people may think! To gather data efficiently and effectively – follow some basic steps: Identify the “construct” to measure. Conceptualize the construct. Operationalize the definition. Develop method for: collecting, entering, analyzing, and reporting these data.

17. Step 1. Identify the construct to measure Looking at your member / volunteer, staff, and agency outcomes: What questions do you want to answer? What information is needed to answer the question? Classifying units of analysis (say individuals) into categories (satisfied with course not satisfied with service experience). Often times we want to “measure” things that in and of themselves are intangible in the social world (“satisfaction” with course, “quality of life,” etc).

18. Step 2. Conceptualize the construct Process of specifying what we mean when we use particular terms. Begin by clarifying what we “mean” by a concept. Example: “Quality of experience” among members / volunteers. What does that “mean” to those interested in measuring it? What does that look like? What are examples of it? Produces an agreed upon meaning for a concept for the purposes of research.

19. Step 3. Operationlize your definition Things like “satisfaction with life” or “fear of crime” are hard to measure directly, so we have to make inferences. Process of defining specific ways to infer the occurrence of specific phenomena. Indicators are observations we think reflect the presence or absence of the phenomena to which the concept refers. How do we “know it when we see it” or “when someone experiences it”

20. As you develop your measures Good research / evaluation strives for both reliability and validity. Applies to qualitative and quantitative measures. Reliable: Using your method – could others replicate your research and get similar results? Valid: When operationalized – will your measure “truly show” the concept you want to study?

21. Step 4. Select a data collection strategy Once you have gone through these steps: Are you already gathering these data? If not - select a data collection strategy Today we will briefly cover 5 strategies to collect data: Quantitative: case record abstraction and surveys Qualitative: 1:1 interviews, focus groups, field observation Meant as introduction to key tasks – not exhaustive summary!

22. Two kinds of data: Qualitative and Quantitative Data come in many shapes, sizes, and formats. Distinction is between numerical (quantitative) and non-numerical (qualitative) data. Different data will be needed to: Answer different types of questions Measure different kinds of outcomes These two types of data Require different data collection and analysis techniques. Both are useful and valuable tools. Each have advantages and disadvantages.

23. Qualitative Vs. Quantitiave Data Most all data start out as initially a qualitative measure - must be quantified so that the researcher can perform statistical analysis. Much controversy about whether using qualitative or quantitative methods and data will “prove the point” better.

24. Reporting on your data Determine your audience and craft report accordingly. Who will hear or read this report - how much background knowledge do they have? What questions will they bring to the table? What topics are of most interest to them? What do you want them to “take away” from the findings Present your question or point of evaluation clearly for your audience. Use jargon / lingo appropriate to the audience: Same data may be packaged differently. Identify themes, recurring ideas, or common experiences Ensure the report is true to the data – not just highlighting those supporting your ideas Map out key findings visually (charts / graphs) Clearly state your data collection processes. Methods transparent: audience should be able to use your process to replicate results. Point out any qualifications or conditions (shortcomings).

25. Part III.Introduction to quantitative measures

26. Quantitative Data In numerical format naturally monetary values, counts, dates Coded / assigned numerical values for analysis placement= 1, non-placement=2 coding into categories with numerical representations (e.g. 1 = yes, 2 = no). Used for: Describing information in aggregate, identifying trends overtime, quantifying outcomes, conducting statistical analysis Able to be analyzed using statistics Examples include: # clients served at agency, # staff, % of staff with BA degrees, % of clients placed in jobs, agency cost per client served

27. Advantages & disadvantages of using quantitative data Advantages Aggregate large volume of data Numbers can be “persuasive” Track trends over time Measure relationship between different variables Disadvantages Training / expertise required to collect, enter, and analyze these data May not shed light on the “whole story” or the “why” of a situation Participants may feel limited to preset response categories

28. Administrative Data or Case Record Abstractions Collection of existing data from case records or files: “Abstracting” key variables from the data for reporting / analytical purposes Because the data are not collected for research purposes – files may have missing data or vary in how items were recorded Unobtrusive research study social behavior without affecting it

29. Value of Record Abstractions / Administrative Data Minimal costs, effort to gather these data – no burden on participants Provides a snapshot of key indicators for your state or agency Gives you ability to aggregate these data for snapshot on progress towards goals / outcomes: Recruitment: # or types of agencies tapped for recruitment partnerships, # or type of sites where applications were distributed WEBBERS on members / volunteers: gender, age, race, education Agencies: # and type of service opportunities members / volunteers placed in

30. What is a survey? Way to collect standardized information from large group of individuals. Collection of data from a scientifically selected group of people. Results can be representative of a larger population. Data collected are used to address specific issues. A standard set of procedures are followed.

31. Advantages and Disadvantages of Surveys Advantages Collecting original data on population too large to observe directly Results can be generalizable to whole population (when using scientific sampling methods) Paper / web give respondents the flexibility to return data at their convenience Disadvantages Can be extremely costly to conduct Item non-response and unit non-response Accounting for sampling bias (based on mode), can leave out some members of the population (reading level, non telephone household, non-English speakers, persons with disability)

32. 4 Modes of Survey Administration Self-administered: Mail Self-administered: Web Interviewer Administered: Telephone Interviewer Administered: In Person

33. 1. Mail Surveys Most common mode of survey data collection Low in cost Response Rates generally low – need multiple waves of follow-up Used a great deal in business surveys when directed at specific groups (such as members of professional organizations) Who does it exclude?

34. 2. Web Surveys New technology, seen most prevalently in convenience samples High costs associated with programming, yet once programmed: data available immediately structure quex. In such a way to eliminate item non-response (benefits / drawbacks of doing this) Respondents can answer at any time, like paper instrument Response options can be personalized based on previous responses Currently still LARGE bias in general population using web mode alone, therefore not recommended (alone) for general population study

35. 3. Telephone Surveys Most large-scale surveys in the US are conducted by telephone using CATI - improves the quality of the data collection Must be tested for correct routing / branches of Qs Avoids an important error - omissions! Can increase cooperation rates. Faster, less expensive than in-person interviewing. Who doesn’t it reach? When might this be a problem?

36. In-Person Surveys(Interviewer Administered) Presence of interviewer may have effects. Increase in cooperation Possible to get immediate clarification on issues in the instrument Possible for bias because of interviewer presence High quality of data - training the interviewers in a classroom like environment Good interviewing techniques stressed Professionalism Avoiding bias Why might this mode not get used as often?

37. Regardless of mode: Use Advance / Cover letters Key Components: Explain the purpose of the survey Organization sponsoring the survey & any relevant endorsements or supporters Lets person know you will be contacting them (or they may contact you) with any relevant details about items needed for survey Provide details on deadlines or submission requests Write from reader’s perspective: “Why should I participate?

38. Schedule of survey data collection Set up your calendar with mail dates Identify total field period (start to finish) Allow sufficient time to Prep mailings Recruit, hire, train interviewers Design / test web survey Process survey returns / enter data Goal: work backwards from end goal or deadline

39. Preparing to field your survey If interviewer-administered (on phone or in person) you must hire interviewers and supervisors. Train them on your survey Ensure they have basic interviewer training Specify DC schedule, QC rates, production rates, and response rate expectations. For mail surveys, training staff on schedule, receipt and follow-up procedures For all methods: developing QC process check on completion of work, including collecting and editing documents.

40. Once the quantitative data are collected … Once the data have been: quality checked, edited, and entered you can begin your analysis! Your analysis should focus on answering the questions you posed when designing your data collection forms (abstractions or surveys). Spreadsheets may be useful to you for Simple entry procedures Few case records Ease of reporting Disadvantages of spreadsheets? Databases may be useful to you for Simple entry, once entry form is designed, Minimizing entry error Ability to: link several datasets, program reports into database / create on-line for field access Disadvantages of databases?

41. Part IV. Introduction to qualitative measures

42. From survey to ethnography... This model of data collection allows for more freedom … not making the respondents feel “boxed in” to prescribed answer categories. Using a standardized format implicitly assumes that all respondents will understand and interpret the questions in the same way Structure can limit researchers ability to gain in depth knowledge about an issue

43. Qualitative data Qualitative data are non-numerical Used for: Examining social world through stories, images, and experience Probing more deeply into constructs, examining the “how” or “why” types of questions Examples include: Transcripts from 1:1 or group interviews Observations made in the field Pictures, texts

44. Why aren’t qualitative data used more? Capturing and analyzing qualitative data sets has been a tough business. Extremely costly process, quite time consuming, often necessitating small sample sizes. “Numbers” can be perceived as more persuasive.

45. 1:1 Qualitative interviews Interaction between participant and interviewer where interviewer has “general plan” of inquiry – but not set questions No specific order of questions Interviewer must be well trained, very knowledgeable in subject matter (for probing) Essentially a “conversation” but participant does 95% of the talking

46. Advantages and disadvantages of 1:1 qualitative interviewing Advantages Participants share info in 1:1 format may not share in group Allow participant to explore concepts more freely / fully Researchers not limited to script or preset response categories Great for exploratory work where you may have limited info on topic Focus on verbal and non-verbal cues. Disadvantages Relies heavily on skill and knowledge of interviewer Costly to implement – per interview costs may limit sample size Due to small number, limits to generalizability Large volume of data to transcribe / analyze

47. Focus groups Group interviews - as they are like in-depth interviews Guided discussion on topics of interest Purpose is to explore rather than describe or explain in a definitive sense Group of 7-12 people too atypical to generalize to whole population Very flexible form of DC, allow participants to frame answers and construct meaning as they wish Examples: member / volunteer service experience: successes, challenges application experience: how heard of opp, why appealed, app process retention issues: why left service, what can be changed agency partnerships: quality of service provided by members / volunteers

48. Advantages & Disadvantages of Focus Groups Advantages Socially oriented research method Flexible – group may raise topics researcher didn’t foresee or anticipate Speedy results Low in cost Disadvantages Less control than individual interviews. Tendency to produce “group think” where people may not readily express ideas that deviate from group’s. Data can be difficult to analyze. Difference between groups can be troublesome. Moderators must be skilled and discussion must be conducted in a conducive environment. Groups are difficult to assemble.

49. Field Observation Methods of collecting data on people, likely in their natural settings. People from somewhere going somewhere else & sharing what they find “Informant” who gives you your data (like a narrator) Participant observation: performed by those who take part in the activities they observe. Gain “verstehen” by immersing themselves in the daily lives of those they study. Non-participant observation: made by an observer who remains as aloof as possible from those being observed.

50. Using Field Observation Decide on a topic where field observations are appropriate Identify your research questions and constructs to measure Can include narrative and quantitative measures Examples can include: observation of worksite or agency observation of member / volunteers with those they serve Attending recruiting events – observing candidates and recruiters

51. Advantages and disadvantages of field observation Advantages Direct observation, rather than descriptions or interpretations (via interviews) from participants’ bias / perspectives Data can richly supplement other sources of info Disadvantages Disruption of natural setting Time consuming / labor intensive Relies heavily on skills of observer Can rely on honesty – level of disclosure of informants Can have ethical dilemmas (participant vs. non-participant) Act of study can change behavior of those observed

52. How can I assess if my program is inclusive and accessible? An accessibility checklist provides guidelines for assessing your program(s) to help ensure : compliance with the law (Section 504 of the Rehabilitation Act and Title II of the Americans with Disabilities Act) how to create an environment that makes people with disabilities feel welcome how to design programs and services so that people with disabilities can fully participate

53. Tips for conducting accessibility checklist at your organization / program Involve Program Directors, service site supervisors, and all relevant staff at site / organization Provide an opportunity for members, including members with disabilities, to provide feedback and share their experiences regarding accessibility and inclusion Be willing to collaborate with disability organizations in your community to access resources and assistive technology Assessing your program and improving areas in need may take time, so it is important to keep it as a priority and be patient!

54. Thank you!