Transition Assessment and Goal Generator (TAGG): A New On-Line Assessment

1. Transition Assessment and Goal Generator (TAGG): A New On-Line Assessment Jim Martin & Amber McConnell University of Oklahoma DepT of Ed Psychology Zarrow Center

2. Quality Transition Education ProducesGreater Outcomes Illustration of two high school graduates. One on large horse and student suited in armor. This student had quality transition education while in school. The other student in on a stick-horse going in the wrong direction. He did not have quality transition education. Transition Education Begins With Transition Assessment

3. Agenda • Purpose of Transition Assessment • Use results to build postsecondary and annual transition goals • Questions to Ask to Pick a Transition Assessment • Is this a good transition assessment to use? • Importance of Annual Transition Goals • The TAGG

4. Purpose of Transition Assessments

5. Questions Students Need to Use Results to Build Postsecondary Goals Postsecondary Goal Questions • Upon graduation from high school: • Where do I want to live? • Where do I want to work? • Where do I want to learn to do the job I want?

6. Questions for Annual Transition Goals • What do I need to learn now: • To live where I want after graduating from high school? • To do the job I want after graduating from high school? • To learn where I want after graduating from high school?

7. A Few Questions to Ask to Help Pick a Transition Assessment

8. Validity Evidence Test developer needs to provide basic validity evidence User needs to examine evidence and assessment to determine if it is appropriate for students and intended purpose

9. Importance Factor The more important decision, the more validity results need to be How valid and reliable are the results used to make transition planning decisions? Often not easy to find out Duties . . .

10. Sources of Validity Evidence Item Development Items Structured and Organized Internal Reliability Stability of Scores (Test-Retest) Fairness Validity Match with other similar assessments: Concurrent Validity Purpose: Does it fit for your uses?

11. Development • How were the items developed? • Did the authors just sit down and make them up? • Did the authors ask knowledgeable sources? • Did the authors use findings from research with students with disabilities? • Are they based on what students need? • Are items developed, then clustered into similar groups? • Are cluster definitions defined first then items developed?

12. More Item Questions • How are the items structured and organized? • Based on common sense categories? • Based upon statistical analyses of user responses to verify theory behind assessment?

13. Reliability • Does evidence of acceptable internal consistency scores exist? • Cronbach’s alpha (.7 or higher) • Does evidence exists that when same users first completed the tool, then completed it again several weeks later, the results are similar? • Does evidence of good test-retest reliability exist?

14. Validity • If the assessment fair? Does evidence exist that no major differences exist across • Race • Ethnic groups • Gender • SES • Does evidence exist that the assessment results match those from another assessment that examines the same concept? • Correlation between Weschler IQ and Standford Binet IQ results

15. Importance Factor The more important decision, the more validity evidence needed Often not easy to discover validity evidence Determining validity is up to the test developers and users

16. Example of What Can Be Learned • The Career Clusters Interest Survey • Free • Supposed to Identify Interests by the 16 career clusters • Easy to use and score • Many educators use it • So, study by Prime and Tracey (2010) . . . . ..

17. Study Results • The CCIS is reliable, but not valid • It measures well time after time, but • The results are very limited in terms of what is being measured • Doesn’t match-up well with another assessment that measures the same concepts • “Its usage could provide a severely restricted set of options . . .”

18. Acceptable Degree of Validity Chart of IEP stages by the level of validity evidence – none, low, moderate, or high

19. The TAGG Transition Assessment and Goal Generator Developed with a Grant from the National Center for Special Education Research and Zarrow Center Funds

20. TAGG Overview An on-line transition assessment for secondary-aged students with disabilities, their educators, and parents designed to provide non-academic annual transition goals to enable students to learn skills to increase likelihood of employment and further education

21. TAGG Will Provide • The TAGG will provide • A graphic profile • Written summary to copy and paste into PLEP section of the IEP • Listing of relative strengths • Listing of relative needs • Annual transition goal suggestions matched to needs

22. Annual Transition Goal Questions the TAGG Can Answer • What do I need to learn now: • To do the job I want after graduating from high school? • To learn where I want to learn after graduating from high school?

23. How Were the TAGG Items Developed? The research team found research studies that identified behaviors of former high school students with disabilities associated with post high school employment or further education The research team developed 10 construct definitions Items developed from constructs Over three years, almost 1,500 students, 130 educators, and 800 parents from across the country completed the TAGG.

24. Initial Structure: Ten Initial Constructs Knowledge of strengths and limitations Actions related to strengths and limitations Disability awareness Employment Goal setting and attainment Persistence Proactive involvement Self-advocacy Supports Utilization of resources

25. Sample Construct Definition Disability Awareness Successful individuals know they have a disability and can express needs to others in a non-stigmatizing manner. Individuals demonstrate knowledge of the disability and can express positive and negative aspects. They express information such as how the disability affects life and what supports are needed and legally allowed to compensate in various situations. The student needs to be able to place the disability within the context of his or her life and is not defined by the disability.

26. Year 1 and Efforts to Build TAGG Structure Users completed the TAGG Applied various factor analyses statistics Went from 10 constructs to 8 Went from 75 items to 34 Construct structure confirmed by second study with year 2 data

27. Professional and Family Constructs After Year 1 Analysis Stayed Strengths and Limitation Disability Awareness Persistence Interacting with Others Goal Setting and Attainment Employment Student Involvement in IEP Support Community Dropped Action Related to Strengths and Limitation Utilization of Resources TAGG-P: (c2=1043.62, df=499, RMSEA=.058, CFI=.92, TLI=.91, RMSR=.0597) TAGG-F:(c2=862.74, df=499, RMSEA=.057, CFI=.91, TLI=.90, RMSR=.058)

28. Student Version Constructs After CFA After CFA Constructs Strengths and Limitations & Support Community Disability Awareness Persistence Student Involvement in IEP Interacting with Others Goal Setting and Attainment Employment Dropped Construct Action Related to Strengths and Limitation Utilization of Resources Combined Constructs Strengths and Limitations Support Community TAGG-S: (c2=819.00, df=505, RMSEA=.047, CFI=.89, TLI=.88, RMSR=.064)

29. Internal Reliability • Does evidence exist that across the TAGG various items respond in the same way? • Each TAGG version has great overall internal consistency and satisfactory subscale consistency (ranging from α =. 89 to α =. 95)

30. Test-Retest Reliability • Does evidence exists that the results of same users completed once, and then again several weeks later, produce similar scores? • 14 weeks after the first TAGG completion a large correlation was found with the second administration for all three version • .80 for professional TAGG • .70 for family TAGG • .70 for student TAGG

31. Fairness Validity Evidence • Do differences exist by gender and racial groups? • No overall difference • Differences with employment construct • Do differences exist by SES? • No overall difference by SES

32. Match Across Versions Validity Evidence • How closely do the different versions assess? • TTAGG has medium correlations, which is excellent for an assessment of this type

33. Concurrent Validity Studies • Match with TAGG and AIR Self-Determination Assessment • Underway – data collected • Match with TAGG and Employer Identified Trait Assessment • Underway – data collected

34. You Decide . . .Does the TAGG Fit Your Use? Based on the given validity evidence Does the TAGG has ample evidence to support how you want to use the results to build the PLAAFP and annual transition goals?

35. On-Line TAGG The on-line TAGG will go live to the public before the end of this year It will look something like this: http://zarrowcenter.ou.edu/taggp4/

36. TAGG Professional Version sample questions for the strengths and limitations construct. The student told someone what he or she does well. The student told someone what he or she has trouble doing.

37. Sample Disability Awareness construct questions. The student expressed the type of support or accommodations needed for his or her disability.

38. Sample items for the persistence construct. The student views not giving up in school as important. The student keeps working to achieve a goal, even when it becomes hard.

39. Sample items for the Interacting with Others construct. 1. The student successfully participates in small groups to complete projects.

40. Sample items for the goal setting and attainment construct. 1. The student creates short-term goals to attain long-term goals. 2. The student adjusts plans to attain goals if they do not work.

41. Sample items for the employment construct. The student expresses wanting a job. The student had an unpaid job, such as working for a family member. The had a paid job.

42. Sample items for the involvement in the IEP construct. The student told the IEP team his or her postschool goals The student discussed his or her present level of performance at the IEP meeting.

43. Sample items for the support community construct. The student accepts help from support people when offered. The student seeks assistance from community agencies.

44. Disability Awareness Profile Sample disability awareness profile bar graph of results for the student, parent, and teacher TAGG versions. The graph is present on a 0 to 9 stanine profile.

45. Combined Score Profile Sample profile bar graph of overall results for the student, parent, and teacher users. The graph is present on a 0 to 9 stanine profile.

46. Greatest and Relative Strengths Summary of greatest strengths and areas of relative strengths, along with area of greatest need by student, family, and teacher user.

47. Areas of Greatest and Relative Need Summary of greatest need, areas of relative need, and a summary of the overall TAGG results. Each can be copied and pasted into the IEP.

48. Summary Statement for IEP Chad Bailey’s skills were assessed using the TAGG, a norm-referenced assessment with research-based items know to be associated with post-school employment and education. Compared to similar students Chad’s scores are average. Results indicate greatest strengths are in the areas of Goal Setting and Attainment. Chad’s relative strengths include Disability Awareness and Student Involvement in the IEP. Greatest needs are in the area of Strengths and Limitations, with Employment being a relative need.

49. Suggested Annual Transition Goals To prepare for success in employment, the student will write an essay describing three situations where the student used his or her strengths with 90% accuracy in the areas of grammar and context by the end of the essay writing unit. Listing of Common Core Standards match to an annual goal. These are presented by grade for difference English benchmarks.

50. On-Going Studies Continue additional validity studies, including SES and TAGG results Follow group of students from high school into adult life Determine relation between high school TAGG profile and adult outcomes Establish criterion levels for constructs Produce profile with prioritized annual transition goals based upon student’s profile and norm group outcomes