Deciding to Adopt, to Adapt or to create

1. Deciding to Adopt, to Adapt or to create Presented by: Robin Anderson 2011 JMU Summer Assessment Institute

2. Today’s Session • The major role of objectives • Adopting instruments • Creating instrument • Adapting instruments

3. The role of objectives Objectives DRIVE the instrument selection or creation process They are the all important first domino They must be well-written

4. Brief Overview of Writing Objectives • Objectives refer to expected or intended student outcomes • A statement that describes what the learner (student, program participant) will know or be able to do after completing the instruction (course, program). CARS website

5. Brief Overview of Writing Objectives • Objectives should specify : • Behavior or activity to be performed by the student • Conditions under which this behavior takes place • Minimum level of acceptable performance

6. Brief Overview of Writing Objectives • EVERY objective MUST be measured, though not necessarily all on the same assessment schedule or with the same method. • Data must be available to show evidence that each objective has been measured • Data can come from standardized objective measures, performance assessments, checklists, or other methods

7. Measuring Objectives • One way to measure your objectives is by adopting an existing instrument in its entirety. • Advantage: Saves time • Items already written • Instrument pilot tested • Reliability and validity information already gathered on other samples

8. Using Existing Instruments Before reviewing existing instruments: • Have specified objectives that you want to measure • If the objectives are developmental or affective in nature, you must have a theoretical basis or definition for the construct you are trying to measure

9. Example: Career decision-making • Understand career decision-making concepts and career theories. • What is the construct of career decision-making? • Give a theoretical and operational definition • Are there stages? • If yes, how are these stages defined? • Is there more than one theory that defines career decision-making? • If yes, which one are you referring to or teaching?(Alignment!)

10. Example • Reviewing the literature is good way to help you define your construct and find instruments. • Super’s theory of self concept and developmental stages is one theory of career decision-making. • Career Development Inventory • Career Maturity Inventory • In addition to these, there are numerous other theories and instruments

11. Instruments in the Literature-What are they measuring? • Say you came across these instruments in your reading of career-decision making. • Career Development Inventory • Career Maturity Inventory • What do they measure? • There are sources that you can search to answer this question.

12. Searching for Instruments • There are several places you can look for pre-existing instruments. • Instruments may be commercial (for a cost) or noncommercial (published or available from author free) • Both the Career Development Inventory and the Career Maturity Inventory are noncommercial instruments, so let’s start with noncommercial sources.

13. Noncommercial Sources • Databases • ERIC • PsycINFO (example) • Limit search by “Title”, on main search page, or “Tests and Measures”, under the limit category • PsycLIT • Tests in Microfiche • At JMU this is located on the 2nd floor of the library, in microforms section B4 • Again, check your institution’s library • Websites • http://www.scp.rochester.edu/SDT/measures/index.html • http://www.lib.jmu.edu/psychology/psyctest.htm All available through JMU library – check your library!

14. More non-commercial sources… • Directories • Directory of Psychological Tests in the Sport and Exercise Sciences • Directory of Unpublished Experimental Mental Measures • Multiple Vols. available • Tests In Print • Bibliography of educational, psychological, and industry instruments

15. Commercial Test Resources • Reference Books • Mental Measurements Yearbook • Available online through many library website or • http://www.unl.edu/buros/ • Provides publisher and purpose • Tests: A comprehensive reference for assessments in psychology, education, and business • Lists tests and brief description of purpose, population, time, cost and publisher • Test Critiques • To be used with #2 • Reviews of instruments

16. Commercial Test Resources---Yes, more • Websites • http://www.acpa.nche.edu/comms/comm09/dragon/dragon-index.html Brief descriptions of measures and publishers contact information • http://buros.unl.edu/buros/jsp/search.jsp Test reviews online • http://www.ets.org/testcoll/index.html ETS test collection library • http://www.pearsonassessments.com/ Pearson assessments- psychological assessments for use by mental health, medical, and career-guidance professionals

17. Commercial and Noncommercial Resources • Databases • Health and Psychosocial Instruments (HAPI) • Websites • http://www.apa.org/science/faq-findtests.html • Books • Measuring self-concept across the life span: issues and instrumentation • Sex and gender issues: A handbook of tests and measures • Handbook of sexuality-related measures • Handbook of tests and measurements for black populations- 2 vols. • Measures of personality and social psychological attitudes • Measures of political attitudes • Handbook of research design and social measurement

18. So I have objectives and I know where to find instruments…now what? • Examine Instruments • Do they measure your objectives? • Are any suitable for your purposes? • Do they cost money to administer or score? • Are they difficult to administer? • How long do they take to complete? • How many items do they contain? • Is there any reliability or validity information?

19. Example Career Development Inventory

20. Example • Since the Career Development Inventory does not measure our objectives, there is really no need to further examine it. • So what do you do if none of the instruments that you find measure your objectives?  • You can write your own items and create an instrument specific to the goals and objectives of your program.

21. What if I can’t find the right instrument? • You can develop one! • Advantage: • You own it • Matches you objectives! • Disadvantages: • Difficult • Time-consuming • Good items are hard to write • As test developer you have responsibilities to study the measure. • If you are in a hurry, a good measure is not quick • Requires expertise

22. Track two is focused on the development of instruments!

23. There is one more option: Adapt • Possible way to capitalize on advantages of both; however… • You are still obligated to do all you do for a new instrument

24. Example Career Development Inventory