Developing Test Specifications

1. Developing Test Specifications Christine Coombe

2. What are Test Specifications? A test is not a thing in and of itself; it is a thing defined by its impact on the people who take it Known as washback Generative blueprints for test design Plan from which test items and tasks can be produced (Davidson & Lynch, 2002)

3. Who should develop tests? Tests should be written by a group of invested individuals Tests must be tuned to their settings Tests must reflect Desires & beliefs (forces that shape the test content) Resource capabilities available to educators Chief tool of test development is the test specification Sometimes referred to as blueprint, plan, guideline, form, rubric

4. History Not a new concept Term probably derived from industrial concept of a �specification� Earliest mention in educational assessment literature was by Ruch in 1929

5. Good tests��! Good tests involve clear thinking Ultimate goal is clarity Regardless of the use of the test score Good tests should be �iterative, consensus-based, specification-driven� (Davidson & Lynch, 2002) Iterative where there are cycles of feedback over time as test grows and evolves Consensus-based-tests should result from dialog & debate; not from a top-down dictate Specification driven-specs are recipes for tests that foster dialog & discovery at a higher level

6. Davidson & Lynch Model No single best format or magic formula for a spec Innumerable ways to design one Davidson & Lynch model is based on the Popham Model (1978) and has five components General Description Prompt Attributes Response Attributes Sample Item Specification Supplement

7. Section 1: General Description The GD section is the object or focus of assessment Indicates behavior or skill to be tested Statement of purpose or reason or motivation for testing Normally a capsule summary that can be read quickly is best

8. Example of GD The Ss will be able to guess the meaning of certain vocabulary words from context. The texts and words will be of either a scientific, academic or general nature.

9. Section 2: Prompt Attributes Called the �stimulus� attributes in Popham model Component of test that details what will be given to test taker Selection of an item or task format Detailed description of what test takers will be asked to do Directions or instructions Form of actual item or task Isn�t usually long or complicated

10. Example of PA The student will be asked to write a letter of complaint about a common situation. Each student will be given a written prompt which includes his role, the role of the addressee, and a minimum of three pieces of information to include in the complaint letter.

11. Section 3: Response Attributes Part of the spec that details how the test taker will respond to the item or task Often difficult to distinguish from the PA

12. Example of RA The test taker will select the one best answer from the four alternatives presented in the test item. The test taker will mark their answers on the answer sheet, filling in the blank or circling the letter of the best alternative.

13. Section 4: Sample Item Purpose is to �bring to life� the GD, PA and RA Establishes explicit format & content patterns for the items or tasks that will be written from specs

14. Example of SI On a recent flight back home to the UAE, Emirates Airlines lost your baggage. Write a complaint letter to Mr. Al-Ahli, the General Manager, telling him about your problem. Be sure to include the following: Your flight details A description of the baggage lost and its contents What you would like Mr. Al-Ahli to do for you

15. Section 5: Specifications Supplement Optional component Designed to allow the spec to include as much detail & info as possible References or lists of something Anything else that would make the spec appear unwieldy

16. Bachman & Palmer Model Bachman & Palmer (1996) spec divided into two parts Structure of the test How many parts or subtests; their ordering & relative importance; number of items/tasks per part Test task specifications Purpose & Definition of the construct Setting & Time allotment Instructions Characteristics of input & expected response Scoring method

17. Alderson, Clapham & Wall Model Alderson, Clapham & Wall (1995) Model Specs should vary in format & content according to audience Different Specs for Test writer Test validator Test user

18. Alderson, Clapham & Wall Model Test Writers format General statement of purpose Test battery List of components & time allowed for each Test focus Description of sub skills/knowledge areas to be tested Source of texts Where appropriate text materials can be found Test Tasks Range of tasks to be used

19. Alderson, Clapham & Wall Model Item Types Range of item types & # of items Rubrics Form & content of instructions given to test takers Test Validator Specs Focus on model of language ability/construct Test User Specs Statement of purpose; sample items or complete tests; description of expected performance at key levels

20. Problems & Issues Reverse Engineering Creation of a spec from an already existing set of items or test Can be useful when there is a desire to move from a testing context where there is no guidance to one which is spec driven Can serve to stimulate thought and provoke discussion

21. Problems & Issues Item/Task Fit-to-Spec How well an item/task generated by a test spec matches what is described in the spec Level of Language & Generality Spec phraseology must be at correct level of generality Not too general, not too specific Ownership Well crafted test is the product of many minds

22. References Alderson, J.C., Clapham, C. & D. Wall. 1995. Language test construction and evaluation. Cambridge: CUP. Bachman, L.F. & A. Palmer. 1996. Language testing in practice. Oxford: OUP. Davidson, F. & B. Lynch. 2002. Testcraft: A teacher�s guide to writing and using language test specifications. New Haven: Yale University Press. Popham, W.J. 1978. Criterion-referenced measurement. Englewood Cliffs, NJ: Prentice-Hall.