Creating A Multiple Measures Placement System

1. Creating A Multiple Measures Placement System An Exercise With Ron Gordon & Armand Brunhoeber

2. The Concept • Although test scores may predict failure, they do not necessarily predict success. • Research shows that students’ backgrounds, environments, and personal habits may have more influence on their potential academic success than their residual academic skills. • Find a way to factor that information into the placement decision at testing time.

3. Decision Process • Which Courses • Which Tests • Back Ground Questions • How many • Variety • Select Weight Values • Use negative weights? • How much possible total weight

4. Planning Flow • Who • Discipline faculty • Testing staff • Counseling • IT • Others – Research?

5. Planning Flow • What • Which courses in which disciplines • What information can students provide at the time of testing? • Which questions will be used in each discipline? • How much total value should additional measures have?

6. Planning Flow • Why • State mandate? • Literature • Improve placement accuracy • Give students better opportunities to learn

7. How Does it Work? • Students answer locally developed background questions during the test session • ACCUPLACER placement rules compute a weight value for the student’s answers based on locally developed weighting rules • The test score, plus a percentage based on the background information is used for placement

8. The Process • Educate faculty and others on how the system functions • Find examples of background questions and edit to fit • Assign weight values to each answer choice. • Write placement rules using the multiple measures editing function

9. Select Questions • Questions must, in some way, relate to student success. • Must be information the student will have at the time of testing • Must be multiple choice • Should solicit behavioral, historical/experiential, and environmental information

10. Limits • Limit total weight so that background information does not allow students to skip a course level • Limit number of questions to a manageable number • More questions adds to testing time • Placement rules can become unmanageable • Answer choices must be mutually exclusive and all inclusive

11. Example • (If Arithmetic, plus all weighted choices is >= 75 OR Algebra, plus all weighted choices is >= 48)AND (Algebra, plus all weighted choices is < 65 OR Algebra Not Taken) AND (CLM, plus all weighted choices is <62 OR CLM Not Taken) Then Placement is Elementary Algebra

12. Example, Continued • If this rule had 5 questions with 4 weighted answer choices each, there would be 80 lines just for weights. • With too many questions, or too many choices per question, rules can become unmanageable

13. Conditional Weights • High School Accomplishments Have Limited Shelf Life • How Much Does it Matter That a 25-Year-old Student Had 2 years of High School Algebra? • Does it Matter That The Same 25-Year-old Student Works for a Surveyor and Uses Algebra Daily?

14. Example • How long has it been since you were enrolled in high school or other formal educational process? • Less than 2 years or still enrolled • 2 to 5 years • More than 5 but less than 7 years • 7 years or more • Use high school data for up to 5 years, experience for more than 5 years.

15. Assigning Weights • Total possible weight should not move student more than one level in either direction • Set maximum possible weight so a student who scores at or above the midpoint of a placement range could move up, but one who scores below the midpoint could not. • Use faculty to select BGQ and assign weight • Guide them

16. Multiple Measures Movement Model

17. Sample Question With Weights • Which choice below best describes you when you read textbooks or other complex information? • I usually need to read material several times before I understand it well -.01 • Sometimes I can understand what I read the first time, but often I must reread it .00 • I usually understand what I read if I take notes or highlight passages. +.01 • I always understand what I read the first time through +.02

18. Think About Uploading to SIS • Multiple measures may change placements, but not scores. • SIS has no place to store multiple measures • To upload multiple measures, the placement (course name) must be uploaded • In ACCUPLACER, course name can be numeric

19. Preparing to Build the System • Assign numeric codes to course names • Determine which tests will be used for each course in each discipline • Create cut score Table • Create a BGQ weight Matrix

20. Building the System • Create Background questions • Assign BGQ to groups • Create branching profiles • Create course groups • Create courses and assign to groups List • Create majors if used • Create placement rules Edit1Edit2 • placement rule sample.pdf

21. Verify • Write most complex rule first • Run verify function in branching profile • Use several BGQ and score combinations to test the placement rule • Compute weighted score for each run • Try to hit cut scores to test for bad weight or answer choice selections

22. Computing the Weight 1

23. Computing the Weight 2

24. Computation • Score is multiplied by 1 plus the accumulated weight. • 85 * (1+.04) = 88.4 • Placement will be based on a score of 88. • Example 2 • 85 * (1+ [-.03]) = 82.45 • Placement will be based on a score of 82

25. Common Errors • Unequal weights between rules • E.G. A response has .01 weight in one rule and -.01 weight in the next rule • Misplaced Parentheses • The multiple measures weights make the rule larger and more difficult to visualize • Misuse of AND/OR • Misuse of arithmetic operators • Wrong answer choice in rule line

26. Troubleshooting • From the score report, determine what the student’s weight should be from the BGQ responses • Using the weight, compute the weighted score • Determine what the placement should be • Examine the appropriate rule for errors

27. Creating a Multiple Measures Placement System An exercise with Ron Gordon & Armand Brunhoeber Thank you for not throwing things at the presenters