Avoiding the False Negative: Placing Students Into Mathematics Courses According to Their Abilities

1. Avoiding the False Negative: Placing Students Into Mathematics Courses According to Their Abilities Margaret J. Hager, EdD Margaret.Hager@uc.edu University of Cincinnati November 2, 2006 How I found title: My question to committee? Did they have any idea of how I came up with the A/A names?How I found title: My question to committee? Did they have any idea of how I came up with the A/A names?

2. My Motivation Many stakeholders believed the advising and placement systems needed improvement It was important to find out the perceptions of the institution so that recommendations made could be successful Webb�s Chapter �focuses on theory building through a critical evaluation of current efforts in mathematical assessment� (p. 661). CITE Program study (perceptions of participants were sought) Multiple ad-hoc groups over the years approached the subjects of advising and placement. Lately the mathematics faculty had not been paying attention. Lit review formally began in Assessment course WEBB�S Chapter from Handbook of Research of Mathematics Teaching: His chapter focuses on theory building through a critical evaluation of current efforts in mathematical assessment.CITE Program study (perceptions of participants were sought) Multiple ad-hoc groups over the years approached the subjects of advising and placement. Lately the mathematics faculty had not been paying attention. Lit review formally began in Assessment course WEBB�S Chapter from Handbook of Research of Mathematics Teaching: His chapter focuses on theory building through a critical evaluation of current efforts in mathematical assessment.

3. Relevant Research Many studies were quantitative in nature Reliability of placement tests: Bridgeman & Wendler (1989), Johnson (1984), Latterell & Regal (2003), Sawyer (1996), Sworder (1986) Cut-off scores: Abraham (1986), Hughes & Nelson (1990); Morante (1987) Use of Multiple Measures: Armstrong (2000), Askt & Hirsch (1991),Bridgeman & Wendler (1989); Hoyt (1999), Jenkins (1989), Jones (1997); Jue (1993); Morante (1987) Quantitative studies were also reported by Webb and Merriam (researchers) Jue reported switching from mandatory placement to using multiple measures Mandatory placement = high success, Blair, (19__), Armstrong (2000), Askt & Hirsch (1991); Cuneo (1995); Cunningham (1995); Dodson (1986); Mitchel (1989) Schoenberger (1985) Older students benefit more from remediationQuantitative studies were also reported by Webb and Merriam (researchers) Jue reported switching from mandatory placement to using multiple measures Mandatory placement = high success, Blair, (19__), Armstrong (2000), Askt & Hirsch (1991); Cuneo (1995); Cunningham (1995); Dodson (1986); Mitchel (1989) Schoenberger (1985) Older students benefit more from remediation

4. Relevant Research (Continued) Revak, Frickenstein, & Cribb (2000) A small amount of studying prior to taking the placement test resulted in significantly higher placement scores. By its very nature this study adds to the theory. We need to assess mathematics differently than other disciplines.By its very nature this study adds to the theory. We need to assess mathematics differently than other disciplines.

5. Relevant Research (Continued) �An assessment involves a situation, responses to that situation, analyses of the responses, interpretation of the results, and interaction among all of these. . . .� (Webb, 1992, p. 680).

6. Problem Statement Why do so many students place into Preparatory mathematics courses even after multiple years of high school mathematics preparation? Much anecdotal information about S taking Pre-calc in HS and testing into Intro Algebra. Concerning the recommendations: An initial hope the process could be better.Much anecdotal information about S taking Pre-calc in HS and testing into Intro Algebra. Concerning the recommendations: An initial hope the process could be better.

7. Problem Statement (Continued) Are there any recommendations that might be made to the mathematics placement process to help ensure proper placement of students into mathematics courses?

8. Problem Statement (Continued) Guiding Questions: 1. Why do students not practice before taking the mathematics placement test? 2. Why do students who place into a lower level mathematics course often choose to take a higher-level mathematics course? 3. What other measures of student preparedness for mathematics courses are possible in addition to the one placement test score? 4. Should mathematics placement be mandatory? 2. S could/can take any course, there are no computer blocks on courses. Also, many S just need a refresher, not the entire courses (Hassett, et al, (1992) S don�t take the test seriously mainly because they were not made aware of importance of test (Hoyt (1999)) Being bored, she forgot how to study. (Hassett et al.) 3. Only one test score is used. 4. Ultimate goal for ease of A/A, especially since English placement has been mandatory for years.2. S could/can take any course, there are no computer blocks on courses. Also, many S just need a refresher, not the entire courses (Hassett, et al, (1992) S don�t take the test seriously mainly because they were not made aware of importance of test (Hoyt (1999)) Being bored, she forgot how to study. (Hassett et al.) 3. Only one test score is used. 4. Ultimate goal for ease of A/A, especially since English placement has been mandatory for years.

9. Method Student Participants (15): Semi-structured focus groups and individual interviews Current students Students at Orientations Students appealing their mathematics placement 3 focus groups mainly to ease the scheduling of student participants or comfort with their peers. All A/A had some level of involvement in mathematics placement testing process Division chairs (department head equivalent) were ones when Compass was first implemented. 3 focus groups mainly to ease the scheduling of student participants or comfort with their peers. All A/A had some level of involvement in mathematics placement testing process Division chairs (department head equivalent) were ones when Compass was first implemented.

10. Method (Continued) Advisors/Administrators (A/A) (20): Semi-structured individual interviews Faculty and Staff Advisors Placement Test Coordinator Assistant Dean of Student Services Placement Test Proctor Division Chairs

11. Method (Continued) Further Data Collection Unobtrusive Placement Testing Observations Document Collection Compass/ESL Manual & Practice Test packet Orientation & Placement Test Proctor Scripts Various Reports/memos on placement test issues Placement test observations were unobtrusive. Noticed more from proctors than students (Proctors did not read script word-for-word). Data analysis patterns helped with writing the descriptive narrativePlacement test observations were unobtrusive. Noticed more from proctors than students (Proctors did not read script word-for-word). Data analysis patterns helped with writing the descriptive narrative

12. Method (Continued) Data Analysis Individual/focus Group answers coded Triangulated with field notes and documents

13. Results Compass/ESL Process Computer-adaptive Students instructed to use only the on-line limited function calculator Average of 7 questions per testing domain All students began in Algebra domain Students not able to check their work Average time spent was approx. 16 minutes Compass �chosen� in 2000. Computer-adaptive (DESCRIBE stop after 7, go higher, or go lower) Instructions FORBID students to use the scientific version of on-line calculator Some students placed with 5 questions 10/2000 - 4/02 all S began Pre-Algebra domain, then as a test 4/02 - 12/02, 50/50 Pre-Alg and Alg domain. Finally, 12/02 - present, all begin in Algebra domain. (With old test, S took Pre-Algebra test, then had to reschedule higher level test, few did) Range of testing times 3 - 58 minutes. 72% < 20 minutes, 92% < 30 minutes.Compass �chosen� in 2000. Computer-adaptive (DESCRIBE stop after 7, go higher, or go lower) Instructions FORBID students to use the scientific version of on-line calculator Some students placed with 5 questions 10/2000 - 4/02 all S began Pre-Algebra domain, then as a test 4/02 - 12/02, 50/50 Pre-Alg and Alg domain. Finally, 12/02 - present, all begin in Algebra domain. (With old test, S took Pre-Algebra test, then had to reschedule higher level test, few did) Range of testing times 3 - 58 minutes. 72% < 20 minutes, 92% < 30 minutes.

14. Results (Continued) Accuracy of Mathematics Placement Test Majority of the students and A/A believed the results were inaccurate There was confusion among some S about whether results were mandatory. 3 S jumped 2 S mentioned no way to study. We need to do a better job of preparing S to take this test. Of 240 S from 04U, 84% took required course, 16% skipped. All participants agreed Appeal Process was necessary.There was confusion among some S about whether results were mandatory. 3 S jumped 2 S mentioned no way to study. We need to do a better job of preparing S to take this test. Of 240 S from 04U, 84% took required course, 16% skipped. All participants agreed Appeal Process was necessary.

15. Results (Continued) Misconceptions Regarding the Placement Process A common belief of A/A was that students did not need to study before taking the test

16. Results (Continued) Mathematics Placement Appeal Process 46 in pilot, 27 judged, 81.5% successful However, there was no formal process that required students to appeal There was confusion among some S about whether results were mandatory. 3 S jumped 2 S mentioned no way to study. We need to do a better job of preparing S to take this test. Of 240 S from 04U, 84% took required course, 16% skipped. All participants agreed Appeal Process was necessary.There was confusion among some S about whether results were mandatory. 3 S jumped 2 S mentioned no way to study. We need to do a better job of preparing S to take this test. Of 240 S from 04U, 84% took required course, 16% skipped. All participants agreed Appeal Process was necessary.

17. Results (Continued) Six-month retake policy with intervention Most A/A did not realize the six-month rule was in place Most A/A believed it was too long of a time-frame Students were not informed of the need for intervention

18. Results (Continued) Practicing Before Test Most students did not practice (many wished they had) Web site with arithmetic problems http://mathematics.clc.uc.edu/hager/index.html Need of practice packet Time consideration in using multiple measures in using and following up on them. Time consideration in using multiple measures in using and following up on them.

19. Results (Continued) Calculator Use Instructed to use limited function on-line version Students wanted their own hand-held calculator Compass has an approved list of calculators

20. Results (Continued) Multiple Measures Implementation was a concern Time and effort in gathering data Current testing was easy to administer Results were almost immediate Time consideration in using multiple measures in using and following up on them. Time consideration in using multiple measures in using and following up on them.

21. Results (Continued) Mandatory Placement Many A/A believed this would make their advising easier Must keep the placement appeal process Will be pursued if web system can block students from skipping any pre-requisite courses

22. Discussion Guiding questions 1. Why do students not practice before taking the mathematics placement test? Students had no information on what to study They were not made aware of the importance of the test and therefore, did not take mathematics placement test seriously (1) No S reported practicing problems before testing. Most wished they had.(1) No S reported practicing problems before testing. Most wished they had.

23. Discussion (Continued) 2. Why do students who place into a lower level mathematics course often choose to take a higher-level mathematics course? They did not believe the results of the mathematics placement test to be reflective of their ability

24. Discussion (Continued) 3. What other measures of student preparedness for mathematics courses are possible in addition to the one placement test score? ACT/SAT scores � HS math grades Motivation to do well � HS GPA Use multiple measures for students near cut-off scores or requesting appeal

25. Discussion (Continued) 4. Should mathematics placement be mandatory? If implemented, it would make A/A job easier. A/A inherently trusted the mathematics placement recommendation Web-based registration system unable to handle it Appeal process should continue

26. Recommendations Changes in mathematics placement process Increase number of questions per testing domain to the maximum number (average of 13) Have students choose the domain appropriate to their level of preparedness Increasing the average number of problems in each domain might have increased average time balanced out by having students begin in the appropriate domain based on their background. S with no Algebra background = Arithmetic Domain, S with Pre-Calc = College Algebra DomainIncreasing the average number of problems in each domain might have increased average time balanced out by having students begin in the appropriate domain based on their background. S with no Algebra background = Arithmetic Domain, S with Pre-Calc = College Algebra Domain

27. Recommendations (Continued) Allow students to use a calculator listed on the Compass/ESL approved list of calculators Allow students to retake the test per the guidelines of Compass/ESL Manual Allow students an additional or optional paper/pencil, machine-scored test as an additional measure of their ability

28. Recommendations (Continued) Five-week co-requisite course Allows students an option to spending 10 weeks in a course when they only need some review: An additional one-credit hour course co-requisite with the Introductory Algebra I course Course is a semi-independent study course where S would take a diagnostic test early in the quarter and take final exam when they are ready.Course is a semi-independent study course where S would take a diagnostic test early in the quarter and take final exam when they are ready.

29. Recommendations (Continued) Information Sheets Describe computer-adaptability Strong suggestion of reviewing prior to testing Where to get practice information List of approved calculators How to choose which domain of questions they should begin in How to interpret results of test Description of the retake and appeal policies Interpretation if test results. There are sub-scores reported. S only need to understand the final placement score. S should be given as much info as possible about retake and appeal policies.Interpretation if test results. There are sub-scores reported. S only need to understand the final placement score. S should be given as much info as possible about retake and appeal policies.

30. Recommendations (Continued) Further recommendations Pursue mandatory placement implementation Follow students What course they tested into, what they took, how well they did Add Algebra section/questions to practice web site

31. Limitations Limitations Generalizability: Cannot generalize to other colleges Low number of placement testing observations Not all students tested were interviewed Research whether S are accurately placed.Research whether S are accurately placed.

32. Future Research Future Research Follow-up with student placements Follow-up with effectiveness of recommendations Research why students skip pre-requisite courses

33. Thank you