By Nathan Lindsay and Dan Stroud January 30, 2013

1. Sizing Up Standardized Assessments: Breaking Down the ETS Proficiency Profile, Major Field Tests, and the RooWriter By Nathan Lindsay and Dan StroudJanuary 30, 2013

2. Assessment at UMKC Consists of a multifaceted approach: • Formative and Summative Assessments • Direct and Indirect Assessments • Direct assessments include embedded assessments (in the classroom), and standardized assessments

3. UMKC Standardized Tests Overview For many years, undergraduate students at UMKC have participated in the following standardized tests: • The ETS-Proficiency Profile test that measures General Education learning outcomes • Major Field Tests (MFTs) that measure learning outcomes for students in their disciplines. • The WEPT (soon to be the RooWriter) that measures students’ writing ability For the ETS tests, only this past year did we purchase a reporting license that allows us to closely analyze the data

4. Question • What are some of the pros and cons of standardized tests?

5. Why Standardized Tests Might be useful Along side in-house Assessments • Standardized tests allow comparisons to both national and peer data (which are requested by external constituents). • There are benefits in the use of longitudinal data. • ETS test design includes strong: • Validity • Reliability • Fairness/Equity • Representation

6. How ETS Develops Test Questions Below is the process by which tests and test questions are developed for future ETS Proficiency Profile tests and MFTs: http://www.ets.org/s/understanding_testing/flash/how_ets_creates_test_questions.html

7. Test Scoring Two Types of Test Scores • Norm-referenced (scaled) scores - These scores pull their meaning from comparing the scores between and individual student or group of students with those of either the same individual or group or another at different points of time. The majority of test results are of this nature. • Criterion-referenced (proficiency classified) scores - These pertain to the level of proficiency students obtained within a particular skill set. This type of score stands on its own – with no need to compare to another group, though they can still use specific comparisons as well.

8. Sampling the Proficiency Profile

9. Sampling the Proficiency Profile

10. Sampling the Proficiency Profile

11. ETS – Proficiency Profile (In 2011-2012, n = 1448 students) • Test given to all students after they have completed 70 credit hours. Completion is a requirement of graduation. • Scores were above benchmark peers for all four areas: • Reading • Critical Thinking • Writing • Mathematics • Scores on individual items exceeded the national average 51 of 53 times. • Higher scores may be due to the fact that most UMKC students take the ETS-PP right before they graduate (which is not always true at other institutions)

12. Major Field Tests (MFTs) • UMKC students take the MFTs in the following areas: • Biology • Business • Psychology • Political Science • Chemistry • Computer Science • Mathematics • Physics 20 students have to take the test to have sufficient comparative data, so results are only available for some of these disciplines.

13. Two Reports available for Major Field Tests • The Custom Comparative Data Report indicates how UMKC’s students' skills and knowledge compare with the skills and knowledge of students at similar institutions. • In the Item Information Report, the items are prioritized first by content, and subsequently by sub-content to make comparisons easier for grouping. Scores compared in this report are referenced to the overall national percentage for each item.

14. Major Field Tests Results at UMKC • Biology (N = 87 Students) – UMKC scored above average on 8 of 9 Assessment Indicators. • Business (N = 192 Students) – UMKC scored above average on 4 of 9 Assessment Indicators. • Psychology (N = 81 Students) – UMKC scored above average on 2 of 6 Assessment Indicators. • Political Science (N = 20 Students) – UMKC scored above average on 1 of 3 Assessment Indicators.

15. Disseminating the ETS Results • The General Education scores have been reviewed by the General Education Committee, the University Assessment Committee, and the Assessment Academy Team. • These MFT reports have been sent to all relevant departments, which have been encouraged to discuss how they can enhance their curriculum to address the areas of weakness within each test. • A summary of our standardized tests will be shared with the Higher Learning Commission. • We will collect data during the 2012-2013 academic year to determine any changes over time.

16. Departmental Dialogue: Converting the data to Action • Use the data to stimulate conversation in the department about the curriculum and pedagogy surrounding these topics. Address the following questions: • How do the content areas of the test relate to our degree’s learning outcomes? Should test items be added? • Are there any areas of the curriculum that could/should be emphasized more strongly? [curriculum mapping] • Do the results have face validity? How do the results compare to our findings from course-based assessments and our program/degree assessments? • How did different student groups/demographics score on the test? Should more sub-groups for analysis be created? • What are other ways that we could address/assess the areas that were lower? Should we develop our own test? • What trends do the scores show longitudinally? • How could we increase student motivation for the test?

17. How to customize MFT Reporting Groups • Cohorts - Determine which students to put in what groups as you design report emphasis • Subgroups - You can create up to 16 subgroups - For each subgroup, you can design two questions with up to 8 possible responses. • There must be a minimum of 5 students in each group, cohort, or subgroup. Minimums for reporting cohorts or subgroups must add up together to 20 as a minimum to report.

18. Discussion Are there other ways that your unit/department could develop test questions that would evaluate the learning outcomes for your degree(s)? Questions to consider: • What’s the point/purpose of the test? • What are the intended uses of this test? • What aspects of each unit or department do we wish to measure? • What scores will be reported? • Is the test in question statistically reliable, valid, and fair?

19. Student Motivation for Standardized Assessments • Key Issue - High stakes for institutions - Low stakes for students • Incentivizing is two tiered - Get students to take the test - Get students to do their best • Best Practice—Connect to Academic Outcomes - Participation grade for a class - Students who attain a score of _______ or higher receives _______?

20. Discussion What incentives could be used to help students to take standardized testing more seriously?

21. Other Ways to Encourage Meaningful Participation • Inform students well in advance • Include the MFT in the syllabus • Send letter or email from Dean/Department Chair • Develop faculty understanding of the purposes of the test • Create interdepartmental competition • Develop an Assessment Day (Long range goal)

22. WEPT • This is a Pass/Fail test, and approximately 5% of students have failed the test the last few semesters • Each test is rated on 4 areas:  • Point/Thesis/Idea – Evaluated on the clarity of purpose throughout the essay. • Development/Support/Evidence – Evaluated on the relevance and convincingness of the proof offered in support of the thesis. • Organization/Logic/Coherence – Evaluated on the ease of understanding and logical arrangement of the essay. • Usage/Mechanics/Documentation – Evaluated on the ease of reading and grammatical and style correctness • The test is graded on a four-point scale • The average scores for area one (i.e., “Point/Thesis/Idea”) were the highest at 3.10, up from the 3.07 in the previous semester. The remaining three scores average fell near 2.82 (2.82, 2.83, and 2.80, respectively).

23. The RooWriter • To replace the WEPT for incoming freshmen in Fall 2013 • An online diagnostic/formative test with many benefits for students, faculty, and the institution (see attached handout) • The University Writing and Reading Board is currently working on finalizing all of the logistics for the test • Please refer to the RooWriter website for an overview: • http://dev.umkc.edu/roowriter/index.cfm

24. Questions • Are there other standardized tests that UMKC should be considering? • What other questions about standardized tests do you still have?