Examining Exams: Learning Objectives as a Function of Course Level Theresa Houlihan and B. Jean Mandernach, PhD Universi

1. Examining Exams: Learning Objectives as a Function of Course Level • Theresa Houlihan and B. Jean Mandernach, PhD • University of Nebraska at Kearney • It is assumed that classes intended for junior- and senior-level students incorporate progressively more advanced learning outcomes. As such, upper-division courses should go beyond simply providing more information; advanced courses should encourage a deeper understanding and application of a student’s knowledge. The current study analyzes the cognitive complexity targeted by various courses at UNK as measured through the depth of understanding required by typical course assessments. Based on the results, we conclude that upper division courses are indeed utilizing exams which measure higher learning objectives as defined by Bloom’s taxonomy. • Introduction • The purpose of this study is to analyze the cognitive complexity targeted by various courses at the University of Nebraska at Kearney as measured through the depth of understanding required by typical course assessments • Background • Bloom et al. (1964) proposed that various intellectual behaviors and cognitive abilities are utilized in the learning process. As such, an educational taxonomy was constructed to classify the different levels of a student’s mastery of a particular topic according to the depth of understanding. Bloom’s Taxonomy identified six hierarchical levels: simple recall knowledge, comprehension, application, analysis, synthesis and complex mental evaluations. This classification system can be used to analyze items that appear on classroom achievement tests to identify the degree of abstraction measured on such tests. The taxonomy explicitly defines relevant terms and phrases that assess each of the competence areas. • Specifically, the study examines whether upper division courses utilize exam items which indeed measure higher order learning as defined by Bloom’s Taxonomy. It is assumed that classes intended for junior- and senior-level students incorporate progressively more advanced learning outcomes. As such, upper division courses should go beyond simply providing more information; advanced courses should encourage a deeper understanding and application of a student’s knowledge. • If assessments are aimed at student integration and appreciation of information, rather than simple regurgitation, then it is assumed that courses will be taught in such a manner. Thus, the tests administered to students should incorporate an increased emphasis on items pertaining to higher level learning abilities. While exams are not the only measure of student performance, they provide insight into course expectations and often provide the impetus that directs student learning. An analysis of the type and depth of learning that occurs in a given class is an essential component as instructors and departments examine the role of a specific course in the overall curriculum. • Hypothesis • We hypothesize that as the level of the course increases, a greater number of higher order learning items will be present on the exams. Specifically, we hypothesized that upper-division courses (300-400) are using exams that include a greater number of higher order items as compared to lower-division classes (100-200). • Methods • Participants • Professors from three departments in the College of Natural and Social Sciences voluntarily submitted current exams. • 22 total exams were submitted including: • Department A - Two upper and two lower division • Department B - Four upper and four lower division • Department C - Five upper and five lower division • All identifying information was removed from the exams prior to the item analysis to ensure objectivity. • Measures and Procedures • Analysis utilized anassessment rubric based on Bloom’s Taxonomy that explicitly identified the relevant process verbs, product objectives, and terminology commonly associated with each level of understanding. Items on each exam were coded according to the target level of understanding based terminology which corresponded to the assessment rubric. Items were then categorized based on the level of learning the question mandated. Department B Results A descriptive analysis calculated the totaled the number of items that appeared in each hierarchal level and computed totals across exams in the upper and lower division, and for each department. Department A Lower Division Courses Department C • Low level learning objectives consist of Knowledge, Comprehension, and Application. • High level learning objectives consist of Analysis, Synthesis, and Evaluation. • Items • Certain types of items tended to be associated with different learning skills: • Multiple choice items tend to assess Knowledge, Comprehension, and Application with 90% of multiple choice rely on lower level learning abilities as compared to 10% using higher level learning. • Essay items typically emphasize the complex abilities of Analysis, Synthesis, and Evaluation with 76% of essay items relied on higher level learning as compared to 24% lower level. • Essay items appear to be the most effective measure of higher level learning, as compared with true-false items, matching, short answer while multiple choice items are the least effective for measuring higher learning. • Discussion • There appeared to be a trend for overall, lower level courses to utilize exams which included an overwhelming number of items that targeted Knowledge-based skills. While upper division courses had a comparatively decreased reliance on Knowledge abilities (with Evaluation skills accounted for a larger percentage of items as compared to lower courses). This provides partial support for the hypothesis that upper-division classes are using exams that include a greater number of higher order items as compared to lower-division classes, although the difference was not extensive. Despite a general reliance across all courses for Knowledge-based items, different and increasingly complex cognitive abilities are being encouraged and assessed in upper-division courses as consistent with Bloom's theory of hierarchical levels of subject knowledge. Specifically, intradepartmental analysis revealed that classes identified as upper division classes all had exams where higher learning items formed a greater percentage of the items as compared to lower classes. But, this trend was not universal when examining Departments A and B compared to Department C • Important Considerations • Exams are only one component of coursework and most classes integrate complex learning tools and evaluations which may include discussions, projects, or papers. As such, courses may be employing alternative assessments which measure and develop higher order learning skills. Our study examined only typical exams presented in the class and was not intended to measure learning outcomes, thus the current study does not provide data from which to judge the overall level of learning acquired from a given course • Conclusions • The current study analyzes the cognitive complexity targeted by various courses at UNK as measured through the depth of understanding required by typical course assessments. An examination of the type and depth of learning that occurs in a given course is an essential component of an effective departmental or program review. Based on the results, we conclude that upper division courses are utilizing exams which measure higher learning objectives as defined by Bloom’s Taxonomy, although the overall emphasis of basic Knowledge is consistent regardless of class level. Upper Division Courses