1 / 67

BSBA Critical Thinking Skills

Assessed: 5 Cycles 2006, 2009, 2010, 2011, 2013. BSBA Critical Thinking Skills.

smorin
Download Presentation

BSBA Critical Thinking Skills

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Assessed: 5 Cycles 2006, 2009, 2010, 2011, 2013 BSBA Critical Thinking Skills

  2. Goal 2: Analytical and Critical Thinking Skills - Demonstrate effective analytical and critical thinking skills to make an appropriate decisionin a complex situation.Student Learning Outcomes:Collect and organize critical data and information to solve a problem.Find appropriate models and frameworks to analyze information and follow logical steps to reach an effective decision.

  3. Where do our students learn and practice critical thinking skills? • General Education Requirements • Required course in Intermediate Composition and Critical Thinking • Required course in Quantitative Reasoning • Additional lower and upper division General Education requirements • Preparation for the Major Requirements • Core Courses in the BSBA Major • Advanced Courses in the BSBA Major

  4. The refinement of the process over five cycles of assessment. How do we assess our students’ critical thinking skills?

  5. Cycle #1: 2006 • Case analyses completed for the college-wide CBA capstone course (MGT 405) were examined. • CBA’s former Director of Assessment developed a rubric after extensive review of existing critical thinking instruments • 2 CBA faculty worked with the former Director of Assessment to rate the case analyses using the rubric following a training and norming session

  6. Rubric used in Cycle #1

  7. Cycle #2: 2009 • Final exams completed for the college-wide CBA capstone course (MGT 405) were examined. • Course Instructor and Director of Assessment developed a rubric after reviewing the 2006 rubric and others from the critical thinking literature. • Rubric was widely distributed to students during the semester. • Instructor & Director of Assessment rated the final exams using the revised rubric following training and norming sessions.

  8. Rubric Construction Details (for Cycle #2 rubric revision) • 7 Point scale from Weak to Excellent • Expectations determined post-hoc based on examination of means for dimensions. • Dimensions: • Clear understanding of the question and issues to be addressed. • Understanding of the relevant concepts and frameworks. • Effective application of relevant concepts to address question/issues. • Ability to effectively use case data (depth & breadth) to illustrate a position. • Ability to think strategically (i.e. integrate across internal & external environment; across functional areas; take a general management view). • Ability to arrive at logical and well reasoned conclusions based on the discussion.

  9. Cycle #2 Rubric

  10. Cycle #3: 2010 • Final exams completed for the college-wide CBA capstone course (MGT 405) were examined. • Course Instructor and Director of Assessment refined 2009 rubric based on mapping & experience. • Rubric was widely distributed to students during the semester. • Two independent CBA faculty members rated the final exams using the revised rubric following training and norming sessions.

  11. Rubric Revision Details (for Cycle #3) • 7 Point scale from Weak to Excellent retained. • Expectations determination post-hoc based on examination of means for dimensions retained. • Dimensions from Cycle #2 Rubric were reduced by one and wording modified in some cases: • Clear understanding of the question and issues to be addressed. • Knowledge of case facts & ability to effectively use case data to illustrate position. • Effective understanding and application of the relevant concepts and frameworks. • Ability to think strategically (i.e. integrate across internal & external environment; across functional areas; holistically from firm perspective). • Ability to arrive at logical and well reasoned conclusions/recommendations.

  12. Cycle #3 Rubric

  13. Cycle #4: 2011 • Process adopted for Cycle #3 retained. • MGT 405 final exams used • Two independent CBA faculty members rated. • Additional rubric revision/refinement in Cycle #4. • Descriptions of levels added to each dimension. • “Approaches Expectations” dropped in favor of 3 point scale to provide consistency across assessment measures: • Exceeds Expectations • Meets Expectations • Below Expectations

  14. Cycle #4 Rubric

  15. Cycle #5: 2013 • Process adopted for Cycle #3, used in Cycle #4 retained. • MGT 405 final exams used • Two independent CBA faculty members rated. • Rubric revision in Cycle #5: • Descriptions of levels as added in Cycle #4 retained. • 3 point scale added in Cycle #4 retained. • Point ranges within three point scale dropped in favor of: • 1 – Below Expectations; 2 – Meets Expectations; 3 – Above Expectations • However, ½ and ¼ point judgments acceptable when appropriate. • Dimensions reduced to two to better align with Critical Thinking SLOs. • Third dimension rating writing used, results not included in this report.

  16. Cycle #5 Rubric

  17. Assessment Conducted: • Cycle #1: Fall Semester 2006 & Spring Semester 2007 in MGT 405 (International Business Strategy & Integration): • Sample Size: 175 • Cycle #2: Fall Semester 2009 in MGT 405 • Sample Size: 124 • Cycle #3: Fall Semester 2010 in MGT 405 • Sample Size: 119 • Cycle #4: Fall Semester 2011 in MGT 405 • Sample Size: 120 • Cycle #5: Data collected In Fall Semester 2012 in MGT 405, analyzed Spring 2013 • Sample Size: 122

  18. Updated through here . . .

  19. SLO & Dimension Mapping • SLO #1: Collect and organize critical data and information to solve a problem. • Cycle #1: Issue Identification • Cycle #2 - #4: Clear understanding of the question and issues to be addressed. • Cycle #1: Use of Evidence & Data • Cycle #2: Ability to effectively use case data (depth & breadth) to illustrate a position. • Cycles #3 & 4: Knowledge of case facts & ability to effectively use case data to illustrate position. • Cycle #5: Single dimension to assess SLO #1 as indicated on rubric.

  20. SLO & Dimension Mapping • SLO #2: Find appropriate models and frameworks to analyze information and follow logical steps to reach an effective decision. • Cycle #1: Models & Frameworks • Cycle #2: 1) Understanding of relevant concepts & frameworks; 2) Effective application of relevant concepts to address question/issues • Cycle #3 & #4: Effective understanding and application of the relevant concepts & frameworks. • Cycle #1: Conclusions & Recommendations • Cycle #2: Ability to arrive at logical and well reasoned conclusions based on discussion. • Cycle #3 & #4: Ability to arrive at logical and well-reasoned conclusions/recommendations. • Cycle #2: Ability to think strategically (i.e. integrate across internal & external environment; across functional areas; take a general management view. • Cycle #3 & #4: Ability to think strategically (integrate across internal & external environments, functional areas, holistically from firm perspective). • Cycle #5: Single dimension to assess SLO #2 as indicated on rubric.

  21. What do we expect of our students? • BENCHMARKS: • 85% of our students should meet or exceed expectations for critical thinking skills • 50% of our students should exceed expectations for critical thinking skills

  22. RESULTS Cycle #1

  23. SLO #1: Collect and organize critical data and information to solve a problem. 53% 23% 13% 11% “85% should meet or exceed expectations”; 83% actually did “50% should exceed expectations”; 53% actually did

  24. SLO #1: Collect and organize critical data and information to solve a problem. 43% 27% 22% 8% “85% should meet or exceed expectations”; 65% actually did “50% should exceed expectations”; 22% actually did

  25. SLO #2:Find appropriate models and frameworks to analyze information and follow logical steps to reach an effective decision. 65% 17% 14% 3% “85% should meet or exceed expectations”; 31% actually did “50% should exceed expectations”; 14% actually did

  26. SLO #2:Find appropriate models and frameworks to analyze information and follow logical steps to reach an effective decision. 45% 26% 23% 7% “85% should meet or exceed expectations”; 71% actually did “50% should exceed expectations”; 26% actually did

  27. Results Summary “85% of our students should meet or exceed expectations” “50% of our students should exceed expectations” YES Issue Identification NO Use of Evidence & Data Models & Frameworks Conclusions & Recommendations • NO • Issue Identification • Use of Evidence & Data • Models & Frameworks • Conclusions & Recommendations

  28. Cycle #1 Closing the Loop • 4 recommendations were made by the assessment team regarding ways to improve the analytic and critical thinking skills of our students. • Faculty should encourage students to make clear problem statements without hedging. • Students should be asked to show their use of models and tools and to demonstrate a clear connection between those models and their analysis. • Faculty should model the use of data in developing solutions to cases and problems. • Students should be given opportunities to learn how to support recommendations with evidence by writing and revising these sections of their papers. • An additional recommendation was to urge faculty to grade assignments with a rubric and to distribute the rubric to students prior to assignment submission

  29. Cycle #1 Closing the Loop • A Memo was sent to all CBA faculty detailing the recommendations and strongly urging them to adopt them • A number of faculty members across the college expressed interest in learning more about the use of rubrics and rubric development. • The CBA Assessment Committee developed a short “rubric primer” which was distributed to the CBA faculty along with examples of rubrics available online and rubrics currently being used in the college.

  30. RESULTS Cycle #2

  31. SLO #1: Collect and organize critical data and information to solve a problem. 39% 53% 5% 3% “85% should meet or exceed expectations”; 92% actually did “50% should exceed expectations”; 39% actually did

  32. SLO #1: Collect and organize critical data and information to solve a problem. 52% 19% 11% 18% “85% should meet or exceed expectations”; 71% actually did “50% should exceed expectations”; 19% actually did

  33. SLO #2:Find appropriate models and frameworks to analyze information and follow logical steps to reach an effective decision. 50% 27% 10% 13% “85% should meet or exceed expectations”; 77% actually did “50% should exceed expectations”; 27% actually did

  34. SLO #2:Find appropriate models and frameworks to analyze information and follow logical steps to reach an effective decision. 47% 24% 17% 12% “85% should meet or exceed expectations”; 71% actually did “50% should exceed expectations”; 24% actually did

  35. SLO #2:Find appropriate models and frameworks to analyze information and follow logical steps to reach an effective decision. 63% 21% 9% 7% “85% should meet or exceed expectations”; 84% actually did “50% should exceed expectations”; 21% actually did

  36. SLO #2:Find appropriate models and frameworks to analyze information and follow logical steps to reach an effective decision. 53% 21% 15% 11% “85% should meet or exceed expectations”; 74% actually did “50% should exceed expectations”; 21% actually did

  37. Results Summary by Dimension “85% of our students should meet or exceed expectations” “50% of our students should exceed expectations” NO Understanding question & issues Understanding relevant concepts & frameworks Application of relevant concepts to issues Effective use of case data Ability to think strategically Ability to arrive at conclusions • YES • Understanding question & issues • NO • Understanding relevant concepts & frameworks • Application of relevant concepts to issues • Effective use of case data • Ability to think strategically • Ability to arrive at conclusions

  38. Results Summary by SLO “85% of our students should meet or exceed expectations” “50% of our students should exceed expectations” NO SLO #1: Collect and organize critical data and information to solve a problem. SLO #2: Find appropriate models and frameworks to analyze information and follow logical steps to reach an effective decision. • NO • SLO #1: Collect and organize critical data and information to solve a problem. • SLO #2: Find appropriate models and frameworks to analyze information and follow logical steps to reach an effective decision.

  39. Cycle #1 & #2 Comparisons* SLO #1 SLO #2 * Dimensions combined to form single SLO measure

  40. Comparison Observations • Our students appear to have improved slightly in meeting expectations between Cycles #1 & #2. • The benchmark was not achieved for any dimension in Cycle #1 but was achieved on one and very nearly on a second dimension in Cycle #2. • Overall, however, benchmarks were not met for either SLO. • Our students continue to struggle with critical thinking.

  41. Following Cycle #2 . . . • The Undergraduate Committee was heartened by the improvement between Cycles #1 & #2 albeit small. • A decision was made to reassess Critical Thinking in one year to determine if the improvement was the beginning of a trend prior to making additional “Loop Closing” decisions.

  42. RESULTS Cycle #3

  43. SLO #1: Collect and organize critical data and information to solve a problem. 67.2% 17.6% 5% 10.1% “85% should meet or exceed expectations”; 85% actually did “50% should exceed expectations”; 18% actually did

  44. SLO #1: Collect and organize critical data and information to solve a problem. 57.1% 16.8% 16.8% 9.2% “85% should meet or exceed expectations”; 74% actually did “50% should exceed expectations”; 17% actually did

  45. SLO #2:Find appropriate models and frameworks to analyze information and follow logical steps to reach an effective decision. 52.1% 21% 16.8% 10.1% “85% should meet or exceed expectations”; 62% actually did “50% should exceed expectations”; 10% actually did

  46. SLO #2:Find appropriate models and frameworks to analyze information and follow logical steps to reach an effective decision. 57.1% 18.5% 12.6% 11.8% “85% should meet or exceed expectations”; 69% actually did “50% should exceed expectations”; 12% actually did

  47. SLO #2:Find appropriate models and frameworks to analyze information and follow logical steps to reach an effective decision. 56.3% 18.5% 18.5% 6.7% “85% should meet or exceed expectations”; 63% actually did “50% should exceed expectations”; 7% actually did

  48. Results Summary by Dimension “85% of our students should meet or exceed expectations” “50% of our students should exceed expectations” NO Understanding question & issues Understanding & application of relevant concepts & frameworks Effective use of case data Ability to think strategically Ability to arrive at conclusions • YES • Understanding question & issues • NO • Understanding & application of relevant concepts & frameworks • Effective use of case data • Ability to think strategically • Ability to arrive at conclusions

  49. Results Summary by SLO “85% of our students should meet or exceed expectations” “50% of our students should exceed expectations” NO SLO #1: Collect and organize critical data and information to solve a problem. SLO #2: Find appropriate models and frameworks to analyze information and follow logical steps to reach an effective decision. • NO • SLO #1: Collect and organize critical data and information to solve a problem. • SLO #2: Find appropriate models and frameworks to analyze information and follow logical steps to reach an effective decision.

  50. Comparison Observations following Cycle #3 • Although the number of students meeting expectations continues to increase, there was a significant decline in those exceeding expectations between Cycles #2 & #3. • The percent of students failing to meet expectations for SLO #2 was particularly discouraging.

More Related