Team: Matthew Bernacki & Pranav Garg Mentors: Erik Zawadzki & Ryan Baker

1. 2011 Summer School Tracing behaviors associated with motivational states and learning outcomes when students learn with the Cognitive Tutor Team: Matthew Bernacki & PranavGarg Mentors: Erik Zawadzki & Ryan Baker

2. Overview We investigated relationships between motivation, learning behaviors and learning outcomes amongst high school students learning geometry using the Cognitive Tutor. We identified a series of 3 sequential behaviors (triplets) and plotted their frequency across the logs of 38 learners in one geometry unit. We conducted a factor analysis to reduce 147 triplets into 28 factors and examined their correlation with self-reports of affective state, self-efficacy for the unit and their achievement goals for mathematics.

3. METHOD: In the Classroom Participants • 38 high school geometry students completing Unit 13 in the Cognitive Tutor, which was a standard component of the their rural high school’s geometry curriculum. Instruments • Cognitive Tutor for Geometry • Unit 13: Circumference and Area of Circles • Achievement Goal Questionnaire-Revised • Elliot & Murayama, 2008 (9 items, 3 per Mastery Approach, Performance Approach, Performance Avoidance subscale) • Academic Self Efficacy Survey • Midgely, et al., 2000; Patterns of Adaptive Learning Survey • Affect (single items constructed for this project) • Boredom, Confusion, Frustration Engaged Concentration, Positive Experience

4. METHOD: Data Mining Procedure • Exported transaction level log file from Cognitive Tutor • Selected only those students who completed the Unit of interest; cleaned data to remove any students who were missing self-report data or a complete log file • Calculated the duration (seconds) to complete each learner action in the OUTCOME column • OK – answered problem step correctly • BUG – incorrectly answered the problem step (common error) • ERROR – incorrectly answered the problem step • HINT [1,2,3]– requested a hint • SWITCH – switched their window to consult a worked example • Recoded Duration by Quartile (1, middle 2&3, 4) • Q1 = Short durations, typically 1-2 seconds; coded as “…_1” • Q2&3 = Medium Durations, typically 2-10 seconds; coded as “…_2” • Q4 = Longest durations, typically upward of 10 seconds; coded as “…_3” • Concatenated Outcome with Q(uartile version of) Duration.

5. METHOD: Data Mining Cont’d. Ran a script in Python to move a sliding window over the OutcomeQDuration column and populated a column with a triplet: [FIRST TRANSACTION_DURATION_SECOND_D_THIRD_D]. Calculated the total number of unique triplets (n = 7,885) and, with a Pivot Table, determined the frequency each occurred per student. Eliminated those that occurred less than 5 times and those that occurred in less than 2 students (n = 147) Imported into SPSS, merged with a file of their self-reported motivational states and official record of learning outcomes Ran a Principle Components Factor Analysis (unrotated) to determine a factor structure. Correlated Factor Scores with motivation and performance data

6. RESULTS

8. THE FACTORS

9. THE FACTORS

10. THEORETICAL CONCLUSIONS • Some behaviors associated with a factor can be interpretted somewhat easily. Factor 12: • BUG_1_OK_3_OK_2{.490} • BUG_3_OK_2_OK_3{.477} • ERROR_3_OK_3_OK_3{.382} • Student made errors, often after some perserveration, then correctly answered items after a medium to long period. • Factor is associated with low frustration.

11. THEORETICAL CONCLUSIONS • However, scores on one factor (#27) were significantly associated with self-reports of confusion and frustration and negatively associated with mastery and performance approach goals. • Only one triplet [OK_1_OK_3_OK_1] loaded higher than .30 on the factor • Low factor loading and meaningfulness of a short period prior to a correct, followed by a long and a short actually run counter to some conclusions based on self-reports.

12. METHODOLOGICAL CONCLUSIONS • Triplets composed of behaviors and their durations can be meaningful measures of behavior • They can also be insufficiently descriptive of a students’ behavior re: specificity • number of behaviors captured • Precision of duration when collapsed to quartile • Meaningfulness of cuts between quartiles

13. NEXT STEPS • Test factor structure across additional units • If not, may make sense to abandon factors and examine relations one behavioral trace at a time • Generate 4-lets and 5-lets to see if these behaviors provide more intuitive glimpses of student behaviors • Once a set of behaviors has been found that associate with motivation • develop a flag for a behavior… and an intervention? • Test structural models with paths from motivational state to behavior to learning outcomes