educational data mining discovery with models n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Educational Data Mining: Discovery with Models PowerPoint Presentation
Download Presentation
Educational Data Mining: Discovery with Models

Loading in 2 Seconds...

play fullscreen
1 / 63

Educational Data Mining: Discovery with Models - PowerPoint PPT Presentation


  • 129 Views
  • Uploaded on

Educational Data Mining: Discovery with Models. Ryan S.J.d. Baker PSLC/HCII Carnegie Mellon University Ken Koedinger CMU Director of PSLC Professor of Human-Computer Interaction & Psychology Carnegie Mellon University. In this segment….

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Educational Data Mining: Discovery with Models' - amaya-contreras


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
educational data mining discovery with models

Educational Data Mining:Discovery with Models

Ryan S.J.d. BakerPSLC/HCII

Carnegie Mellon University

Ken Koedinger CMU Director of PSLC

Professor of Human-Computer Interaction & Psychology

Carnegie Mellon University

in this segment
In this segment…
  • We will discuss Discovery with Models in (some) detail
last time
Last time…
  • We gave a very simple example of Discovery with Models using Bayesian Knowledge Tracing
uses of knowledge tracing
Uses of Knowledge Tracing
  • Can be interpreted to learn about skills
why do discovery with models
Why do Discovery with Models?
  • We have a model of some construct of interest or importance
    • Knowledge
    • Meta-Cognition
    • Motivation
    • Affect
    • Collaborative Behavior
      • Helping Acts, Insults
    • Etc.
why do discovery with models1
Why do Discovery with Models?
  • We can now use that model to
    • Find outliers of interest by finding out where the model makes extreme predictions
    • Inspect the model to learn what factors are involved in predicting the construct
    • Find out the construct’s relationship to other constructs of interest, by studying its correlations/associations/causal relationships with data/models on the other constructs
    • Study the construct across contexts or students, by applying the model within data from those contexts or students
    • And more…
finding outliers of interest
Finding Outliers of Interest
  • Finding outliers of interest by finding out where the model makes extreme predictions
    • As in the example from Bayesian Knowledge Tracing
    • As in Ken’s example yesterday of finding upward spikes in learning curves
model inspection
Model Inspection
  • By looking at the features in the Gaming Detector, Baker, Corbett, & Koedinger (2004, in press) were able to see that
  • Students who game the system and have poor learning
    • game the system on steps they don’t know
  • Students who game the system and have good learning
    • game the system on steps they already know
model inspection a tip
Model Inspection: A tip
  • The simpler the model, the easier this is to do
  • Decision Trees and Linear/Step Regression: Easy.
model inspection a tip1
Model Inspection: A tip
  • The simpler the model, the easier this is to do
  • Decision Trees and Linear/Step Regression: Easy.
  • Neural Networks and Support Vector Machines: Fuhgeddaboudit!
take model of a construct
Take Model of a Construct
  • And see whether it co-occurs with other constructs of interest
example
Example
  • Detector of gaming the system (in fashion associated with poorer learning) correlated with questionnaire items assessing various motivations and attitudes(Baker et al, 2008)
example1
Example
  • Detector of gaming the system (in fashion associated with poorer learning) correlated with questionnaire items assessing various motivations and attitudes(Baker et al, 2008)
  • Surprise: Nothing correlated very well(correlations between gaming and some attitudes statistically significant, but very weak – r < 0.2)
example2
Example
  • More on this in a minute…
studying a construct across contexts
Studying a Construct Across Contexts
  • Often, but not always, involves:
model transfer1
Model Transfer
  • Richard said that prediction assumes that the
  • Sample where the predictions are made
  • Is “the same as”
  • The sample where the prediction model was made
  • Not entirely true
model transfer2
Model Transfer
  • It’s more that prediction assumes the differences “aren’t important”
  • So how do we know that’s the case?
model transfer3
Model Transfer
  • You can use a classifier in contexts beyond where it was trained, with proper validation
  • This can be really nice
    • you may only have to train on data from 100 students and 4 lessons
    • and then you can use your classifier in cases where there is data from 1000 students and 35 lessons
  • Especially nice if you have some unlabeled data set with nice properties
    • Additional data such as questionnaire data(cf. Baker, 2007; Baker, Walonoski, Heffernan, Roll, Corbett, & Koedinger, 2008)
validate the transfer
Validate the Transfer
  • You should make sure your model is valid in the new context(cf. Roll et al, 2005; Baker et al, 2006)
  • Depending on the type of model, and what features go into it, your model may or may not be valid for data taken
    • From a different system
    • In a different context of use
    • With a different population
validate the transfer1
Validate the Transfer
  • For example
  • Will an off-task detector trained in schools work in dorm rooms?
validate the transfer2
Validate the Transfer
  • For example
  • Will a gaming detector trained in a tutor where {gaming=systematic guessing, hint abuse}
  • Work in a tutor where{gaming=point cartels}
validate the transfer3
Validate the Transfer
  • However
  • Will a gaming detector trained in a tutor unit where {gaming=systematic guessing, hint abuse}
  • Work in a different tutor unit where {gaming=systematic guessing, hint abuse}?
baker corbett koedinger roll 2006
Baker, Corbett, Koedinger, & Roll (2006)
  • We tested whether
  • A gaming detector trained in a tutor unit where {gaming=systematic guessing, hint abuse}
  • Would work in a different tutor unit where {gaming=systematic guessing, hint abuse}
scheme
Scheme
  • Train on data from three lessons, test on a fourth lesson
  • For all possible combinations of 4 lessons (4 combinations)
transfer lesson vs training lessons
Transfer lesson .vs. Training lessons
  • Ability to distinguish students who game from non-gaming students
  • Overall performance in training lessons: A’ = 0.85
  • Overall performance in test lessons: A’ = 0.80
  • Difference is NOT significant, Z=1.17, p=0.24 (using Strube’s Adjusted Z)
so transfer is possible
So transfer is possible…
  • Of course 4 successes over 4 lessons from the same tutor isn’t enough to conclude that any model trained on 3 lessons will transfer to any new lesson
slide35
If…
  • If we posit that these four cases are “successful transfer”, and assume they were randomly sampled from lessons in the middle school tutor…
studying a construct across contexts1
Studying a Construct Across Contexts
  • Using this detector(Baker, 2007)
research question
Research Question
  • Do students game the system because of state or trait factors?
  • If trait factors are the main explanation, differences between students will explain much of the variance in gaming
  • If state factors are the main explanation, differences between lessons could account for many (but not all) state factors, and explain much of the variance in gaming
  • So: is the student or the lesson a better predictor of gaming?
application of detector
Application of Detector
  • After validating its transfer
  • We applied the gaming detector across 35 lessons, used by 240 students, from a single Cognitive Tutor
  • Giving us, for each student in each lesson, a gaming frequency
model
Model
  • Linear Regression models
  • Gaming frequency = Lesson + a0
  • Gaming frequency = Student + a0
model1
Model
  • Categorical variables transformed to a set of binaries
  • i.e. Lesson = Scatterplot becomes
  • 3DGeometry = 0
  • Percents = 0
  • Probability = 0
  • Scatterplot = 1
  • Boxplot = 0
  • Etc…
slide43
r2
  • The correlation, squared
  • The proportion of variability in the data set that is accounted for by a statistical model
slide44
r2
  • The correlation, squared
  • The proportion of variability in the data set that is accounted for by a statistical model
slide45
r2
  • However, a limitation
  • The more variables you have, the more variance you should be expected to predict, just by chance
slide46
r2
  • We should expect
  • 240 students
  • To predict gaming better than
  • 35 lessons
  • Just by overfitting
our good friend bic
Our good friend BiC
  • Bayesian Information Criterion(Raftery, 1995)
  • Makes trade-off between goodness of fit and flexibility of fit (number of parameters)
the lesson
The Lesson
  • Gaming frequency = Lesson + a0
  • 35 parameters
  • r2 = 0.55
  • BiC’ = -2370
    • Model is significantly better than chance would predict given model size & data set size
the student
The Student
  • Gaming frequency = Student + a0
  • 240 parameters
  • r2 = 0.16
  • BiC’ = 1382
    • Model is worse than chance would predict given model size & data set size!
in this talk
In this talk…
  • Discovery with Models to
    • Find outliers of interest by finding out where the model makes extreme predictions
    • Inspect the model to learn what factors are involved in predicting the construct
    • Find out the construct’s relationship to other constructs of interest, by studying its correlations/associations/causal relationships with data/models on the other constructs
    • Study the construct across contexts or students, by applying the model within data from those contexts or students
necessarily
Necessarily…
  • Only a few examples given in this talk
in the last 3 days we have discussed or at least mentioned 5 broad areas of edm
In the last 3 days we have discussed (or at least mentioned)5 broad areas of EDM
  • Prediction
  • Clustering
  • Relationship Mining
  • Discovery with Models
  • Distillation of Data for Human Judgment
now it s your turn
Now it’s your turn
  • To use these techniques to answer important questions about learners and learning
  • To improve these techniques, moving forward
to learn more
To learn more
  • Baker, R.S.J.d. (under review) Data Mining in Education. Under review for inclusion in the International Encyclopedia of Education
    • Available upon request
  • Baker, R.S.J.d., Barnes, T., Beck, J.E. (2008) Proceedings of the First International Conference on Educational Data Mining
  • Romero, C., Ventura, S. (2007) Educational Data Mining: A Survey from 1995 to 2005. Expert Systems with Applications, 33 (1), 135-146.
slide63
r2
  • Nine variables of random junk successfully got an r2 of 1 on ten data points
  • And that’s what we call overfitting 