1 / 62

Using Data-Driven Discovery Techniques for the Design and Improvement of Educational Systems

Using Data-Driven Discovery Techniques for the Design and Improvement of Educational Systems. John Stamper Pittsburgh Science of Learning Center Human-Computer Interaction Institute Carnegie Mellon University 4/8/2013. The Classroom of the Future.

ady
Download Presentation

Using Data-Driven Discovery Techniques for the Design and Improvement of Educational Systems

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Using Data-Driven Discovery Techniques for the Design and Improvement of Educational Systems John Stamper Pittsburgh Science of Learning CenterHuman-Computer Interaction InstituteCarnegie Mellon University 4/8/2013

  2. The Classroom of the Future Which picture represents the “Classroom of the Future”?

  3. The Classroom of the Future The answer is both! Depends of how much money you have... … but maybe not what you think…

  4. The Classroom of the Future Rich vs. Poor • Poor kids will be forced to rely on “cheap” technology • Rich kids will have access to “expensive” teachers We are seeing this today! • Waldorf school in Silicon Valley – no technology • NGLC Wave III Grants • MOOCs • Growth of adaptive technology companies • Online instruction • … and more…

  5. What does this mean? My view is that we cannot stop this, I believe we must accept that economics will force this route. We should focus on improving learning technology • New ways to improve teacher-student access • Add more adaptive features to learning software Adaptive learning, at scale, using data!

  6. Educational Data Mining • “Educational Data Mining is an emerging discipline, concerned with developing methods for exploring the unique types of data that come from educational settings, and using those methods to better understand students, and the settings which they learn in.” • www.educationaldatamining.org

  7. Types of EDM methods(Baker & Yacef, 2009) • Prediction • Classification • Regression • Density estimation • Clustering • Relationship mining • Association rule mining • Correlation mining • Sequential pattern mining • Causal data mining • Distillation of data for human judgment • Discovery with models

  8. Emerging Communities • Society for Learning Analytics Research • First conference: LAK2011 • International Educational Data Mining Society • First conference: EDM2008 • Publishing JEDM since 2009 • Plus an emerging number of great people working in this area who are (not yet) closely affiliated with either community

  9. Emerging Communities • Joint goal of exploring the “big data” now available on learners and learning • To promote • New scientific discoveries & to advance learning sciences • Better assessment of learners along multiple dimensions • Social, cognitive, emotional, meta-cognitive, etc. • Individual, group, institutional, etc. • Better real-time support for learners

  10. EDM Methods to discuss • Prediction – understand what the student knows • Discovery with models – improve understanding of the structure of knowledge

  11. LearnLabPittsburgh Science of Learning Center (PSLC) • Created to bridge the Chasm between science & practice • Low success rate (<10%) of randomized field trials • LearnLab = a socio-technical bridge between lab psychology & schools • E-science of learning & education • Social processes for research-practice engagement • Purpose: Leverage cognitive theory and computational modeling to identify the conditions that cause robust student learning

  12. LearnLab: Data-driven improvement infrastructure Ed tech + wide use = Research in practice • 2004-14, ~$50 million • Tech enhanced courses, assessment, & research • School cooperation • In vivo experiments Algebra Cognitive Tutor + = Chemistry Virtual Lab English Grammar Tutor Educational Games

  13. Interaction data is surprisingly revealing Online interactions => state tests • Accurate assessment during learning • Detect student work ethic, engagement … • Discover better models of what is hard to learn R = .82 Learning Curve Analysis Flat curve => improvement opportunity

  14. Central Repository Secure place to store & access research data Supports various kinds of research Primary analysis of study data Exploratory analysis of course data Secondary analysis of any data set Analysis & Reporting Tools Focus on student-tutor interaction data Data Export Tab delimited tables you can open with your favorite spreadsheet program or statistical package Web services for direct access DataShop 14

  15. Repository • Allows for full data management • Controlled access for collaboration • File attachments • Paper attachments • Great for secondary analyses How big is DataShop?

  16. How big is DataShop? As of April 2013

  17. What kinds of data? • By domain based on studies from the Learn Labs • Data from intelligent tutors • Data from online instruction • Data from games The data is fine grained at a transaction level!

  18. Web Application

  19. Explore data through the DataShop tools Where is DataShop? http://pslcdatashop.org Linked from DataShop homepage and learnlab.org http://pslcdatashop.web.cmu.edu/about/ http://learnlab.org/technologies/datashop/index.php Getting to DataShop 19

  20. DataShop Terminology • KC: Knowledge component • also known as a skill/concept/fact • a piece of information that can be used to accomplish tasks • tagged at the step level • KC Model: • also known as a cognitive model or skill model • a mapping between problem steps and knowledge components

  21. Getting the KC Model Right! The KC model drives instruction in adaptive learning • Problem and topic sequence • Instructional messages • Tracking student knowledge

  22. What makes a good KC Model? • A correct expert model is one that is consistent with student behavior. • Predicts task difficulty • Predicts transfer between instruction and test • The model should fit the data!

  23. Good KC Model => Good Learning Curve An empirical basis for determining when a cognitive model is good Accurate predictions of student task performance & learning transfer Repeated practice on tasks involving the same skill should reduce the error rate on those tasks => A declining learning curve should emerge

  24. A Good Learning Curve

  25. How do we make KC Models?

  26. Traditionally CTA has been used But Cognitive Task Analysis has some issues… • Extremely human driven • It is highly subjective • Leading to differing results from different analysts And these human discovered models are usually wrong!

  27. If Human centered CTA is not the answer How should these models be designed? They shouldn’t! The models should be discovered not designed!

  28. Solution • We have lots of log data from tutors and other systems • We can harness this data to validate and improve existing student models

  29. Human-Machine Student Model Discovery DataShop provides easy interface to add and modify KC models and ranks the models using AFM

  30. Human-Machine Student Model Discovery 3 strategies for discovering improvements to the student model • Smooth learning curves • No apparent learning • Problems with unexpected error rates

  31. A good cognitive model produces a learning curve Without decomposition, using just a single “Geometry” skill, no smooth learning curve. But with decomposition, 12 skills for area, (Rise in error rate because poorer students get assigned more problems) a smooth learning curve. Is this the correct or “best” cognitive model?

  32. Inspect curves for individual knowledge components (KCs) Some do not =>Opportunity to improve model! Many curves show a reasonable decline

  33. No apparent Learning

  34. Problems with Unexpected Error Rates

  35. Inspect problems to hypothesize new KC labels • Here scaffolding is originally absent, but other problems have fixed scaffolding • They start with columns for square & area

  36. These strategies suggest an improvement • Hypothesized there were additional skills involved in some of the compose by addition problems • A new student model (better BIC value) suggests the splitting the skill.

  37. Redesign based on Discovered Model Our discovery suggested changes needed to be made to the tutor • Resequencing – put problems requiring fewer skills first • Knowledge Tracing – adding new skills • Creating new tasks – new problems • Changing instructional messages, feedback or hints

  38. Study : Current tutor is control Current fielded tutor only uses scaffolded problems

  39. Study: Treatment Scaffolded, given areas, plan-only, & unscaffolded Isolate practice on problem decomposition

  40. Study Results Instructional time (minutes) by step type Post-test % correct by item type Much more efficient & better learning on targeted decomposition skills

  41. Translational Research Feedback Loop Design Discover Deploy Data

  42. Can a data-driven process be automated & brought to scale? Yes! Combine Cognitive Science, Psychometrics, Machine Learning … Collect a rich body of data Develop new model discovery algorithms, visualizations, & on-line collaboration support

  43. DataShop’s “leaderboard” ranks discovered cognitive models100s of datasets coming from ed tech in math, science, & language Some models are machine generated (based on human-generated learning factors) Some models are human generated

  44. Metrics for model prediction • AIC & BIC penalize for more parameters, fast & consistent • 10 fold cross validation • Minimize root mean squared error (RMSE) on unseen data

  45. Automated search for better models Learning Factors Analysis (LFA)(Cen, Koedinger, & Junker, 2006) • Method for discovering & evaluating cognitive models • Finds model “Q matrix” that best predicts student learning data • Inputs • Data: Student success on tasks over time • Factors hypothesized to explain learning • Outputs • Rank order of most predictive Q matrix • Parameter estimates for each

  46. Simple search process example: modifying Q matrix by input factor to get new Q’ matrix • Q matrix factor Sub split by factor Neg-result • Produces new Q matrix • Two new KCs (Sub-Pos & Sub-Neg) replace old KC (Sub) • Redo opportunity counts

  47. LFA: Best First Search Process • Search algorithm guided by a heuristic: AIC • Startwith single skillcog model (Q matrix) Cen, H., Koedinger, K., Junker, B. (2006).  Learning Factors Analysis: A general method for cognitive model evaluation and improvement. 8th International Conference on Intelligent Tutoring Systems.

  48. Scientist “crowd”sourcing: Feature input comes “for free” Scientist generated models Union of all hypothesized KCs in human generated models

  49. Validating Learning Factors Analysis Koedinger, McLaughlin, & Stamper (2012). Automated student model improvement. In Proceedings of the Fifth International Conference on Educational Data Mining. [Conference best paper.] Discovers better cognitive models in 11 of 11 datasets …

  50. Data from a variety of educational technologies & domains Statistics Online Course English Article Tutor Algebra Cognitive Tutor Numberline Game

More Related