1 / 23

Doing More with Less : Student Modeling and Performance Prediction with Reduced Content Models

Doing More with Less : Student Modeling and Performance Prediction with Reduced Content Models. Yun Huang, University of Pittsburgh Yanbo Xu , Carnegie Mellon University Peter Brusilovsky , University of Pittsburgh. This talk….

tanith
Download Presentation

Doing More with Less : Student Modeling and Performance Prediction with Reduced Content Models

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Doing More with Less : Student Modeling and Performance Prediction with Reduced Content Models Yun Huang, University of Pittsburgh YanboXu, Carnegie Mellon University Peter Brusilovsky, University of Pittsburgh

  2. This talk… • What? More effective student modeling and performance prediction • How? A novel framework reducing content model without loss of quality • Why? Better and cheaper • Reduced to 10%~20% while maintaining or improving performance (up to 8% better AUC) • Beat expert based reduction

  3. Outline • Motivation • Content Model Reduction • Experiments and Results • Conclusion and Future Work

  4. Motivation • In some domains and some types of learning content, each content problem (item) is related to largenumber of domain concepts (Knowledge Component, KCs) • It complicates modeling due to increasing noise and decreasing efficiency • We argue that we only need a subset of the most important KCs!

  5. Content model • The focus of this study: Java • Each problem involves a complete program and relates to many concepts • Original content model • Each problem is indexed by a set of Java concepts from ontology • In our context of study, number of concepts per problem can range from 9 to 55!

  6. An example of original content model • class definition • static method • public class • public method • void method • String array • int type variable declaration • int type variable initialization • for statement • assignment • increment • multiplication • less or equal • nested loop

  7. Challenges • Select best concepts to model problems • Traditional feature selection focuses on selecting a subset of features for all datapoints(a domain). item level not domain level

  8. Our intuitions of reduction methods Intuition 1 “for statement” appears 2 times in this problem -- it should be important for this problem! • Three types of methods from different information sources and intuitions: “assignment” appears in a lot of problems -- it should be trivial for this problem! Intuition 2: When “nested loops” appears, students always get it wrong -- it should be important for this problem! Intuition 3: Expert labeled “assignment”, “less than” as prerequisite concepts, while “nested loops”, “for statement” as outcome concepts --- outcome concepts should be the important ones for current problem!

  9. Reduction Methods • Content-based methods • A problem = a document, a KC = a word • Use IDFand TFIDF keyword weighting approach to compute KC importance score. • Response-based Method • Train a logistic regression (PFA) to predict student response • Use the coefficient representing the initial easiness (EASINESS-COEF) of a KC. • Expert-based Method Use only the OUTCOME concepts as the KCs for an item.

  10. Item-level ranking of KC importance • For each method, we define SCORE function assigning a score to a KC in an item • The higher the score, the more important a KC is in an item. • Then, we do item-level ranking: a KC's importance can be differentiated • by different score values, or/and • by its different ranking positions in different items

  11. Reduction Sizes • What is the best number of KCs each method should reduce to? • Reducing non-adaptively to items (TopX): Select x KCs per item with the highest importance scores. • Reducing adaptively to items (TopX%): Select x% KCs per item with the highest importance scores

  12. Evaluating Reduction on PFA and KT • We evaluate by the prediction performance of two popular student modeling and performance prediction models • Performance Factor Analysis (PFA): logistic regression model predicting student response • Knowledge Tracing (KT): Hidden Markov Models predicting student response and inferring student knowledge level *We select a variant that can handle multiple KCs.

  13. Outline • Motivation • Content Model Reduction • Experiments and Results • Conclusion and Future Work

  14. Tutoring System Collected from JavaGuide, a tutor for learning Java programming. Java code Students give values for a variable or the output Each question is generated from a template, and students can try multiple attempts

  15. Experimental Setup • Dataset • 19, 809 observations, about 69.3% correct • 132 students on 94 question templates (items) • A problem is indexed into 9 ~ 55 KCs, 124 KCs in total • Classification metric: Area Under Curve (AUC) • 1: perfect classifier, 0.5: random classifier • Cross-validation: Two runs of 5-fold CV where in each run 80% of the users are in train, and the remaining are in test. • We list the mean AUC on test sets across the 10 runs, and use Wilcoxon Signed Ranks Test (alpha = 0.05) to test AUC comparison significance.

  16. Reduction v.s. original on PFA • Flat (or roughly in bell shapes) with fluctuations • Reduction to a moderate size can provide comparableor even better prediction than using original content models. • Reduction could hurtif the size goes too small (e.g. < 5), possibly because PFA was designed for fitting items with multiple KCs.

  17. Reduction v.s. original on KT • Reduction provides gain ranging a much bigger span and scale! • KT achieves the best performance when the reduction size is small: it may bemore sensitive than PFA to the size! • Our reduction methods have selected promising KCs that are the important ones for KT making predictions!

  18. Automatic v.s. expert-based (OUTCOME) reduction method • IDF and TFIDF can be comparable to or outperform OUTCOME method! • E-COEF provides much gain on KT than PFA, suggesting PFA coefficients can provide useful extra information for reducing the KT content models. (+/−: signicantly better/worse than OUTCOME, : the optimal mean AUC)

  19. Outline • Motivation • Content Model Reduction • Experiments and Results • Conclusion and Future Work

  20. “Everything should be made as simple as possible, but not simpler.” -- Albert Einstein

  21. Conclusion • “Content model should be made as simple as possible, but not simpler.” • Given the proper reduction size, reduction enables prediction performance better! • Different model reacts to reduction differently! • KT is more sensitive to reduction than PFA • Different models achieve the best balance between model complexity and model fitin different ranges • We are the first to explore reduction extensively! • More ideas for selecting important KCs? • Larger datasets? • Other domains?

  22. Acknowledgement • Advanced Distributed Learning Initiative (http://www.adlnet.gov/). • LearnLab 2013 Summer School at CMU(Dr. Kenneth R. Koedinger, Dr. Jose P. Gonzalez-Brenes, Dr. Zachary A. Pardosfor advising and initiating the project)

  23. Thank you for listening !

More Related