Using mixed effects modeling to compare different grain sized skill models
Download
1 / 19

Using Mixed-Effects Modeling to Compare Different Grain-Sized Skill Models - PowerPoint PPT Presentation


  • 129 Views
  • Uploaded on

Using Mixed-Effects Modeling to Compare Different Grain-Sized Skill Models. Mingyu Feng, Worcester Polytechnic Institute Neil T. Heffernan, Worcester Polytechnic Institute Murali Mani, Worcester Polytechnic Institute Cristina Heffernan , Worcester Public Schools. The “ASSISTment” System.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Using Mixed-Effects Modeling to Compare Different Grain-Sized Skill Models' - kiana


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Using mixed effects modeling to compare different grain sized skill models

Using Mixed-Effects Modeling to Compare Different Grain-Sized Skill Models

Mingyu Feng, Worcester Polytechnic Institute

Neil T. Heffernan, Worcester Polytechnic Institute

Murali Mani, Worcester Polytechnic Institute

Cristina Heffernan, Worcester Public Schools


The assistment system
The “ASSISTment” System Grain-Sized Skill Models

  • An e-assessment and e-learning system that does both ASSISTing of students and assessMENT (movie)

    • Massachusetts Comprehensive Assessment System “MCAS”

  • Web-based system built on Common Tutoring Object Platform (CTOP) [1]

We are giving away accounts!

[1] Nuzzo-Jones., G. Macasek M.A., Walonoski, J., Rasmussen K. P., Heffernan, N.T., Common Tutor Object Platform, an e-Learning Software Development Strategy, WPI technical report. WPI-CS-TR-06-08.

AAA’06-W06


Assistment
ASSISTment Grain-Sized Skill Models

Geometry

  • We break multi-step problems into “scaffolding questions”

  • “Hint Messages”: given on demand that give hints about what step to do next

  • “Buggy Message”: a context sensitive feedback message

  • Skills

    • The state reports to teachers on 5 areas

    • We seek to report on more and finer grain-sized skills

  • Demo (two triangles problem)

(Demo/movie)

The original question

a. Congruence

b. Perimeter

c. Equation-Solving

The 1st scaffolding question

Congruence

The 2nd scaffolding question

Perimeter

A buggy message

A hint message

AAA’06-W06


How was the skill models created
How was the Skill Models Created Grain-Sized Skill Models

AAA’06-W06


How was the skill models created1

[2] Grain-Sized Skill ModelsPardos, Z. A., Heffernan, N. T., Anderson, B., & Heffernan C. (2006). Using Fine-Grained Skill Models to Fit Student Performance with Bayesian Networks. Workshop in Educational Data Mining held at the Eight International Conference on Intelligent Tutoring Systems. Taiwan. 2006.

How was the Skill Models Created

Multi-mapped model (WPI-5) vs. single-mapped model (MCAS-5) ?

AAA’06-W06


Previous work on skill models
Previous Work on Skill Models Grain-Sized Skill Models

  • Fine grained skill models in reporting

    • Teachers get reports that they think are credible and useful. [3]

[3] Feng, M., Heffernan, N.T. (in press). Informing Teachers Live about Student Learning: Reporting in the Assistment System. To be published in Technology, Instruction, Cognition, and Learning Journal Vol. 3. Old City Publishing, Philadelphia, PA. 2006

AAA’06-W06


AAA’06-W06 Grain-Sized Skill Models


AAA’06-W06 Grain-Sized Skill Models


Previous work on skill models1
Previous Work on Skill Models Grain-Sized Skill Models

  • Tracking skill performance over time [4][5]

Number Sense

[4] Feng, M., Heffernan, N.T., & Koedinger, K.R. (2006). Addressing the Testing Challenge with a Web-Based E-Assessment System that Tutors as it Assesses. Proceedings of the Fifteenth International World Wide Web Conference. pp. 307-316. ACM Press: New York, NY. 2006.

[5]Feng, M., Heffernan, N.T., & Koedinger, K.R. (2006). Predicting state test scores better with intelligent tutoring systems: developing metrics to measure assistance required. In Ikeda, Ashley & Chan (Eds.). Proceedings of the Eight International Conference on Intelligent Tutoring Systems. Springer-Verlag: Berlin. pp. 31-40. 2006.

AAA’06-W06


AAA’06-W06


Research questions
Research Questions Grain-Sized Skill Models

  • RQ1: Would adding response data to scaffolding questions help us do a better job of tracking students’ knowledge?

  • RQ2: How does the finer-grained skill model (WPI-78) do on estimating external test scores comparing to the skill model with only 5 categories (WPI-5) and the one even with only one category (WPI-1)?

  • RQ3:Does introducing item difficulty information help to build a better predictive model?

AAA’06-W06


Data source
Data Source Grain-Sized Skill Models

  • 497 students of two middle schools

  • Students used the ASSISTment system every other week from Sep. 2004 to May 2005

  • Real state test score in May 2005

  • Item level online data

    • students’ binary response (1/0) to items that are tagged in different skill models

  • Some statistics

    • Average usage: 7.3 days, Minimum usage: 6 days

    • 138,000 data points (43,000 original data points)

    • Average question answered

      • Original: 87, Scaffolding: 189

Online data of 700 8th grade students available for researchers! If you want access, talk to Neil Heffernan and Kenneth Koedinger.

AAA’06-W06


How is the data organized
How is the Data Organized? Grain-Sized Skill Models

AAA’06-W06


Approach

  • Predict State Test Scores Grain-Sized Skill Models

    • Identify skills associated with each test item in all skill models

    • Student full score = item fractional score (prob(response=1))

Approach

  • Fit mixed-effects logistic regression model on the longitudinal online data

    • using skills as a factor

    • predicting prob(response=1) on an item tagged with certain skill at certain time

    • The fitted model gives learning parameters (initial knowledge + learning rate) of each skill of individual student

  • Compare skill models by Mean Absolute Difference (MAD) and %Err (= MAD/full score)

AAA’06-W06


1 Grain-Sized Skill Models

1

Data Preprocessing Strategies

  • Scaffolding Credit

    • Scaffolding only shows in case of wrong answer to original

    • We assume correct responses to all scaffolding questions if a student correctly answered the original one

  • Partial Blame

    • Only blame the skill of the worst performance overall

AAA’06-W06


RQ1: Grain-Sized Skill ModelsWill Scaffolding Response Help?

  • Why?

    • Using more training data

    • Deal with credit-blame issue better

    • More “identifiability” per skill

    • Scaffolding questions provide valuable information [4][5][6][7]

Answer: Yes!

[6] Walonoski, J., Heffernan, N.T. (2006). Detection and Analysis of Off-Task Gaming Behavior in Intelligent Tutoring Systems. In Ikeda, Ashley & Chan (Eds.). Proceedings of the Eighth International Conference on Intelligent Tutoring Systems. Springer-Verlag: Berlin. pp. 382-391. 2006

[7] Walonoski, J., Heffernan, N.T. (2006).Prevention of Off-Task Gaming Behavior in Intelligent Tutoring Systems. In Ikeda, Ashley & Chan (Eds.). Proceedings of the Eighth International Conference on Intelligent Tutoring Systems. Springer-Verlag: Berlin. pp. 722-724. 2006.

AAA’06-W06


> Grain-Sized Skill Models

>

>

>

P-values of both Paired t-tests are below 0.05

RQ2: Does finer grained model predict better?

Is 12.12% any good for assessment purpose?

MCAS-simulation result: 11.12%

AAA’06-W06


Conclusion
Conclusion Grain-Sized Skill Models

  • Recall RQ1, RQ2.

  • Positive answer to both RQ1 and RQ2.

  • RQ3: Item difficulty was introduced as a factor to improve the predictive models. We ended up with better internally fitted models, but surprisingly no significant enhancement on the prediction of state test.

AAA’06-W06


Some of the ASSISTMENT TEAM (2004-2005) Grain-Sized Skill Models

* This research was made possible by the US Dept of Education, Institute of Education Science, "Effective Mathematics Education Research" program grant #R305K03140, the Office of Naval Research grant # N00014-03-1-0221, NSF CAREER award to Neil Heffernan, and the Spencer Foundation. Authors Razzaq and Mercado were funded by the National Science Foundation under Grant No. 0231773. All the opinions in this article are those of the authors, and not those of any of the funders.

Leena RAZZAQ*, Mingyu FENG, Goss NUZZO-JONES, Neil T. HEFFERNAN,

Kenneth KOEDINGER+, Brian JUNKER+, Steven RITTER, Andrea KNIGHT+,

Edwin MERCADO*, Terrence E. TURNER, Ruta UPALEKAR, Jason A. WALONOSKI

Michael A. MACASEK, Christopher ANISZCZYK, Sanket CHOKSEY, Tom LIVAK, Kai RASMUSSEN

Carnegie Learning


ad