Worcester Polytechnic Institute
This presentation is the property of its rightful owner.
Sponsored Links
1 / 101

Worcester Polytechnic Institute Introduces PowerPoint PPT Presentation


  • 53 Views
  • Uploaded on
  • Presentation posted in: General

Worcester Polytechnic Institute Introduces. Learn about research involving this online formative assessment system and hear about how it is used in classrooms. In this Webinar you will hear from:.

Download Presentation

Worcester Polytechnic Institute Introduces

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Worcester polytechnic institute introduces

Worcester Polytechnic Institute

Introduces

Learn about research involving this online formative assessment system and hear about how it is used in classrooms.

In this Webinar you will hear from:

Courtney Mulcahy, an 8th grade teacher who uses ASSISTment in her formative assessment process every day.

Neil Heffernan, a computer science professor about his research using this tool.

Matt Militello, a professor of school leadership on how school leaders can use such a system.

Cristina Heffernan, a professional developer who trains teachers to use the system and incorporate it into their classrooms.

And Russ Rapose, District curriculum coordinator for Milbury Public Schools


If old mcdonald had an assessment

If Old McDonald Had An Assessment…

Matthew Militello

Assistant Professor


The press the push

The Press… The Push

The Press:

  • More than ever we are TOLD, MANDATED, and SANCTIONED to use data to inform education decisions.

  • Specifically, assessment data. Means state tests.

    The Push:

  • These data are not timely, not informative for teachers; That is, these data points are autopsy data.

  • TMI: Too Much Information.


In search of the right data

In Search of the Right Data!

Central Office

Student/School Growth

School Leaders

Teacher Effectiveness

Teachers

Student Cognition


Formative assessment systems fas

Formative Assessment Systems (FAS)

There is Demand…. Supply Followed

Multi-Million Dollar Industry- Cost usually based on per student use

Not all FAS are the SAME!


How do districts select support and use fas

How Do Districts Select, Support, and Use FAS?

  • Three DistrictsThree FAS

  • Technical FeaturesImplementation

    • Actual Use


Framework

Framework


Framework1

Framework

Fit is defined by the INTENT of the districts selection (e.g. teacher use (student-diagnostic data) or district use (student growth model data) and the PURPOSE of the assessment system—including validity.


Framework2

Framework

Readiness is definedas: (1) building capacity in regard to both knowledge of the assessment product and (2) the motivation to use the product in practice.


Framework3

Framework

Coherence is how educators used the assessment systems fit into the context and the standard operating procedures of the district. For instance, the creation of trigger mechanisms from review of data reports.


Woodbury schools

Woodbury Schools

FAS-

NWEA-MAP

Good fit for Central Office,

but not teachers

Growth data forced on

daily practice

Teachers ill-equipped


Can t get beyond fit

Can’t Get Beyond FIT!


Can t get beyond fit1

Can’t Get Beyond FIT!


Can t get beyond fit2

Can’t Get Beyond FIT!


Needle in a haystack a fas with real time student level data

Needle in a Haystack:A FAS with Real-time, Student-Level Data!

  • Fit! Finally, Formative Assessments for TEACHERS

    • Technical Features Tutoring System for Real-Time Data and Diagnostic Data: PURPOSE

    • Teacher-Level Use- Desire For This Data: INTENT


Worcester polytechnic institute introduces

Using Intelligent Tutoring Systems to Help Math Teachers in the Formative Assessment Process. Prof. Neil HeffernanWorcester Polytechnic Institute


Who am i

Who Am I?

  • Teacher for America : teacher of middle school math and science in Baltimore City

  • PhD in Computer Science and Cognitive Science

    • Licensed to Carnegie Learning Inc.

  • Tenure professor at Worcester Polytechnic

  • Grant funded research on developing interventions


Quick project history 2003

Quick Project History - 2003

Teacher are being asked to use data to inform instruction, but every minute in the classroom spent on assessment is a minute lost to instruction.

US Department of Education (2003) Funded us to build a assessment system for teachers that did not take away teacher’s time.


Worcester polytechnic institute introduces

Lindeman, Carolina Ruiz, George Heineman, Murali Mani, Ryung Kim and others 30 others


Definitions

Definitions

A diagnostic assessment is an assessment designed to uncover any misconceptions or gaps in knowledge or skills of individual students so that teachers can base their instruction on the current state of students’ knowledge, skills, or understanding.

Formative assessment is a process used by teachers and students during instruction that provides feedback to adjust ongoing teaching and learning to improve students’ achievement of intended instructional outcomes.


Research questions for us dept of ed

Research Questions for US Dept of Ed

1.Does ASSISTments work as an assessment tool?

  • Reliably predict state test performance?

  • Reliably advise teachers & students on what knowledge to focus on?

    2.Does ASSISTments help enhance student learning?

  • Does the System effectively teach as it assesses?

  • Can teachers use the advice of this System to lead to higher student learning?


The assistment system

The ASSISTment System

  • A web-based assessment system, designed to collect formative assessment data on student math skills

  • Usage has grown as our content has grown.

  • Currently, thousands of students use the system.


You just saw one assistment

You Just Saw One ASSISTment

  • A question within this “tutor” is what we call an ASSISTment .

  • We have 2,000 such ASSISTments for 7th and 8th grade. (66 hours of instruction)

  • We have another 1,000 for 6th grade.

  • We have 1000 for 9-10th grade math,

  • There are Three other types of content.

    • Teacher created questions

    • Mastery Learning questions (track 130 skills in 8th grade math)

    • Connected Math matching units (500 ASSISTments with tutoring)


Many different types of reports

Many Different Types of Reports

State Test Report

Knowledge component reports (based upon 130 middle school skills)

Item Report (e.g. for small formative assessments)

Mastery Learning Report (which students have reached mastery?)


Worcester polytechnic institute introduces

Names Obscured on Purpose


Report to a teacher on their own state test

Report to a Teacher on Their Own State Test


Research questions for us dept of ed1

1.Does ASSISTments work as an assessment tool?

Reliably predict state test performance?

Reliably advise teachers & students on what knowledge to focus on?

2.Does ASSISTments help enhance student learning?

Does the System effectively teach as it assesses?

Can teachers use the advice of this System to lead to higher student learning?

Research Questions for US Dept of Ed


Report to a teacher on their state

Report to a Teacher on Their State


Predict student knowledge as measured by state test scores

Predict Student Knowledge as Measured by State Test Scores

Feng, M., Beck, J,. Heffernan, N. & Koedinger, K. (2008)  Can an Intelligent Tutoring System Predict Math Proficiency as Well as a Standardized Test? In Baker & Beck (Eds.). Proceedings of the 1st International Conference on Education Data Mining. Montreal, Canada.  pp.107-116. ISBN - 13: 9780615306292 

Feng, M., Heffernan, N. T., & Koedinger, K. R. (2006b). Addressing the testing challenge with a Web-based e-assessment system that tutors as it assesses. Proceedings of the Fifteenth International World Wide Web Conference (WWW-06). New York, NY: ACM Press. ISBN:1-59593-332-9.  pp. 307-316. [Nominated for Best Student Paper]  

Feng, M., Heffernan, N.T., & Koedinger, K.R. (2009). Addressing the assessment challenge in an Intelligent Tutoring System that tutors as it assesses. The Journal of User Modeling and User-Adapted Interaction.Vol 19: p243-266.

  • We have state scores from 2004 to 2009 for thousands of students.

    • We showed our error in predicting the state test is as low as the test itself.

    • We showed we can even predict their 10th grade math score from their 8th grade data.

    • We have shown we can more reliably predict their state test score if we use how much assistance they need.


Research questions for us dept of ed2

Research Questions for US Dept of Ed

1.Does ASSISTments work as an assessment tool?

  • Reliably predict state test performance?

  • Reliably advise teachers & students on what knowledge to focus on?

    2.Does ASSISTments help enhance student learning?

  • Does the System effectively teach as it assesses?

  • Can teachers use the advice of this System to lead to higher student learning?


We have shown that our transfer model is good enough to improve assessment

We have shown that our transfer model is good enough to improve assessment.

  • Using finer-grained skill model can improve accuracy of the assessment of student knowledge.

  • Feng, M., Heffernan, N. T., Mani, M., & Heffernan, C. (2007). Assessing students’ performance longitudinally: Item difficulty parameter vs. skill learning tracking.  The National Council on Educational Measurement 2007 Annual Conference, Chicago.

  • Feng , M, Heffernan, N., Heffernan, C. & Mani, M. (2009) Using Mixed-Effects Modeling to Analyze Different Grain-Sized Skill Models. Journal of Technology, Learning, and Assessment. (Featured Article of the year!)

  • Pardos, Z. A., Heffernan, N. T., Anderson, B. & Heffernan, C. (2007). The effect of model granularity on student performance prediction using Bayesian networks.  The International User Modeling Conference 2007.


Research questions for us dept of ed3

Research Questions for US Dept of Ed

1.Does ASSISTments work as an assessment tool?

  • Reliably predict state test performance?

  • Reliably advise teachers & students on what knowledge to focus on?

    2.Does ASSISTments help enhance student learning?

  • Does the System effectively teach as it assesses?

  • Can teachers use the advice of this System to lead to higher student learning?


Can teachers use the advice of this system to lead to higher student learning

Can teachers use the advice of this system to lead to higher student learning?

  • Quasi Experiment

    • Results: Student in schools that used ASSISTments did better on state test than students in control schools.

    • Results: Gaps were closed in achievement

  • Recruitment for a large 60 school field trial is being organized.

    $25 million dollar grant

    “i3” Investing in Innovation Fund grant line

    • If you are interested talk to me.


Stories of formative assessment

Mastery Learning

Stories of Formative Assessment

Online Exit Card

Courtney Mulcahy, an 8th grade teacher who uses ASSISTment in her formative assessment process every day.

Book Homework Online


The formative assessment cycle

The Formative Assessment Cycle

Objective

Teacher

Student

Lesson

Assessment


The cycle can move through two or three iterations depending on the situation

The Cycle can move through two or three iterations depending on the situation.

Objective

Lesson

Objective

Lesson

Objective

Assessment

Lesson

Assessment

Assessment


Worcester polytechnic institute introduces

Online Exit Card

Objective

Lesson

Assessment

  • Find the side of a triangle using the Pythagorean Theorem

  • Use Connected Math Book

  • Three question Problem Set online on ASSISTment


Worcester polytechnic institute introduces

Assessment

  • Three question Problem Set online on ASSISTment


Worcester polytechnic institute introduces

Teacher

Student

Assessment

There were 43 students on this item report, not all are shown.

  • Three question Problem Set online on ASSISTment


Worcester polytechnic institute introduces

CORRECT

a2+b2 = c2

122+x2 = 142

144 + x2 = 196

x2 = 54

x = 7.21

INCORRECT

a2+b2 = c2

122+142 = x2

144 + 196 = x2

340= x2

18.44 = x


Worcester polytechnic institute introduces

Online Exit Card

Objective

Lesson

Assessment

Objective

Lesson

  • Find the side of a triangle using the Pythagorean Theorem

  • Use Connected Math Book

  • Three question Problem Set online on ASSISTment

  • Find the leg of a right triangle using the Pythagorean Theorem.

  • Discussed the different elements of the Pythagorean Theorem.


Worcester polytechnic institute introduces

Mastery Learning

Objective

Lesson

Assessment

  • Find Mean Median Mode Range from data in a Stem and Leaf Plot

  • Lessons in 7th Grade

  • Mastery Learning Questions on Mean Median, Mode and Range from Stem and Leaf plots


Worcester polytechnic institute introduces

Students work until they get 3 (or another chosen number) right in a row. The are very involved in their learning.

Assessment

  • Mastery Learning Questions on Mean Median, Mode and Range from Stem and Leaf plots


Worcester polytechnic institute introduces

Mastery Learning

Objective

Lesson

Assessment

Objective

Lesson

  • Find Mean Median Mode Range from data in a Stem and Leaf Plot

  • Lessons in 7th Grade

  • Mastery Learning Questions on Mean Median, Mode and Range from Stem and Leaf plots

  • Find Mean Median Mode Range from data in a Stem and Leaf Plot

  • Tutoring in ASSIST-ment.


Worcester polytechnic institute introduces

Students gets instant feedback and support.

Lesson

  • Tutoring in ASSIST-ment.


Worcester polytechnic institute introduces

Mastery Learning

Objective

Lesson

Assessment

Assessment

Objective

Lesson

  • Find Mean Median Mode Range from data in a Stem and Leaf Plot

  • Lessons in 7th Grade

  • Mastery Learning Questions on Mean Median, Mode and Range from Stem and Leaf plots

  • Students must work until they get 3 right in a row.

  • Can give a static post test.

  • Find Mean Median Mode Range from data in a Stem and Leaf Plot

  • Tutoring in ASSIST-ment.


Worcester polytechnic institute introduces

Assessment

  • Students must work until they get 3 right in a row.

  • Can give a static post test.


Worcester polytechnic institute introduces

Objective

Lesson

Assessment

  • Students will be able to solve questions involving exponential functions.

  • Use CMP Book

  • Homework from the Book, with questions online.

  • One question asks about linear equations.

Book Homework Online


Worcester polytechnic institute introduces

Homework was Page 13 # 22 and 23 Page 16 # 34


Worcester polytechnic institute introduces

Assessment

  • Homework from the Book, with questions online.

  • One question asks about linear equations.

From Connected Mathematics Project Page 16 of Growing Growing Growing.


Worcester polytechnic institute introduces

This data informed both teacher and students of the need for the next objective

Assessment

  • Homework from the Book, with questions online.

  • One question asks about linear equations.


Worcester polytechnic institute introduces

Objective

Lesson

Assessment

Objective

Lesson

Assessment

  • Students will be able to solve questions involving exponential functions.

  • Use CMP Book

  • Homework from the Book, with questions online.

  • One question asks about linear equations.

  • While studying exponential functions students will be able to answer questions about linear relationships.

  • Read what is being asked carefully.

  • Discuss the linear equation question in class.

  • Next nights homework had multiple questions about comparing linear and exponential functions.

Book Homework Online


Worcester polytechnic institute introduces

Assessment

  • Next nights homework had multiple questions about comparing linear and exponential functions.

From Connected Mathematics Project Page 14 of Growing Growing Growing.


Worcester polytechnic institute introduces

Homework was Page 14 – 17 # 24 – 30 and 35 - 38


Stories of formative assessment1

Mastery Learning

Stories of Formative Assessment

Online Exit Card

Book Homework Online


Does it teach as it assesses

Does It teach as It Assesses?

  • We have done 6 different randomized controlled experiment, and ASSISTments consistently leads to statistically meaningful learning results.

  • Two examples

    • A homework study

    • A classroom study


Comparing traditional paper and pencil homework with web based homework

Comparing Traditional Paper and Pencil Homework with Web-Based Homework

Mendicino, M., Razzaq, L. & Heffernan, N. T. (2009). Improving Learning from Homework Using Intelligent Tutoring Systems.Journal of Research on Technology in Education (JRTE). Published by the International Society For Technology in Education (ISTE). Spring 2009 issue.  41:3 p 331-346.

Leena Razzaq, Michael Mendicino & Neil Heffernan

Worcester Polytechnic Institute & West Virginia University

Intelligent Tutoring Systems 2008


Digital divide is real

Digital Divide Is Real.

  • Worcester is an urban district with only about 70% of homes with Internet.

  • Students already get weekly homework that requires the Internet.

  • Worcester needs supports for students who don’t have computers at home.

    • School libraries

    • Printable worksheet

    • Before school and homeroom

  • These studies prepare us for a future where kids can get the benefit of computers.


Maine lap top initiative

Maine Lap Top Initiative

Every student in the state get a lap top. Over half the districts let students take the lap top home.

State scores have gone up in English but stayed flat in Math. (Prof. David Silvernail)

Richmond Virginia (Henrico Country)


Is homework important

Is Homework Important?

  • Etta Kralovec, author of The End of Homework: How Homework Disrupts Families, Overburdens Children, and Limits Learning

    • "When work goes home, teachers have little control over who does the work. Teachers [are] unable to scaffold new knowledge for students, and [are] unaware of each student's true educational progress.”

  • http://www.hno.harvard.edu/gazette/2000/09.28/homework.html

  • Hattie(1999) meta-analysis across over 100 studies showed that homework has a .4 effect size.


Literature review other comparisons of pph to tutoring systems

Literature Review: Other Comparisons of PPH to Tutoring Systems

  • Little prior work but the best of it is at the college level

    • Mastering Physics – (Warnakulasooriya & Pritchard, 2005)

    • Andes (VanLehn et al., 2005) produced a mean effect size of 0.6 over paper-and-pencil homework.


Possible advantages of web based homework

Possible Advantages of Web Based Homework

  • Better learning for students

    • Immediate feedback to students with tutoring

  • More help for teachers

    • Time saver

    • Common errors analyzed

    • Formative assessment—Teachers use the data to guide instruction more effectively.

    • Homework help—Students take homework more seriously because they know it will be graded and the grade recorded.


Possible disadvantages

Possible Disadvantages

  • Many of these systems do not take students’ work into consideration.

  • Students may try to do more math in their heads.

  • Teacher may spend less time looking at student work.

  • Cheating may be easier among students.


Experimental design

Experimental Design

  • Purpose: to determine if students can learn more by doing their math homework with the ASSISTment System than when doing traditional paper-and-pencil homework

  • Control: Student did their work on paper. The next day teachers did a 10 minute review of the questions. Teachers used the same feedback the ASSISTments system gave.

  • Materials:

  • Pretest and Posttest of 10 items

  • Homework was also 10 questions but with different numbers.

  • Topic A: Number Scene Topic B: Mixed


Sample number sense

Sample “Number Sense”

1. Which of the following is closest to the product of 397.8 * 10.3?

o A) 3,000

o B) 30,000

o C) 400

o D) 4,000

2. Which of the following shows the numbers in order from least to greatest?

o A) 0.452, 0.51, 0.432

o B) 0.452, 0.432, 0.51

o C) 0.432, 0.51, 0.452

o D) 0.432, 0.452, 0.51

4. What is 15/60 as a percent?

5. Write 10/25 as a decimal.

6. What is the value of the following expression?

7 * 6 + 3

7. Destinee has a total of 12 fish in her aquarium. Exactly 9 of the fish are goldfish. What percent of the fish in the aquarium are goldfish?


Experimental design continued

Experimental Design Continued

We did a repeated measures analysis where each student participated in each condition but in different orders. This allows us to deal with the large variation in different students learning rates. We eliminate the need to model the variation between teachers.

We counter-balanced the homework assignments in a Latin square design. The order of seeing the topics were randomly assigned to different classrooms.

West Virginia- 4 math teachers


Experimental design continued1

Experimental Design Continued


Experimental design continued2

Experimental Design Continued

To deal with the fact that not everyone had internet at home we only analyzed students that did their homework both nights in both conditions. Left us with 28 students

You can’t ask students if they internet at home

Its embarrassing

They don’t know if they will be able to have access


Results

Results

  • Students learned significantly in both conditions

  • Students learned reliably more with ASSISTments (2.3 problems verse 1.14 problems) with 0.6 effect size (p = 0.05).


Results not due to a few students

Results Not Due to a Few Students

Top 5 gain scores were in the web based homework condition.

Our worst gain score is in our WBH condition.


Conclusions

Conclusions

  • Students do learn significantly (.6 standard deviations more) more from using the ASSISTment System compared to traditional paper-and-pencil homework.


Implications

Implications

  • Results of this study are promising in that kids can learn a good deal more with homework.

  • Only study of its kind I am aware of in K12 that isolated the benefit of web based homework.

  • Could be important to policy makers, particularly considering the popularity of one-to-one computing initiatives

  • Will become be more relevant as the digital divide narrows.


Does an intelligent tutor lead to more learning than traditional classroom practice

Does an Intelligent Tutor lead to more learning than traditional classroom practice?

  • Most math classrooms give a lecture and then a period of time for practice.

  • We showed that kids learn a lot more when getting immediate feedback from the ITS than the “business as usual” control.

  • Results: Students learned more when using the computer compared to the traditional classroom.

  • Razzaq, L., Mendicino, M. & Heffernan, N. (2008) Comparing classroom problem-solving with no feedback to web-based homework assistance. In Woolf, Aimeur, Nkambou and Lajoie (Eds.) Proceeding of the 9th International Conference on Intelligent Tutoring Systems. pp. 426 -437. Springer-Verlag: Berlin.


Research questions for us dept of ed4

Research Questions for US Dept of Ed

1.Does ASSISTments work as an assessment tool?

  • Reliably predict state test performance?

  • Reliably advise teachers & students on what knowledge to focus on?

    2.Does ASSISTments help enhance student learning?

  • Does the System effectively teach as it assesses?

  • Can teachers use the advice of this System to lead to higher student learning?


Can teachers use the advice of this system to lead to higher student learning1

Can teachers use the advice of this system to lead to higher student learning?

  • Quasi Experiment

    • Results: Student in schools that used ASSISTments did better on state test than students in control schools.

    • Results: Gaps were closed in achievement

  • Recruitment for a large 60 school field trial is being organized.

    $25 million dollar grant

    “i3” Investing in Innovation Fund grant line

    • If you are interested talk to me.


We have users in over a dozen towns

We Have Users in Over a Dozen Towns

Worcester, Auburn, Fitchburg—Classroom formative assessment

Shrewsbury—Nightly homework & Mastery Learning

Grafton—Open response explanation writing

Milbury—Benchmark Assessment every 9 weeks, teacher reports

Maine: Mastery Learning for homework


What have i not told you about

What have I not told you about?

Cristina Heffernan

  • Parent Notification System

  • How teacher and schools put in their own content.

  • Stick around for the next 30 minutes to learn how to

    • Create our own account,

    • Let kids log in, and

    • Build your own content.


Thank you

Thank you


Worcester polytechnic institute introduces

Extra slides


Cognitive tutor carnegie learning inc

Cognitive Tutor – Carnegie Learning Inc.


Replicated field studies

Replicated Field Studies

  • Controlled, full year classroom experiments

  • Replicated over 3 years in urban schools

  • In Pittsburgh & Milwaukee

  • Results:

    50-100% better on problem solving & representation use.

    15-25% better on standardized tests.

Koedinger, Anderson, Hadley, & Mark (1997). Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education, 8.


An intelligent tutoring system success

An Intelligent Tutoring System Success

  • Cognitive Tutor Algebra

  • Most widely used ITS

    • 1000+ schools across the country

    • Marketed by local spin-off company Carnegie Learning

  • “Exemplary Curricula” by US Dept of Ed. In What Works Clearing House

  • Most cited Journal of AI-ED paper

    • Koedinger, Anderson, Hadley, & Mark (1997). Intelligent tutoring goes to school in the big city.

. . .


  • Login