Worcester Polytechnic Institute
Download
1 / 101

Worcester Polytechnic Institute Introduces - PowerPoint PPT Presentation


  • 102 Views
  • Uploaded on

Worcester Polytechnic Institute Introduces. Learn about research involving this online formative assessment system and hear about how it is used in classrooms. In this Webinar you will hear from:.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Worcester Polytechnic Institute Introduces' - amora


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Worcester Polytechnic Institute

Introduces

Learn about research involving this online formative assessment system and hear about how it is used in classrooms.

In this Webinar you will hear from:

Courtney Mulcahy, an 8th grade teacher who uses ASSISTment in her formative assessment process every day.

Neil Heffernan, a computer science professor about his research using this tool.

Matt Militello, a professor of school leadership on how school leaders can use such a system.

Cristina Heffernan, a professional developer who trains teachers to use the system and incorporate it into their classrooms.

And Russ Rapose, District curriculum coordinator for Milbury Public Schools


If old mcdonald had an assessment

If Old McDonald Had An Assessment…

Matthew Militello

Assistant Professor


The press the push
The Press… The Push

The Press:

  • More than ever we are TOLD, MANDATED, and SANCTIONED to use data to inform education decisions.

  • Specifically, assessment data. Means state tests.

    The Push:

  • These data are not timely, not informative for teachers; That is, these data points are autopsy data.

  • TMI: Too Much Information.


In search of the right data
In Search of the Right Data!

Central Office

Student/School Growth

School Leaders

Teacher Effectiveness

Teachers

Student Cognition


Formative assessment systems fas
Formative Assessment Systems (FAS)

There is Demand…. Supply Followed

Multi-Million Dollar Industry- Cost usually based on per student use

Not all FAS are the SAME!


How do districts select support and use fas
How Do Districts Select, Support, and Use FAS?

  • Three Districts Three FAS

  • Technical Features Implementation

    • Actual Use



Framework1
Framework

Fit is defined by the INTENT of the districts selection (e.g. teacher use (student-diagnostic data) or district use (student growth model data) and the PURPOSE of the assessment system—including validity.


Framework2
Framework

Readiness is definedas: (1) building capacity in regard to both knowledge of the assessment product and (2) the motivation to use the product in practice.


Framework3
Framework

Coherence is how educators used the assessment systems fit into the context and the standard operating procedures of the district. For instance, the creation of trigger mechanisms from review of data reports.


Woodbury schools
Woodbury Schools

FAS-

NWEA-MAP

Good fit for Central Office,

but not teachers

Growth data forced on

daily practice

Teachers ill-equipped





Needle in a haystack a fas with real time student level data
Needle in a Haystack:A FAS with Real-time, Student-Level Data!

  • Fit! Finally, Formative Assessments for TEACHERS

    • Technical Features Tutoring System for Real-Time Data and Diagnostic Data: PURPOSE

    • Teacher-Level Use- Desire For This Data: INTENT


Using Intelligent Tutoring Systems to Help Math Teachers in the Formative Assessment Process. Prof. Neil HeffernanWorcester Polytechnic Institute


Who am i
Who Am I? the Formative Assessment Process.

  • Teacher for America : teacher of middle school math and science in Baltimore City

  • PhD in Computer Science and Cognitive Science

    • Licensed to Carnegie Learning Inc.

  • Tenure professor at Worcester Polytechnic

  • Grant funded research on developing interventions


Quick project history 2003
Quick Project History - 2003 the Formative Assessment Process.

Teacher are being asked to use data to inform instruction, but every minute in the classroom spent on assessment is a minute lost to instruction.

US Department of Education (2003) Funded us to build a assessment system for teachers that did not take away teacher’s time.



Definitions
Definitions Kim and others 30 others

A diagnostic assessment is an assessment designed to uncover any misconceptions or gaps in knowledge or skills of individual students so that teachers can base their instruction on the current state of students’ knowledge, skills, or understanding.

Formative assessment is a process used by teachers and students during instruction that provides feedback to adjust ongoing teaching and learning to improve students’ achievement of intended instructional outcomes.


Research questions for us dept of ed
Research Questions for US Dept of Ed Kim and others 30 others

1. Does ASSISTments work as an assessment tool?

  • Reliably predict state test performance?

  • Reliably advise teachers & students on what knowledge to focus on?

    2. Does ASSISTments help enhance student learning?

  • Does the System effectively teach as it assesses?

  • Can teachers use the advice of this System to lead to higher student learning?


The assistment system
The Kim and others 30 othersASSISTment System

  • A web-based assessment system, designed to collect formative assessment data on student math skills

  • Usage has grown as our content has grown.

  • Currently, thousands of students use the system.


You just saw one assistment
You Just Saw One Kim and others 30 othersASSISTment

  • A question within this “tutor” is what we call an ASSISTment .

  • We have 2,000 such ASSISTments for 7th and 8th grade. (66 hours of instruction)

  • We have another 1,000 for 6th grade.

  • We have 1000 for 9-10th grade math,

  • There are Three other types of content.

    • Teacher created questions

    • Mastery Learning questions (track 130 skills in 8th grade math)

    • Connected Math matching units (500 ASSISTments with tutoring)


Many different types of reports
Many Different Types of Reports Kim and others 30 others

State Test Report

Knowledge component reports (based upon 130 middle school skills)

Item Report (e.g. for small formative assessments)

Mastery Learning Report (which students have reached mastery?)


Names Obscured on Purpose Kim and others 30 others


Report to a teacher on their own state test
Report to a Teacher on Kim and others 30 othersTheir Own State Test


Research questions for us dept of ed1

1. Does ASSISTments work as an assessment tool? Kim and others 30 others

Reliably predict state test performance?

Reliably advise teachers & students on what knowledge to focus on?

2. Does ASSISTments help enhance student learning?

Does the System effectively teach as it assesses?

Can teachers use the advice of this System to lead to higher student learning?

Research Questions for US Dept of Ed


Report to a teacher on their state
Report to a Teacher on Their State Kim and others 30 others


Predict student knowledge as measured by state test scores
Predict Student Knowledge as Measured by State Test Scores Kim and others 30 others

Feng, M., Beck, J,. Heffernan, N. & Koedinger, K. (2008)  Can an Intelligent Tutoring System Predict Math Proficiency as Well as a Standardized Test? In Baker & Beck (Eds.). Proceedings of the 1st International Conference on Education Data Mining. Montreal, Canada.  pp.107-116. ISBN - 13: 9780615306292 

Feng, M., Heffernan, N. T., & Koedinger, K. R. (2006b). Addressing the testing challenge with a Web-based e-assessment system that tutors as it assesses. Proceedings of the Fifteenth International World Wide Web Conference (WWW-06). New York, NY: ACM Press. ISBN:1-59593-332-9.  pp. 307-316. [Nominated for Best Student Paper]  

Feng, M., Heffernan, N.T., & Koedinger, K.R. (2009). Addressing the assessment challenge in an Intelligent Tutoring System that tutors as it assesses. The Journal of User Modeling and User-Adapted Interaction.Vol 19: p243-266.

  • We have state scores from 2004 to 2009 for thousands of students.

    • We showed our error in predicting the state test is as low as the test itself.

    • We showed we can even predict their 10th grade math score from their 8th grade data.

    • We have shown we can more reliably predict their state test score if we use how much assistance they need.


Research questions for us dept of ed2
Research Questions for US Dept of Ed Kim and others 30 others

1. Does ASSISTments work as an assessment tool?

  • Reliably predict state test performance?

  • Reliably advise teachers & students on what knowledge to focus on?

    2. Does ASSISTments help enhance student learning?

  • Does the System effectively teach as it assesses?

  • Can teachers use the advice of this System to lead to higher student learning?


We have shown that our transfer model is good enough to improve assessment
We have shown that our transfer model is good enough to improve assessment.

  • Using finer-grained skill model can improve accuracy of the assessment of student knowledge.

  • Feng, M., Heffernan, N. T., Mani, M., & Heffernan, C. (2007). Assessing students’ performance longitudinally: Item difficulty parameter vs. skill learning tracking.  The National Council on Educational Measurement 2007 Annual Conference, Chicago.

  • Feng , M, Heffernan, N., Heffernan, C. & Mani, M. (2009) Using Mixed-Effects Modeling to Analyze Different Grain-Sized Skill Models. Journal of Technology, Learning, and Assessment. (Featured Article of the year!)

  • Pardos, Z. A., Heffernan, N. T., Anderson, B. & Heffernan, C. (2007). The effect of model granularity on student performance prediction using Bayesian networks.  The International User Modeling Conference 2007.


Research questions for us dept of ed3
Research Questions for US Dept of Ed improve assessment.

1. Does ASSISTments work as an assessment tool?

  • Reliably predict state test performance?

  • Reliably advise teachers & students on what knowledge to focus on?

    2. Does ASSISTments help enhance student learning?

  • Does the System effectively teach as it assesses?

  • Can teachers use the advice of this System to lead to higher student learning?


Can teachers use the advice of this system to lead to higher student learning
Can teachers use the advice of this system to lead to higher student learning?

  • Quasi Experiment

    • Results: Student in schools that used ASSISTments did better on state test than students in control schools.

    • Results: Gaps were closed in achievement

  • Recruitment for a large 60 school field trial is being organized.

    $25 million dollar grant

    “i3” Investing in Innovation Fund grant line

    • If you are interested talk to me.


Stories of formative assessment

Mastery Learning student learning?

Stories of Formative Assessment

Online Exit Card

Courtney Mulcahy, an 8th grade teacher who uses ASSISTment in her formative assessment process every day.

Book Homework Online


The formative assessment cycle
The Formative Assessment Cycle student learning?

Objective

Teacher

Student

Lesson

Assessment


The cycle can move through two or three iterations depending on the situation
The Cycle can move through two or three iterations depending on the situation.

Objective

Lesson

Objective

Lesson

Objective

Assessment

Lesson

Assessment

Assessment


Online Exit Card on the situation.

Objective

Lesson

Assessment

  • Find the side of a triangle using the Pythagorean Theorem

  • Use Connected Math Book

  • Three question Problem Set online on ASSISTment


Assessment on the situation.

  • Three question Problem Set online on ASSISTment


Teacher on the situation.

Student

Assessment

There were 43 students on this item report, not all are shown.

  • Three question Problem Set online on ASSISTment


CORRECT on the situation.

a2+b2 = c2

122+x2 = 142

144 + x2 = 196

x2 = 54

x = 7.21

INCORRECT

a2+b2 = c2

122+142 = x2

144 + 196 = x2

340= x2

18.44 = x


Online Exit Card on the situation.

Objective

Lesson

Assessment

Objective

Lesson

  • Find the side of a triangle using the Pythagorean Theorem

  • Use Connected Math Book

  • Three question Problem Set online on ASSISTment

  • Find the leg of a right triangle using the Pythagorean Theorem.

  • Discussed the different elements of the Pythagorean Theorem.


Mastery Learning on the situation.

Objective

Lesson

Assessment

  • Find Mean Median Mode Range from data in a Stem and Leaf Plot

  • Lessons in 7th Grade

  • Mastery Learning Questions on Mean Median, Mode and Range from Stem and Leaf plots


Students work until they get 3 (or another chosen number) right in a row. The are very involved in their learning.

Assessment

  • Mastery Learning Questions on Mean Median, Mode and Range from Stem and Leaf plots


Mastery Learning right in a row. The are very involved in their learning.

Objective

Lesson

Assessment

Objective

Lesson

  • Find Mean Median Mode Range from data in a Stem and Leaf Plot

  • Lessons in 7th Grade

  • Mastery Learning Questions on Mean Median, Mode and Range from Stem and Leaf plots

  • Find Mean Median Mode Range from data in a Stem and Leaf Plot

  • Tutoring in ASSIST-ment.


Students gets instant feedback and support. right in a row. The are very involved in their learning.

Lesson

  • Tutoring in ASSIST-ment.


Mastery Learning right in a row. The are very involved in their learning.

Objective

Lesson

Assessment

Assessment

Objective

Lesson

  • Find Mean Median Mode Range from data in a Stem and Leaf Plot

  • Lessons in 7th Grade

  • Mastery Learning Questions on Mean Median, Mode and Range from Stem and Leaf plots

  • Students must work until they get 3 right in a row.

  • Can give a static post test.

  • Find Mean Median Mode Range from data in a Stem and Leaf Plot

  • Tutoring in ASSIST-ment.


Assessment right in a row. The are very involved in their learning.

  • Students must work until they get 3 right in a row.

  • Can give a static post test.


Objective right in a row. The are very involved in their learning.

Lesson

Assessment

  • Students will be able to solve questions involving exponential functions.

  • Use CMP Book

  • Homework from the Book, with questions online.

  • One question asks about linear equations.

Book Homework Online


Homework was Page 13 # 22 and 23 Page 16 # 34 right in a row. The are very involved in their learning.


Assessment right in a row. The are very involved in their learning.

  • Homework from the Book, with questions online.

  • One question asks about linear equations.

From Connected Mathematics Project Page 16 of Growing Growing Growing.


This data informed both teacher and students of the need for the next objective

Assessment

  • Homework from the Book, with questions online.

  • One question asks about linear equations.


Objective the next objective

Lesson

Assessment

Objective

Lesson

Assessment

  • Students will be able to solve questions involving exponential functions.

  • Use CMP Book

  • Homework from the Book, with questions online.

  • One question asks about linear equations.

  • While studying exponential functions students will be able to answer questions about linear relationships.

  • Read what is being asked carefully.

  • Discuss the linear equation question in class.

  • Next nights homework had multiple questions about comparing linear and exponential functions.

Book Homework Online


Assessment the next objective

  • Next nights homework had multiple questions about comparing linear and exponential functions.

From Connected Mathematics Project Page 14 of Growing Growing Growing.



Stories of formative assessment1

Mastery Learning the next objective

Stories of Formative Assessment

Online Exit Card

Book Homework Online


Does it teach as it assesses
Does It teach as It Assesses? the next objective

  • We have done 6 different randomized controlled experiment, and ASSISTments consistently leads to statistically meaningful learning results.

  • Two examples

    • A homework study

    • A classroom study


Comparing traditional paper and pencil homework with web based homework

Comparing the next objectiveTraditional Paper and Pencil Homework with Web-Based Homework

Mendicino, M., Razzaq, L. & Heffernan, N. T. (2009). Improving Learning from Homework Using Intelligent Tutoring Systems.Journal of Research on Technology in Education (JRTE). Published by the International Society For Technology in Education (ISTE). Spring 2009 issue.  41:3 p 331-346.

Leena Razzaq, Michael Mendicino & Neil Heffernan

Worcester Polytechnic Institute & West Virginia University

Intelligent Tutoring Systems 2008


Digital divide is real
Digital Divide Is Real. the next objective

  • Worcester is an urban district with only about 70% of homes with Internet.

  • Students already get weekly homework that requires the Internet.

  • Worcester needs supports for students who don’t have computers at home.

    • School libraries

    • Printable worksheet

    • Before school and homeroom

  • These studies prepare us for a future where kids can get the benefit of computers.


Maine lap top initiative
Maine Lap Top Initiative the next objective

Every student in the state get a lap top. Over half the districts let students take the lap top home.

State scores have gone up in English but stayed flat in Math. (Prof. David Silvernail)

Richmond Virginia (Henrico Country)


Is homework important
Is Homework Important? the next objective

  • Etta Kralovec, author of The End of Homework: How Homework Disrupts Families, Overburdens Children, and Limits Learning

    • "When work goes home, teachers have little control over who does the work. Teachers [are] unable to scaffold new knowledge for students, and [are] unaware of each student's true educational progress.”

  • http://www.hno.harvard.edu/gazette/2000/09.28/homework.html

  • Hattie(1999) meta-analysis across over 100 studies showed that homework has a .4 effect size.


Literature review other comparisons of pph to tutoring systems
Literature Review: Other Comparisons of PPH to Tutoring Systems

  • Little prior work but the best of it is at the college level

    • Mastering Physics – (Warnakulasooriya & Pritchard, 2005)

    • Andes (VanLehn et al., 2005) produced a mean effect size of 0.6 over paper-and-pencil homework.


Possible advantages of web based homework
Possible Advantages of Web Based Homework Systems

  • Better learning for students

    • Immediate feedback to students with tutoring

  • More help for teachers

    • Time saver

    • Common errors analyzed

    • Formative assessment—Teachers use the data to guide instruction more effectively.

    • Homework help—Students take homework more seriously because they know it will be graded and the grade recorded.


Possible disadvantages
Possible Disadvantages Systems

  • Many of these systems do not take students’ work into consideration.

  • Students may try to do more math in their heads.

  • Teacher may spend less time looking at student work.

  • Cheating may be easier among students.


Experimental design
Experimental Design Systems

  • Purpose: to determine if students can learn more by doing their math homework with the ASSISTment System than when doing traditional paper-and-pencil homework

  • Control: Student did their work on paper. The next day teachers did a 10 minute review of the questions. Teachers used the same feedback the ASSISTments system gave.

  • Materials:

  • Pretest and Posttest of 10 items

  • Homework was also 10 questions but with different numbers.

  • Topic A: Number Scene Topic B: Mixed


Sample number sense
Sample “Number Sense” Systems

1. Which of the following is closest to the product of 397.8 * 10.3?

o A) 3,000

o B) 30,000

o C) 400

o D) 4,000

2. Which of the following shows the numbers in order from least to greatest?

o A) 0.452, 0.51, 0.432

o B) 0.452, 0.432, 0.51

o C) 0.432, 0.51, 0.452

o D) 0.432, 0.452, 0.51

4. What is 15/60 as a percent?

5. Write 10/25 as a decimal.

6. What is the value of the following expression?

7 * 6 + 3

7. Destinee has a total of 12 fish in her aquarium. Exactly 9 of the fish are goldfish. What percent of the fish in the aquarium are goldfish?


Experimental design continued
Experimental Design Continued Systems

We did a repeated measures analysis where each student participated in each condition but in different orders. This allows us to deal with the large variation in different students learning rates. We eliminate the need to model the variation between teachers.

We counter-balanced the homework assignments in a Latin square design. The order of seeing the topics were randomly assigned to different classrooms.

West Virginia- 4 math teachers



Experimental design continued2
Experimental Design Continued Systems

To deal with the fact that not everyone had internet at home we only analyzed students that did their homework both nights in both conditions. Left us with 28 students

You can’t ask students if they internet at home

Its embarrassing

They don’t know if they will be able to have access


Results
Results Systems

  • Students learned significantly in both conditions

  • Students learned reliably more with ASSISTments (2.3 problems verse 1.14 problems) with 0.6 effect size (p = 0.05).


Results not due to a few students
Results Not Due to a Few Students Systems

Top 5 gain scores were in the web based homework condition.

Our worst gain score is in our WBH condition.


Conclusions
Conclusions Systems

  • Students do learn significantly (.6 standard deviations more) more from using the ASSISTment System compared to traditional paper-and-pencil homework.


Implications
Implications Systems

  • Results of this study are promising in that kids can learn a good deal more with homework.

  • Only study of its kind I am aware of in K12 that isolated the benefit of web based homework.

  • Could be important to policy makers, particularly considering the popularity of one-to-one computing initiatives

  • Will become be more relevant as the digital divide narrows.


Does an intelligent tutor lead to more learning than traditional classroom practice
Does an Intelligent Tutor lead to more learning than traditional classroom practice?

  • Most math classrooms give a lecture and then a period of time for practice.

  • We showed that kids learn a lot more when getting immediate feedback from the ITS than the “business as usual” control.

  • Results: Students learned more when using the computer compared to the traditional classroom.

  • Razzaq, L., Mendicino, M. & Heffernan, N. (2008) Comparing classroom problem-solving with no feedback to web-based homework assistance. In Woolf, Aimeur, Nkambou and Lajoie (Eds.) Proceeding of the 9th International Conference on Intelligent Tutoring Systems. pp. 426 -437. Springer-Verlag: Berlin.


Research questions for us dept of ed4
Research Questions for US Dept of Ed traditional classroom practice?

1. Does ASSISTments work as an assessment tool?

  • Reliably predict state test performance?

  • Reliably advise teachers & students on what knowledge to focus on?

    2. Does ASSISTments help enhance student learning?

  • Does the System effectively teach as it assesses?

  • Can teachers use the advice of this System to lead to higher student learning?


Can teachers use the advice of this system to lead to higher student learning1
Can teachers use the advice of this system to lead to higher student learning?

  • Quasi Experiment

    • Results: Student in schools that used ASSISTments did better on state test than students in control schools.

    • Results: Gaps were closed in achievement

  • Recruitment for a large 60 school field trial is being organized.

    $25 million dollar grant

    “i3” Investing in Innovation Fund grant line

    • If you are interested talk to me.


We have users in over a dozen towns
We Have Users in Over a Dozen Towns student learning?

Worcester, Auburn, Fitchburg—Classroom formative assessment

Shrewsbury—Nightly homework & Mastery Learning

Grafton—Open response explanation writing

Milbury—Benchmark Assessment every 9 weeks, teacher reports

Maine: Mastery Learning for homework


What have i not told you about
What have I not told you about? student learning?

Cristina Heffernan

  • Parent Notification System

  • How teacher and schools put in their own content.

  • Stick around for the next 30 minutes to learn how to

    • Create our own account,

    • Let kids log in, and

    • Build your own content.


Thank you
Thank you student learning?


Extra slides student learning?



Replicated field studies
Replicated Field Studies student learning?

  • Controlled, full year classroom experiments

  • Replicated over 3 years in urban schools

  • In Pittsburgh & Milwaukee

  • Results:

    50-100% better on problem solving & representation use.

    15-25% better on standardized tests.

Koedinger, Anderson, Hadley, & Mark (1997). Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education, 8.


An intelligent tutoring system success
An Intelligent Tutoring System Success student learning?

  • Cognitive Tutor Algebra

  • Most widely used ITS

    • 1000+ schools across the country

    • Marketed by local spin-off company Carnegie Learning

  • “Exemplary Curricula” by US Dept of Ed. In What Works Clearing House

  • Most cited Journal of AI-ED paper

    • Koedinger, Anderson, Hadley, & Mark (1997). Intelligent tutoring goes to school in the big city.

. . .


ad