Loading in 5 sec....

Worcester Polytechnic Institute IntroducesPowerPoint Presentation

Worcester Polytechnic Institute Introduces

- 83 Views
- Uploaded on
- Presentation posted in: General

Worcester Polytechnic Institute Introduces

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Worcester Polytechnic Institute

Introduces

Learn about research involving this online formative assessment system and hear about how it is used in classrooms.

In this Webinar you will hear from:

Courtney Mulcahy, an 8th grade teacher who uses ASSISTment in her formative assessment process every day.

Neil Heffernan, a computer science professor about his research using this tool.

Matt Militello, a professor of school leadership on how school leaders can use such a system.

Cristina Heffernan, a professional developer who trains teachers to use the system and incorporate it into their classrooms.

And Russ Rapose, District curriculum coordinator for Milbury Public Schools

The Press… The Push

The Press:

- More than ever we are TOLD, MANDATED, and SANCTIONED to use data to inform education decisions.
- Specifically, assessment data. Means state tests.
The Push:

- These data are not timely, not informative for teachers; That is, these data points are autopsy data.
- TMI: Too Much Information.

In Search of the Right Data!

Central Office

Student/School Growth

School Leaders

Teacher Effectiveness

Teachers

Student Cognition

Formative Assessment Systems (FAS)

There is Demand…. Supply Followed

Multi-Million Dollar Industry- Cost usually based on per student use

Not all FAS are the SAME!

How Do Districts Select, Support, and Use FAS?

- Three DistrictsThree FAS
- Technical FeaturesImplementation
- Actual Use

Framework

Fit is defined by the INTENT of the districts selection (e.g. teacher use (student-diagnostic data) or district use (student growth model data) and the PURPOSE of the assessment system—including validity.

Framework

Readiness is definedas: (1) building capacity in regard to both knowledge of the assessment product and (2) the motivation to use the product in practice.

Framework

Coherence is how educators used the assessment systems fit into the context and the standard operating procedures of the district. For instance, the creation of trigger mechanisms from review of data reports.

Woodbury Schools

FAS-

NWEA-MAP

Good fit for Central Office,

but not teachers

Growth data forced on

daily practice

Teachers ill-equipped

Needle in a Haystack:A FAS with Real-time, Student-Level Data!

- Fit! Finally, Formative Assessments for TEACHERS
- Technical Features Tutoring System for Real-Time Data and Diagnostic Data: PURPOSE
- Teacher-Level Use- Desire For This Data: INTENT

Using Intelligent Tutoring Systems to Help Math Teachers in the Formative Assessment Process. Prof. Neil HeffernanWorcester Polytechnic Institute

Who Am I? the Formative Assessment Process.

- Teacher for America : teacher of middle school math and science in Baltimore City
- PhD in Computer Science and Cognitive Science
- Licensed to Carnegie Learning Inc.

- Tenure professor at Worcester Polytechnic
- Grant funded research on developing interventions

Quick Project History - 2003 the Formative Assessment Process.

Teacher are being asked to use data to inform instruction, but every minute in the classroom spent on assessment is a minute lost to instruction.

US Department of Education (2003) Funded us to build a assessment system for teachers that did not take away teacher’s time.

Lindeman, Carolina Ruiz, George Heineman, Murali Mani, Ryung Kim and others 30 others

Definitions Kim and others 30 others

A diagnostic assessment is an assessment designed to uncover any misconceptions or gaps in knowledge or skills of individual students so that teachers can base their instruction on the current state of students’ knowledge, skills, or understanding.

Formative assessment is a process used by teachers and students during instruction that provides feedback to adjust ongoing teaching and learning to improve students’ achievement of intended instructional outcomes.

Research Questions for US Dept of Ed Kim and others 30 others

1.Does ASSISTments work as an assessment tool?

- Reliably predict state test performance?
- Reliably advise teachers & students on what knowledge to focus on?
2.Does ASSISTments help enhance student learning?

- Does the System effectively teach as it assesses?
- Can teachers use the advice of this System to lead to higher student learning?

The Kim and others 30 othersASSISTment System

- A web-based assessment system, designed to collect formative assessment data on student math skills
- Usage has grown as our content has grown.
- Currently, thousands of students use the system.

You Just Saw One Kim and others 30 othersASSISTment

- A question within this “tutor” is what we call an ASSISTment .
- We have 2,000 such ASSISTments for 7th and 8th grade. (66 hours of instruction)
- We have another 1,000 for 6th grade.
- We have 1000 for 9-10th grade math,
- There are Three other types of content.
- Teacher created questions
- Mastery Learning questions (track 130 skills in 8th grade math)
- Connected Math matching units (500 ASSISTments with tutoring)

Many Different Types of Reports Kim and others 30 others

State Test Report

Knowledge component reports (based upon 130 middle school skills)

Item Report (e.g. for small formative assessments)

Mastery Learning Report (which students have reached mastery?)

Names Obscured on Purpose Kim and others 30 others

Report to a Teacher on Kim and others 30 othersTheir Own State Test

1.Does ASSISTments work as an assessment tool? Kim and others 30 others

Reliably predict state test performance?

Reliably advise teachers & students on what knowledge to focus on?

2.Does ASSISTments help enhance student learning?

Does the System effectively teach as it assesses?

Can teachers use the advice of this System to lead to higher student learning?

Research Questions for US Dept of EdReport to a Teacher on Their State Kim and others 30 others

Predict Student Knowledge as Measured by State Test Scores Kim and others 30 others

Feng, M., Beck, J,. Heffernan, N. & Koedinger, K. (2008) Can an Intelligent Tutoring System Predict Math Proficiency as Well as a Standardized Test? In Baker & Beck (Eds.). Proceedings of the 1st International Conference on Education Data Mining. Montreal, Canada. pp.107-116. ISBN - 13: 9780615306292

Feng, M., Heffernan, N. T., & Koedinger, K. R. (2006b). Addressing the testing challenge with a Web-based e-assessment system that tutors as it assesses. Proceedings of the Fifteenth International World Wide Web Conference (WWW-06). New York, NY: ACM Press. ISBN:1-59593-332-9. pp. 307-316. [Nominated for Best Student Paper]

Feng, M., Heffernan, N.T., & Koedinger, K.R. (2009). Addressing the assessment challenge in an Intelligent Tutoring System that tutors as it assesses. The Journal of User Modeling and User-Adapted Interaction.Vol 19: p243-266.

- We have state scores from 2004 to 2009 for thousands of students.
- We showed our error in predicting the state test is as low as the test itself.
- We showed we can even predict their 10th grade math score from their 8th grade data.
- We have shown we can more reliably predict their state test score if we use how much assistance they need.

Research Questions for US Dept of Ed Kim and others 30 others

1.Does ASSISTments work as an assessment tool?

- Reliably predict state test performance?
- Reliably advise teachers & students on what knowledge to focus on?
2.Does ASSISTments help enhance student learning?

- Does the System effectively teach as it assesses?
- Can teachers use the advice of this System to lead to higher student learning?

We have shown that our transfer model is good enough to improve assessment.

- Using finer-grained skill model can improve accuracy of the assessment of student knowledge.
- Feng, M., Heffernan, N. T., Mani, M., & Heffernan, C. (2007). Assessing students’ performance longitudinally: Item difficulty parameter vs. skill learning tracking. The National Council on Educational Measurement 2007 Annual Conference, Chicago.
- Feng , M, Heffernan, N., Heffernan, C. & Mani, M. (2009) Using Mixed-Effects Modeling to Analyze Different Grain-Sized Skill Models. Journal of Technology, Learning, and Assessment. (Featured Article of the year!)
- Pardos, Z. A., Heffernan, N. T., Anderson, B. & Heffernan, C. (2007). The effect of model granularity on student performance prediction using Bayesian networks. The International User Modeling Conference 2007.

Research Questions for US Dept of Ed improve assessment.

1.Does ASSISTments work as an assessment tool?

- Reliably predict state test performance?
- Reliably advise teachers & students on what knowledge to focus on?
2.Does ASSISTments help enhance student learning?

- Does the System effectively teach as it assesses?
- Can teachers use the advice of this System to lead to higher student learning?

Can teachers use the advice of this system to lead to higher student learning?

- Quasi Experiment
- Results: Student in schools that used ASSISTments did better on state test than students in control schools.
- Results: Gaps were closed in achievement

- Recruitment for a large 60 school field trial is being organized.
$25 million dollar grant

“i3” Investing in Innovation Fund grant line

- If you are interested talk to me.

Mastery Learning student learning?

Stories of Formative AssessmentOnline Exit Card

Courtney Mulcahy, an 8th grade teacher who uses ASSISTment in her formative assessment process every day.

Book Homework Online

The Cycle can move through two or three iterations depending on the situation.

Objective

Lesson

Objective

Lesson

Objective

Assessment

Lesson

Assessment

Assessment

Online Exit Card on the situation.

Objective

Lesson

Assessment

- Find the side of a triangle using the Pythagorean Theorem

- Use Connected Math Book

- Three question Problem Set online on ASSISTment

Assessment on the situation.

- Three question Problem Set online on ASSISTment

Teacher on the situation.

Student

Assessment

There were 43 students on this item report, not all are shown.

- Three question Problem Set online on ASSISTment

CORRECT on the situation.

a2+b2 = c2

122+x2 = 142

144 + x2 = 196

x2 = 54

x = 7.21

INCORRECT

a2+b2 = c2

122+142 = x2

144 + 196 = x2

340= x2

18.44 = x

Online Exit Card on the situation.

Objective

Lesson

Assessment

Objective

Lesson

- Find the side of a triangle using the Pythagorean Theorem

- Use Connected Math Book

- Three question Problem Set online on ASSISTment

- Find the leg of a right triangle using the Pythagorean Theorem.

- Discussed the different elements of the Pythagorean Theorem.

Mastery Learning on the situation.

Objective

Lesson

Assessment

- Find Mean Median Mode Range from data in a Stem and Leaf Plot

- Lessons in 7th Grade

- Mastery Learning Questions on Mean Median, Mode and Range from Stem and Leaf plots

Students work until they get 3 (or another chosen number) right in a row. The are very involved in their learning.

Assessment

- Mastery Learning Questions on Mean Median, Mode and Range from Stem and Leaf plots

Mastery Learning right in a row. The are very involved in their learning.

Objective

Lesson

Assessment

Objective

Lesson

- Find Mean Median Mode Range from data in a Stem and Leaf Plot

- Lessons in 7th Grade

- Mastery Learning Questions on Mean Median, Mode and Range from Stem and Leaf plots

- Find Mean Median Mode Range from data in a Stem and Leaf Plot

- Tutoring in ASSIST-ment.

Students gets instant feedback and support. right in a row. The are very involved in their learning.

Lesson

- Tutoring in ASSIST-ment.

Mastery Learning right in a row. The are very involved in their learning.

Objective

Lesson

Assessment

Assessment

Objective

Lesson

- Find Mean Median Mode Range from data in a Stem and Leaf Plot

- Lessons in 7th Grade

- Mastery Learning Questions on Mean Median, Mode and Range from Stem and Leaf plots

- Students must work until they get 3 right in a row.
- Can give a static post test.

- Find Mean Median Mode Range from data in a Stem and Leaf Plot

- Tutoring in ASSIST-ment.

Assessment right in a row. The are very involved in their learning.

- Students must work until they get 3 right in a row.
- Can give a static post test.

Objective right in a row. The are very involved in their learning.

Lesson

Assessment

- Students will be able to solve questions involving exponential functions.

- Use CMP Book

- Homework from the Book, with questions online.
- One question asks about linear equations.

Book Homework Online

Homework was Page 13 # 22 and 23 Page 16 # 34 right in a row. The are very involved in their learning.

Assessment right in a row. The are very involved in their learning.

- Homework from the Book, with questions online.
- One question asks about linear equations.

From Connected Mathematics Project Page 16 of Growing Growing Growing.

This data informed both teacher and students of the need for the next objective

Assessment

- Homework from the Book, with questions online.
- One question asks about linear equations.

Objective the next objective

Lesson

Assessment

Objective

Lesson

Assessment

- Students will be able to solve questions involving exponential functions.

- Use CMP Book

- Homework from the Book, with questions online.
- One question asks about linear equations.

- While studying exponential functions students will be able to answer questions about linear relationships.
- Read what is being asked carefully.

- Discuss the linear equation question in class.

- Next nights homework had multiple questions about comparing linear and exponential functions.

Book Homework Online

Assessment the next objective

- Next nights homework had multiple questions about comparing linear and exponential functions.

From Connected Mathematics Project Page 14 of Growing Growing Growing.

Homework was Page 14 – 17 # 24 – 30 and 35 - 38 the next objective

Mastery Learning the next objective

Stories of Formative AssessmentOnline Exit Card

Book Homework Online

Does It teach as It Assesses? the next objective

- We have done 6 different randomized controlled experiment, and ASSISTments consistently leads to statistically meaningful learning results.
- Two examples
- A homework study
- A classroom study

Comparing the next objectiveTraditional Paper and Pencil Homework with Web-Based Homework

Mendicino, M., Razzaq, L. & Heffernan, N. T. (2009). Improving Learning from Homework Using Intelligent Tutoring Systems.Journal of Research on Technology in Education (JRTE). Published by the International Society For Technology in Education (ISTE). Spring 2009 issue. 41:3 p 331-346.

Leena Razzaq, Michael Mendicino & Neil Heffernan

Worcester Polytechnic Institute & West Virginia University

Intelligent Tutoring Systems 2008

Digital Divide Is Real. the next objective

- Worcester is an urban district with only about 70% of homes with Internet.
- Students already get weekly homework that requires the Internet.
- Worcester needs supports for students who don’t have computers at home.
- School libraries
- Printable worksheet
- Before school and homeroom

- These studies prepare us for a future where kids can get the benefit of computers.

Maine Lap Top Initiative the next objective

Every student in the state get a lap top. Over half the districts let students take the lap top home.

State scores have gone up in English but stayed flat in Math. (Prof. David Silvernail)

Richmond Virginia (Henrico Country)

Is Homework Important? the next objective

- Etta Kralovec, author of The End of Homework: How Homework Disrupts Families, Overburdens Children, and Limits Learning
- "When work goes home, teachers have little control over who does the work. Teachers [are] unable to scaffold new knowledge for students, and [are] unaware of each student's true educational progress.”

- http://www.hno.harvard.edu/gazette/2000/09.28/homework.html
- Hattie(1999) meta-analysis across over 100 studies showed that homework has a .4 effect size.

Literature Review: Other Comparisons of PPH to Tutoring Systems

- Little prior work but the best of it is at the college level
- Mastering Physics – (Warnakulasooriya & Pritchard, 2005)
- Andes (VanLehn et al., 2005) produced a mean effect size of 0.6 over paper-and-pencil homework.

Possible Advantages of Web Based Homework Systems

- Better learning for students
- Immediate feedback to students with tutoring

- More help for teachers
- Time saver
- Common errors analyzed
- Formative assessment—Teachers use the data to guide instruction more effectively.
- Homework help—Students take homework more seriously because they know it will be graded and the grade recorded.

Possible Disadvantages Systems

- Many of these systems do not take students’ work into consideration.
- Students may try to do more math in their heads.
- Teacher may spend less time looking at student work.
- Cheating may be easier among students.

Experimental Design Systems

- Purpose: to determine if students can learn more by doing their math homework with the ASSISTment System than when doing traditional paper-and-pencil homework
- Control: Student did their work on paper. The next day teachers did a 10 minute review of the questions. Teachers used the same feedback the ASSISTments system gave.
- Materials:
- Pretest and Posttest of 10 items
- Homework was also 10 questions but with different numbers.
- Topic A: Number Scene Topic B: Mixed

Sample “Number Sense” Systems

1. Which of the following is closest to the product of 397.8 * 10.3?

o A) 3,000

o B) 30,000

o C) 400

o D) 4,000

2. Which of the following shows the numbers in order from least to greatest?

o A) 0.452, 0.51, 0.432

o B) 0.452, 0.432, 0.51

o C) 0.432, 0.51, 0.452

o D) 0.432, 0.452, 0.51

4. What is 15/60 as a percent?

5. Write 10/25 as a decimal.

6. What is the value of the following expression?

7 * 6 + 3

7. Destinee has a total of 12 fish in her aquarium. Exactly 9 of the fish are goldfish. What percent of the fish in the aquarium are goldfish?

Experimental Design Continued Systems

We did a repeated measures analysis where each student participated in each condition but in different orders. This allows us to deal with the large variation in different students learning rates. We eliminate the need to model the variation between teachers.

We counter-balanced the homework assignments in a Latin square design. The order of seeing the topics were randomly assigned to different classrooms.

West Virginia- 4 math teachers

Experimental Design Continued Systems

Experimental Design Continued Systems

To deal with the fact that not everyone had internet at home we only analyzed students that did their homework both nights in both conditions. Left us with 28 students

You can’t ask students if they internet at home

Its embarrassing

They don’t know if they will be able to have access

Results Systems

- Students learned significantly in both conditions
- Students learned reliably more with ASSISTments (2.3 problems verse 1.14 problems) with 0.6 effect size (p = 0.05).

Results Not Due to a Few Students Systems

Top 5 gain scores were in the web based homework condition.

Our worst gain score is in our WBH condition.

Conclusions Systems

- Students do learn significantly (.6 standard deviations more) more from using the ASSISTment System compared to traditional paper-and-pencil homework.

Implications Systems

- Results of this study are promising in that kids can learn a good deal more with homework.
- Only study of its kind I am aware of in K12 that isolated the benefit of web based homework.
- Could be important to policy makers, particularly considering the popularity of one-to-one computing initiatives
- Will become be more relevant as the digital divide narrows.

Does an Intelligent Tutor lead to more learning than traditional classroom practice?

- Most math classrooms give a lecture and then a period of time for practice.
- We showed that kids learn a lot more when getting immediate feedback from the ITS than the “business as usual” control.
- Results: Students learned more when using the computer compared to the traditional classroom.

- Razzaq, L., Mendicino, M. & Heffernan, N. (2008) Comparing classroom problem-solving with no feedback to web-based homework assistance. In Woolf, Aimeur, Nkambou and Lajoie (Eds.) Proceeding of the 9th International Conference on Intelligent Tutoring Systems. pp. 426 -437. Springer-Verlag: Berlin.

Research Questions for US Dept of Ed traditional classroom practice?

1.Does ASSISTments work as an assessment tool?

- Reliably predict state test performance?
- Reliably advise teachers & students on what knowledge to focus on?
2.Does ASSISTments help enhance student learning?

- Does the System effectively teach as it assesses?
- Can teachers use the advice of this System to lead to higher student learning?

Can teachers use the advice of this system to lead to higher student learning?

- Quasi Experiment
- Results: Student in schools that used ASSISTments did better on state test than students in control schools.
- Results: Gaps were closed in achievement

- Recruitment for a large 60 school field trial is being organized.
$25 million dollar grant

“i3” Investing in Innovation Fund grant line

- If you are interested talk to me.

We Have Users in Over a Dozen Towns student learning?

Worcester, Auburn, Fitchburg—Classroom formative assessment

Shrewsbury—Nightly homework & Mastery Learning

Grafton—Open response explanation writing

Milbury—Benchmark Assessment every 9 weeks, teacher reports

Maine: Mastery Learning for homework

What have I not told you about? student learning?

Cristina Heffernan

- Parent Notification System
- How teacher and schools put in their own content.
- Stick around for the next 30 minutes to learn how to
- Create our own account,
- Let kids log in, and
- Build your own content.

Thank you student learning?

Extra slides student learning?

Cognitive Tutor – Carnegie Learning Inc. student learning?

Replicated Field Studies student learning?

- Controlled, full year classroom experiments
- Replicated over 3 years in urban schools
- In Pittsburgh & Milwaukee
- Results:
50-100% better on problem solving & representation use.

15-25% better on standardized tests.

Koedinger, Anderson, Hadley, & Mark (1997). Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education, 8.

An Intelligent Tutoring System Success student learning?

- Cognitive Tutor Algebra
- Most widely used ITS
- 1000+ schools across the country
- Marketed by local spin-off company Carnegie Learning

- “Exemplary Curricula” by US Dept of Ed. In What Works Clearing House
- Most cited Journal of AI-ED paper
- Koedinger, Anderson, Hadley, & Mark (1997). Intelligent tutoring goes to school in the big city.

. . .