Systematic methods of evaluation and monitoring
Download
1 / 44

Systematic Methods of Evaluation and Monitoring - PowerPoint PPT Presentation


  • 94 Views
  • Uploaded on

Systematic Methods of Evaluation and Monitoring. Early Reading First Conference Wild Horse Pass Chandler, Arizona October, 2004. ACE Partners. Gadsden Elementary School District- Lead Fiscal Agency Cocopah Indian Tribe Head Start Somerton Elementary School District Head Starts

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Systematic Methods of Evaluation and Monitoring' - adamdaniel


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Systematic methods of evaluation and monitoring

Systematic Methods of Evaluation and Monitoring

Early Reading First Conference

Wild Horse Pass

Chandler, Arizona

October, 2004


Ace partners
ACE Partners

  • Gadsden Elementary School District- Lead Fiscal Agency

  • Cocopah Indian Tribe Head Start

  • Somerton Elementary School District Head Starts

  • WACOG identified Head Starts

  • Arizona State University- Lead Curriculum and Alignment

  • Southwest Institute- Lead Community Development and Evaluation


Faculty and staff

Olivia Zepeda- Project Director

James Christie- Co-Proj Director

Jay Blanchard- Co-Proj Director

Tanis Bryan- Co-Proj Director

Karen Burstein- Evaluator

Robert Bernhart, Business Director

Marla Chamberlain- Mentor

Elizabeth “Bibi” Garza- Mentor

Lolinda Lee- Mentor

Roselia Ramirez- Mentor

5 Parent Liaisons

16 Head Start Teachers

19 Instructional Aids

Technology staff from ASU

Consultants in Early Literacy and Head Start

Faculty and Staff


Academic centers of excellence ace
Academic Centers of ExcellenceACE*

  • Rosalia Ramirez, Teacher Mentor, WACOG

  • Marla Chamberlain, Teacher Mentor, Gadsden Schools

  • Brenda Casillas, Teacher, WACOG

  • Connie Felix, Teacher, WACOG

  • Tanis Bryan, Karen Burstein, SWI

  • James Christie, Jay Blanchard, ASU

  • Cevriye Ergul, Pen-Chiang Chao, ASU/SWI

  • *ACE is a collaborative project between Gadsden School District, WACOG HeadStart, Somerton School District, Cocopah Indian Reservation, Southwest Institute for Families and Children with Special Needs (SWI), and Arizona State University


Evaluation and monitoring
Evaluation and Monitoring

  • Standardized, Norm Referenced

    • Twice annually

    • All participants and a matched control group

    • Measure the progress of the PROJECT

  • Curriculum-Based Measurement

    • Weekly

    • A small group of children form each class based on standardized test results and teacher report

    • Monitor and adjust instruction for children


STANDARDIZED ASSESSMENTS

Individually or group administered

Compares individual to group

Used for program evaluation and planning

CURRICULUM BASED ASSESSMENT

Individually administered

Compares individual to curriculum

Used for individual evaluation and instructional planning

EFFECTIVE PROGRAM EVALUATION AND MONITORING CONSITS OF A BALANCE BETWEEN STANDARDIZED NORM REFERENCED AND CURRICULUM-BASED ASSESSMENTS



Ace 3 assessment
ACE3 Assessment

  • Initial assessment: We want to know what children know at the beginning of the project in the areas of:

    • Oral Language- understanding of language and spoken vocabulary

    • Phonologic Awareness- recognizes/ demonstrates that language is made up of individual sounds

    • Alphabet Knowledge- demonstrates the relationship between sounds and letters

    • Concepts of Print- demonstrates how to use printed materials, e.g., top of page, bottom, beginning, end, word v. letter


  • Standardized assessments are professionally developed tests administered under standard conditions, producing scores that can be used to evaluate programs. The type of standardized test required by ERF is designed to determine whether groups of children within a program are progressing or meeting the standards of learning as compared to children not engaged in ERF programs.

  • Scores of standardized tests also indicate how children within each tested program in each site performed, how groups performed, and how groups of children across the state performed.

  • NCLB requires that scores for schools and districts be disaggregated so that the performance of children from different subgroups can be examined.

  • Standardized tests aligned with state standards are essential for administrators to determine whether schools/sites are meeting their goals under NCLB.


  • Standardized testing achieves standardization by norming practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard.


Standardization procedures
Standardization Procedures practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard.

  • Pilot testing

  • Controlled environment (no distractions)

  • Trained observers, evaluators

  • Written administration instructions

  • Guidelines for motivation of children/ question responses

  • Cross-check procedures for scoring and data input


Assessing measures
Assessing Measures practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard.

  • Make sure that the tests measure what you want them to measure (content validity)

  • Choose measures that have been validated on populations of children who have like characteristics of age, language, cognitive ability, and regional background

  • Train assessors to administer tests and compare the results of different assessors so that controls are in place to eliminate the effects of these different examiners.


Ace measures
ACE Measures practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard.

Individually Administered, Twice Annually, Approximate Time 25 minutes per child.

  • IDEA Individual Proficiency Test- Oral Language, English Proficiency

  • Peabody Picture Vocabulary Test- III- Receptive Vocabulary

  • Predictive Assessment of Reading- Vocabulary, Phonemic Awareness, Rapid Object Naming (Spanish and English)

  • Head Start Letter Naming

  • Get Ready to Read- Vocabulary, Concepts of Print, Letter Naming, Phonemic Awareness

  • Get It, Got It, Go! (Baby DIBLES) Vocabulary, Phonemic Awareness


Norm referenced tests
Norm Referenced Tests practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard.

  • Standardized tests, the results of which compare individual scores to those of a norm, or group average. The average is typically gathered from individuals of like age,

  • Results are reported in scores such as (per)centiles, stanines, normal curve equivalencies,


Standardization is of little importance if the practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard.

results of assessment are to be used in

isolation from all other factors. In other

words, if the purpose is simply to learn about

the state of a single subject/child, a unique

assessment (CBM) might be devised to

furnish the information desired. However, if

the assessment is to be used for the purpose

of comparison, generalization, or decision-

making, then standardization is essential.


Monitoring children s learning
Monitoring practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard. Children’s Learning

  • A major responsibility of educators is to teach children academic skills.

  • Teaching includes:

    • Monitoring whether children are learning what is being taught and

    • Making changes in instruction when children are not making adequate progress.


Testing to monitor children s learning
Testing to Monitor practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard. Children’s Learning

Testing is one way of monitoring what children are learning.

  • When professionals and the public think about testing, they tend to think in terms of standardized test scores.

  • “No Child Left Behind” includes extensive mandates to use standardized tests to hold schools accountable for gains in children’s academic performance.


Limitations of standardized tests
Limitations of Standardized Tests practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard.

The tests typically used:

  • Norms are based on a national average

  • Lack overlap with local curriculum

  • Given infrequently

  • Not sensitive to short-term academic gains

  • Do not provide teachers with the information they need to make classroom instructional decisions


Curriculum based measurement cbm
Curriculum Based Measurement practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard. (CBM)

  • Teachers need-and value- information that is directly applicable to their day-to-day practice and linked to specific instructional objectives and learning concepts.

  • An alternative to standardized tests is Curriculum Based Measurement (CBM)


CBM practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard.

  • CBM is a method of monitoring student educational progress using materials taken from the child's classroom curriculum.

  • CBM tests are constructed from the lessons taught.

  • CBM is Fast! (2 minutes, weekly)

  • CBM is Inexpensive and Easy!


CBM practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard.

  • CBM allows teachers to

    • Continuously measure their students' growth in performance,

    • Determine if their students' are learning at the expected rate

      CBM provides data for teachers to evaluate their instructional strategies when students fail to demonstrate expected growth.


Advantages of cbm over other testing methods
Advantages of CBM over other testing methods practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard.

  • Based on classroom curriculum

  • Individual referenced

  • Peer referenced

  • Provides teachers with information to make instructional change decisions

  • Allows for direct and continuous monitoring of student achievement related to expected curriculum outcomes

  • Highly sensitive to student growth

  • Time efficient

  • Cost effective

  • Produces results that are easy to understand


Purpose of cbm in the ace
Purpose of CBM in the ACE practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard.

  • Each teacher in the ACE project is using weekly CBM in order to:

  • Monitor children’s acquisition of receptive and expressive vocabulary, letter recognition, and phonemic awareness skills.

  • Monitor how well children with special needs and at-risk are doing relative to higher achieving children.

  • Make adaptations in instruction based on comparisons of individual and group data. teachers make


Cbm procedures
CBM Procedures practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard.

Steps to CBM:

  • 1. Identify children (each ACE classroom is monitoring 4 children: 1 receiving special education services, 1 at-risk for special education (referred but did not qualify for special education), 1 typical achiever, and 1 “super star”).

  • The “super star” child defines what is possible; the average achieving child’ s “sets the standard” for what is reasonable.

  • Teachers selections of average and super star achievers will be compared with the results of standardized tests administered at the beginning of school.


Cbm procedures1
CBM Procedures practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard.

2. Develop CBM: Collect samples from each week’s lessons (e.g., receptive vocabulary, expressive vocabulary, alliteration).

3. Alliteration items are on cards for easier administration.

4. Prepare a separate score sheet for each child. The score sheet includes the vocabulary and letters being tested as well the materials needed to assess (e.g., posters, small books).


Sample score sheet
Sample Score Sheet practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard.


Curriculum based measurement procedures
Curriculum-Based practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard. Measurement Procedures

  • Administer CBM weekly.

  • Teachers select a time that allow them to be consistent each week in administering the CBM.

  • Ideally, teachers should use the same wording, speed, and wait time for each child. In reality, there are time differences based on the child’s needs.

  • Record number of items done correctly on score sheet.


Aim line
Aim Line practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard.

Graph the results for the children in each classroom.

The average achieving child’s performance defines the “aim line”

In the ACE project, we are defining the aim line as the expected performance based on the typically developing child’s performance.


Curriculum based measurement cont
Curriculum-Based Measurement (Cont.) practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard.

8. Develop a line for each child on a graph


Interpreting the results
Interpreting the Results practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard.

Decision Rule 1: If three consecutive data points are below the aim line (See sample graph) for a child, teacher asks, why.

  • upsetting events in the home?

  • illness?

  • absences?

  • behavior difficulties?

  • CBM items too difficult?


Interpreting the results1
Interpreting the Results practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard.


Interpreting results
Interpreting Results practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard.

Instructional changes for Decision Rule 1:

  • Change the classroom environment

    • change the student's seating

    • change time of instruction or testing

  • Modify Instruction

    • change the type of instruction (small group or one-on-one instruction)

    • Give more time for guided and independent practice of skills taught

    • Reteach when necessary

    • Review skills

    • Simplify directions.


Interpreting the results2
Interpreting the Results practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard.

Decision Rule 2: If the child's performance is not consistently above or below the aim line (i.e., more than three consecutive data points), don't make any instructional changes(See the sample graph for decision rule 2).


Administration of curriculum based measurement cont
Administration of practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard. Curriculum-based measurement (Cont.)


Interpreting the results3
Interpreting the Results practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard.

Decision Rule 3: If the child's performance is above the aim line for three consecutive data points, consider enhancing and enriching the instructional program (See the sample graph for decision rule 3).


Interpreting the results4
Interpreting the Results practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard.


Using the results
Using the Results practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard.

Continue to measure and monitor

Regularly inform parentsabouttheir child's

progress toward the goals.

  • A written explanation of individual child’s graph might be sent home. The written note may include an explanation of child’s progress on the graph.

  • Note positive changes in the child’s progress on the graph

  • Arrange conferences for parents who might need some assistance in understanding the concept and may need an explanation of the CBM.


Data from ACE CBM— practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard.

Feb to May, 2004


CBM Mean Percentage of Correct Response practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard. for the Three Groups

Typical AtRiskSpecialNeeds (n = 25) (n = 7) (n = 21)

Letter Identification Unit 1 81.00 (30.4) 59.53 (34.5) 50.20 (37.4)

Letter Identification Unit 2 72.92 (30.4) 61.63 (27.3) 51.15 (34.8)

Letter Identification Unit 3 76.08 (28.7) 50.00 (32.3) 60.04 (27.2)

Show Me Unit 1 73.29 (14.9) 48.19 (14.3) 69.23 (24.0)

Show Me Unit 2 78.83 (13.4) 56.70 (19.4) 70.66 (25.5)

Show Me Unit 3 82.37 (13.5) 46.67 (27.1) 68.64 (21.7)

Tell Me Unit 1 59.38 (20.3) 47.56 (17.1) 50.71 (26.0)

Tell Me Unit 2 59.83 (17.6) 47.23 (15.1) 52.69 (25.7)

Tell Me Unit 3 75.07 (19.0) 57.51 (14.2) 57.48 (25.0)

Alliteration 70.31 (27.1) 69.96 (21.7) 50.49 (29.6)


CBM Mean Percentage of Correct Response practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard. for the Three Groups

S

Typical (n = 25)

At Risk (n = 7)

Special Needs (n = 21)

Letter Identification Show Me Tell Me Alliteration


An Example of CBM Performance: Typical vs. Special Needs practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard.

Typical Child

Child with Special Needs

Letter Identification Show Me Tell Me Alliteration


Comparisons Between the Three Groups practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard.

Statistically Significant Group Differences (ANOVA):

Letter Identification Unit 1: F (2, 50) = 4.87, p = .012

Letter Identification Unit 3: F (2, 53) = 3.25, p = .047

Show Me Unit 1: F (2, 50) = 4.80, p = .012

Show Me Unit 2: F (2, 53) = 3.73, p = .031

Show Me Unit 3: F (2, 53) = 10.61, p < .001

Tell Me Unit 3: F (2, 53) = 4.83, p = 0.12

Alliteration: F (2, 53) = 3.43, p = 0.40

Significant Post Hoc Comparisons:

Letter Identification Unit 1: Typical (M = 81.0) > Special Needs (M = 50.2), p = .009

Show Me Unit 1: Typical (M = 73.3) > At Risk (M = 48.2), p = .009

Show Me Unit 2: Typical (M = 78.8) > At Risk (M = 56.7), p = .028

Show Me Unit 3: Typical (M = 82.4) > At Risk (M = 46.7), p < .001 Typical (M = 82.4) > Special Needs (M = 60.6), p = 039 Special Needs (M = 60.6) > At Risk (M = 46.7), p = .027

Tell Me Unit 3: Typical (M = 75.1) > Special Needs (M = 57.5), p = .015

Alliteration: Typical (M = 70.3) > Special Needs (M = 50.5), p = .040


Comparisons Within the Three Groups practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard.

Typical Children (n = 25):

Letter Identification:No significant changes over time

Show Me:Statistically significant, Wilks’ Λ = .71, F (2, 23) = 4.75, p = .019, multivariate η2 = .29 (effect size) Children’s “Show Me” performance improved over time Unit 3 (M = 82.4) > Unit 2 (M = 78.8) > Unit 1 (M = 73.3)

Tell Me:Statistically significant, Wilks’ Λ = .54, F (2, 23) = 9.92, p = .001, multivariate η2 = .46 (effect size) Children’s “Tell Me” performance improved over time Unit 3 (M = 75.1) > Unit 2 (M = 59.8) > Unit 1 (M = 53.4)

At-Risk Children (n = 7):No significant changes

Children with special needs (n = 21):No significant changes


ad