Rea brown bag sat10 growth methods and reporting
Download
1 / 25

REA Brown Bag – SAT10 Growth Methods and Reporting - PowerPoint PPT Presentation


  • 96 Views
  • Uploaded on

REA Brown Bag – SAT10 Growth Methods and Reporting. Tom Watkins, Ph.D Director of Research, Evaluation and Assessment, SPPS. Goals for Brown Bag (2 or more of the following):.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' REA Brown Bag – SAT10 Growth Methods and Reporting' - luigi


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Rea brown bag sat10 growth methods and reporting

REA Brown Bag – SAT10 Growth Methods and Reporting

Tom Watkins, Ph.D

Director of Research, Evaluation and Assessment, SPPS


Goals for brown bag 2 or more of the following
Goals for Brown Bag (2 or more of the following):

1. To describe some strengths and weaknesses of various approaches for calculating achievement growth on the SAT10 in SPPS.

2. To describe some strengths and weaknesses of various approaches for reporting achievement growth on the SAT10 in SPPS.

3. To examine the degree of correlation between different growth measures at the student score level

4. To examine the degree to which different growth measures produce the same growth categories (average, hi or low) at the grade level


Goals for brown bag
Goals for Brown Bag

1. To describe some strengths and weaknesses of various approaches for calculating achievement growth on the SAT10 in SPPS.

4. To examine the degree to which different growth measures produce the same growth categories (average, hi or low) at the grade level within school, and at the group level within the district


Part 1 strengths and weaknesses of approaches for calculating achievement growth
Part 1: Strengths and Weaknesses of Approaches for Calculating Achievement Growth

  • Illustrations and Basis for SAT10 Growth

    • Scaled Score Improvement (SSI – OLS Reg)

    • Percent Gaining (NCE change 0 or more)

    • Index Rate Growth (SAT10 to MCA)

    • Heirarchical Linear Modeling (HLM)

  • General Psychometric Considerations

  • General Practical Considerations

  • SPPS Considerations



  • The method for calculating growth for student groups from MAT7 to SAT10 (similar to what has been used since):

  • For each grade level and subject area, run a simple linear regression of SAT10 scaled scores on MAT7 scaled scores.

  • Save the standardized residuals (unstandardized residuals were used for the MAT7 SSI).

  • For groups with N's of 10-39, a mean standardized residual at or below -.20 is "low growth", and residuals at or above +.20 is "high growth".

  • For groups with N's of 40-79, a mean standardized residual at or below -.15 is "low growth", and residuals at or above +.15 is "high growth".

  • For groups with N's of 80+, a mean standardized residual at or below -.10 is "low growth", and residuals at or above +.10 is "high growth".


Basis of Index Growth from SAT10 to MCA MAT7 to SAT10 (similar to what has been used since):


Basis for mlm approach hlm one type steenbergen 2002
Basis for MLM Approach (HLM one type) MAT7 to SAT10 (similar to what has been used since):(Steenbergen, 2002)

  • Uses “Nested” Data - Distinguishing Student-Level from Teacher, Grade, School and District Levels

  • Data Intensive

  • Theory Intensive

  • Requires More Assumptions


General MAT7 to SAT10 (similar to what has been used since):Practical Considerations in Growth and Gain Models


General MAT7 to SAT10 (similar to what has been used since):Psychometric Considerations in Growth and Gain Models


Additional spps growth method considerations
Additional SPPS Growth Method Considerations MAT7 to SAT10 (similar to what has been used since):

  • Reflects local input (Administrator’s Academy, Accountability Framework)

  • Live or dead growth norm

  • Local, national or state growth comparisons

  • Provides estimates of growth of student groups (ethnic, FRL, ELL, Spec Ed)

  • Multiple measures/methods preferred

  • As simple as appropriate, but no simpler.

  • Acknowledges real student growth patterns

  • Descriptive vs. VAM approach (Davison, Thum)

  • Consider link to reported growth history, and likely future

  • Consider link to formative assessment system


Additional SPPS MAT7 to SAT10 (similar to what has been used since): Considerations in Growth and Gain Models


Part 2 in what ways do different growth measures produce the same results
Part 2: In what ways do different growth measures produce the same results?

Growth categories (average, hi or low):

  • at the grade level within the school,

  • at the group level within the district


Some relevant items in cresst s standards for educational accountability systems baker et al 2002
Some Relevant Items in CRESST’s Standards for Educational Accountability Systems (Baker, et al., 2002)

  • (A1) Accountability expectations should be made public and understandable to all participants in the system.

  • (A2) Accountability systems should employ different types of data from multiple sources.

  • (A4) Accountability systems should include the performance of all students, including subgroups that historically have been difficult to assess.

  • (B9) The validity of a measure that has been administered as a part of an accountability system should be documented for the various purposes of the system.

  • (B10) If tests are to help improve system performance, there should be information provided to document that test results are modifiable by quality instruction and effort.

To what degree do these standards apply to School Improvement Systems?


Buried treasure implications celio harvey 2005
Buried Treasure – Implications Accountability Systems (Baker, et al., 2002)(Celio & Harvey, 2005)

  • Less may be more

  • Parsimony and power should be respected

  • Status data necessary but not sufficient

  • Potential to change board-supt relations

  • Cost-benefit analysis

  • Professional development and tech assistance will be required

  • State leaders have a significant role to play


Buried treasure indicators celio harvey 2005
Buried Treasure – Indicators Accountability Systems (Baker, et al., 2002)(Celio & Harvey, 2005)

  • Achievement (reading and math)

  • Elimination of the achievement gap

  • Student attraction (school ability to attract students)

  • Student engagement with the school

  • Student retention/completion

  • Teacher Attraction and retention

  • Funding Equity



So where are we in spps
So, Where Are We in SPPS? Carlson, 2002)

  • Put growth models in perspective (only part of one piece of the puzzle)

  • Transition from SAT10 to MCA-II growth

  • Need to dedicate resources to solid cost-benefit analyses, using multiple outcomes

  • Have to prioritize and commit to indicators so solid professional development and technical assistance is possible.


Partial List of References Carlson, 2002)

Baker, E. L., Linn, R. L., Herman, J. L., & Koretz, D. (Winter, 2002). Standards for Educational Accountability Systems. Policy Brief 5: Center for Research on Evaluation, Standards and Student Testing.

Celio.M.B & Harvey, J. (2005). Buried treasure: Developing a management guide from mountains of school data. http://www.crpe.org/pubs.shtml#leadership

Olson, L. (2005, July 13). Education Department convenes working group on ‘growth’ models. Education Week. Available at http://www.edweek.org/ew/articles/2005/07/13/42growth.h24.html .

O’Malley, K., Vansickle, T. & Housson, S. (2005, June). Measuring growth: Where policy makers and psychometricians meet. Paper presented at the annual national conference on Large-Scale Assessment. San Antonio, TX.

Thum, Y.M (2005, April). Designing school productivity indicators: Connecting goals, data & models. Paper presented at the annual meeting of the American Educational Research Association, Montreal.


ad