EDN 523Educational Research Validity and Educational Reform
Accountability Models In the current age of educational reform, high-stakes decisions based on large-scale testing performance are becoming increasingly common. The decisions associated with test performance carry significant consequences (e.g., rewards and sanctions). The degree of confidence in, and the defensibility of test score interpretations depends upon valid, consistent, and reliable measurements. Stated differently, as large-scale assessment becomes more visible to the public, the roles of validity (and reliability) become more important.
The Validity of Test Scores Validity is a test’s most important characteristic. Why? It is the degree to which a test measures what it is supposed to measure and as a consequence allows and supports appropriate interpretation of the scores.
Types of Test Validity • Construct Validity – the degree to which a test measures an intended hypothetical construct. • Content Validity – the degree to which a test measures an intended content area.
Construct Validity • The most important question to ask: What is this test really measuring? • As Professor Kozloff shared with you last week, we must first define what we want to measure: Constructs or concepts can be non-observable traits, such as intelligence, which are “invented” terms to explain educational outcomes.
Constructs • Constructs underlie the variables we want to measure. • You cannot see a construct; you can only observe its effect. • You can however, use constructs to explain differences among individuals.
For Example • It has been observed that some students learn faster than others; • Some students learn more than others; • And some students retain what they learn longer than other students Using this information, we created a construct called intelligencethat is related to learning and that everyone possesses to a greater or lesser degree.
A Theory of Intelligence • From the observed and collected data, the construct of intelligence led to the development of a theory of intelligence. • Tests were developed to measure the “amount” or how much intelligence a person has. • Sometimes (at least occasionally) students’ whose achievement and/or test scores indicate that they have a lot of “it” tend to do better in school and other learning environments than those who have less of “it.”
Construct Validity • Research studies involving a construct are valid only to the extent that the test or instrument used actually measures the intended construct and not some unanticipated or intervening variable. • Determining test construct validity is not easy!
For Example: Billy Bob’s IQ Test Let’s say Billy Bob owns a testing company in Texas. Billy Bob has designed an IQ test that he is marketing to public schools across the country. If we wanted to determine if Billy Bob’s IQ Test was construct valid, then we would need to carry out several validation studies.
Validating Billy Bob’s Test • First, we could see if students who scored “high” on Billy Bob’s Test learned faster, learned more, and retained more of what they learned than students who scored “low” on his test. • Next, we could compare scores on Billy Bob’s Test at the beginning of the year with student’s grades at the end of the school year. • We could also compare students’ scores on Billy Bob’s Test with scores on other, well-established IQ tests to see if they were highly related.
Professor Kozloff’s Findings Using Billy Bob’s Test and relating it to Dr. Kozloff’s presentation last week…if the findings “say” that a student has an IQ of 140 and this is confirmed whencompared with the student’s scores on other, well-established IQ tests and other measurements, then these “matches” strengthen the constructvalidity or validness of Billy Bob’s Test as an intelligence measuring instrument.
What about Test Content Validity? Content Validity is based on professional judgments about the relevance of the testcontent to the particular domain of interest, such as the NC Standard Course Of Study, and about the representativeness with which items and/or task content on the instrument “cover” the domain.
NC ABCs of Accountability The Content Validity of the NC ABC Accountability Tests is directly related to whether or not the items, tasks, and concepts are aligned with the domains, constructs, and/or variables we are attempting to assess and measure. The establishment of evidence of test relevance and representativeness of the “target” domains is a critical first step in being able to say that the test score interpretations are based on strong content validity.
Content ↔Alignment ↔ Construct • Alignment is a key issue in as much as it provides one avenue for establishing evidence for score interpretation. Validity is not a static quality; it is an evolving property and validation is a continuing process. • Evaluating test alignment should occur regularly, taking its place in the recurring process of assessment development and revision of the testing instruments. • An objective analysis of alignment as tests are adopted, built, or revised ought to be conducted on an ongoing basis. • This is a critical step in establishing evidence of the validity of test score or performance interpretation
Test Evaluation Professor Kozloff discussed program decision making last week relative to evaluating whether or not a new reading program would be needed if students have low reading test scores. He recommended that we ask the following questions: How was the reading evaluated? Did the reading test DIRECTLY measure reading skills?
For Example: • Did students get points if they guessed at what words said? • Did students get points off if they guessed rather than sounded out the words? • Are we measuring reading or are we measuring guessing? • Did the test use objective/quantitative data or subjective/qualitative data?
What are we really measuring? Math Example: If we simply look at data on math ability among individuals, it initially appears that shoe size is directly related to a person’s math skills. Each time we test or measure a person’s math ability and compare it to their shoe size, there is a 1:1 positive correlation; the larger a person’s shoe size, the better they are in solving higher level math problems. How is that possible?
A Valid Measurement: Age Age Shoe Size ↔ Math Ability
A Valid Measurement Education Level ↔ Earning Power College Graduate’s Earning Power High School Graduate’s Earning Power Are they related? You bet your derriere they are! College graduates average lifetime earnings = $ 2.2 million HS graduates average lifetime earnings = $ 1.2 million Difference = $ 1 million 30 year work life/$ 1 million = $33, 333 more per year
Cost of a College Education • The average yearly cost at a public 4-year college is $8,655 for in-state tuition, room and board. • Over a four year period, the total cost for a college degree, considering increases and miscellaneous expenses, is approximately $40,000. • This data supports the contention that, though the cost of higher education is significant, given the earnings disparity that exists between those who earn a bachelor's degree and those who do not, the individual rate of return on an investment in higher education is sufficiently high enough to warrant the cost.
Individual and Community Benefits • Rate of return for college educated individual is almost twice as much income during his or her work life than a high school graduate. • A college educated person contributes taxes into the national coffers and buys goods, services, and products to keep the country’s economy humming (anadditional $425 billion by 2015) • A person with a college degree typically makes better lifestyle choices which decreases health costs and burdens on medical support services.
Cost of An Uneducated Person It is estimated that there are 3.8 million youth between the ages of 18 and 24 who are neither employed nor in school—roughly 15 percent of all young adults. Since 2000 alone, the ranks of these non-engaged young adults grew by 700,000, a 19 percent increase over 3 years (Annie E. Casey Foundation, 2004).
High School Dropouts – A National Crisis • Almost 70% of prisoners did not complete high school. • There appears to be some kind of relationship between dropping out of high school and a life of crime. • Limited alternatives, easily turned to the darkside, and being reinforced for criminal behavior.
The Cost of Crime • What is the impact of poorly educated citizens on our economy and quality of life? • What are the costs?
Estimated Cost of Crime in the US Two Data Sources: U.S. Department of Justice Office of Justice Programs Bureau of Justice Statistics Data, (2005). D. A. Anderson, (1999) "The Aggregate Burden of Crime," Journal of Law and Economics,
1. Cost of Crime: Police, Courts, and Prisons [ Source: Justice Expenditure and Employment Extracts $70 Billion $60 Billion $40 Billion Total = $ 170 Billion
2. Crime-Related Production Costs? Cost of goods and services that would be unnecessary in a country with low crime rates. • Money spent on locks and safes = $4 billion • Surveillance cameras = $1.4 billion • Computer virus screening and security = $ 8 billion • Drug trafficking = $160 billion • Federal agencies to fight crime = $23 billion • Medical care to treat the victims of crime = $8.9 billion • Newborns exposed to cocaine & heroine = $28 billion Total = $ 233.3 billion
Opportunity Costs of Crime In addition to the direct cost of resources devoted to crime, there is a sizable loss of time by people who are potential victims of crime and by those who have committed crime. And as the saying goes - time is money.
Time Lost to Crime • If there were no crime, there would be no criminals. Instead of sitting idle in jails, these people - over 1 million of them - could be productive members of the economy, so chalk up $35 billion. Also, there is much time lost in planning and executing crimes - $4.1 billion worth. • Then add to that the value of work time lost by victims ($876 million) and the time spent on neighborhood watches ($655 million).
Total for Lost Time Due to Crime • $39.1 Billion – Prisoners not working • $ 1.5 Billion –Victims lost work time • $ 4.1 Billion –Time planning & committing crimes Total = $ 44.7 Billion
Life and Health Crime Costs • We accounted for the cost of medical expenses, but we should also consider the cost of a life and injury in addition to the direct costs incurred by the medical system. • There are approximately 72,000 crime related deaths a year and 2.5 million crime related injuries every year. • Let’s put a dollar value on a life and injuries. What is a life worth? Conservative estimate: $80,000. An injury: $52,637. • With these numbers, the total cost of crime for lost life and injuries comes to $ 137.8 billion
Crime Transfer Costs - Fraud We also need to add the cost of property and money that is stolen or obtained through fraud. • Fraud at work $203 Billion • Unpaid taxes $123 Billion • Health insurance fraud $108 Billion • Auto theft $8.9 billion • Coupon fraud racks up $912 million annually. Total comes to $603 billion.
Grand Total (Conservative Estimate) • Police, Courts, and Prisons = $ 170.0 B • Crime-Related Production Costs = $ 234.0 B • Lost Time Due to Crime = $ 44.7 B • Life and Health Crime Costs = $ 137.8 B • Crime Transfer Costs (Fraud) = $ 603.0 B Grand Total = $1.12 Trillion Dollars Per Year
An Uneducated Person: Impact Effects • Adults without a high school diploma are twice as likely to be unemployed. • They will earn $260,000 less over a lifetime than a high school graduate and $1 million less than a college graduate • Dropouts make up nearly 70 percent of inmates crowding state prisons and at least half those on welfare. • Crime costs our country $1.12 Trillion Dollars A Year
The High School Diploma Gateway or Barrier to a prosperous, successful life If you have it, you have opportunities If you don’t have it, you have a cross to bear
Failure to Graduate High School What is the primary reason students drop out of school and do not graduate with a diploma? THEY CANNOT READ!
So, What is the Big Deal About Reading? Let’s see…educated citizenry, economic well-being, secure and prosperous nation, successful individuals capable of providing for themselves and their families…what do you think?
NC Reading Legislation: Senate Bill 16 Passed in 1995 sponsored by then state senator Beverly Perdue, required the State Board of Education to reorganize the NCDPI and to develop an accountability plan for the state. The result was the ABCs of Public Education, which included a plan to revise the Standard Course of Study. The General Assembly accepted the accountability plan and subsequently enacted Senate Bill 1139, the School-Based Management and Accountability Program.
Senate Bill 1139 • Sponsored by former Senator Leslie Winner (D-40th Dist.), established the requirement that the State Board of Education develop a comprehensive reading plan for North Carolina as part of the revised Standard Course of Study. • According to the bill, “the General Assembly believes that the first, essential step in the complex process of learning to read is the accurate pronunciation of written words and that phonics is the most reliable approach to arriving at the accurate pronunciation of a printed word.”
Outcomes of Senate Bill 1139 • The bill mandated that the reading plan include early and systematic phonics. • A member of a prominent reading interest group, who shall remain nameless, (but their initials are Whole Language) called the bill the “phonics legislation.” • A representative from the NCDPI spoke to the legislature’s focus on phonics and suggested that, “some members of the legislature had the opinion that teachers weren’t teaching phonics, that the ‘whole language movement’ had gotten the upper hand in the state and that reading skills were not being taught.”
Reading Legislation • In 1997, Congress asked the Director of the National Institute of Child Health and Human Development (NICHD) at the National Institutes of Health, in consultation with the Secretary of Education, to convene a national panel to assess the effectiveness of different approaches used to teach children to read. • For over two years, the National Reading Panel (NRP) reviewed research-based findings on reading instruction and held open panel meetings in Washington, DC, and regional meetings across the United States.
NRP Research Report • The National Reading Panel only looked at studies that met the most rigorous standards of scientifically based research. • On April 13, 2000, the NRP concluded its work and submitted "The Report of the National Reading Panel: Teaching Children to Read," at a hearing before the U.S. Senate Appropriations Committee's Subcommittee on Labor, Health and Human Services, and Education.
Impact of NRP Report • Part of the No Child Left Behind Act of 2001, signed into law on January 8, 2002, was President Bush's unequivocal commitment to ensuring that every child can read by the end of third grade. • To accomplish this goal, the new Reading First Initiative significantly increased the federal investment in scientifically based reading instruction programs in the early grades.
Reading First • NCLB established Reading First as a new high-quality, evidence-based program for students • The new Reading First State Grant program makes six-year grants to states, which in turn make competitive subgrants to local communities. • Building on a solid foundation of research, the program is designed to select, implement, and provide professional development for teachers using scientifically based reading programs, and to ensure accountability through ongoing, valid and reliable screening, diagnostic, and classroom-based assessment.
Reading First: SBR Research The Goal of North Carolina's Reading First (NCRF) initiative is to ensure that all children learn to read well by the end of the third grade. This goal will be accomplished by applying scientifically based reading research to reading instruction in all North Carolina schools. The initiative requires phonics instruction.
Phonics Model of Reading Instruction • Phonemic Awareness Attentiveness to the sounds of spoken language. • Phonics Decoding unfamiliar words using knowledge of the alphabetic principle. • Fluency Grade-appropriate oral reading with appropriate speed, accuracy, and expression. • Vocabulary Development Knowledge of word meanings to facilitate effective spoken and written language communication. • Text Comprehension Use of a variety of comprehension strategies to monitor comprehension to construct meaning from print.
SBR and Standardized Tests • Test Construct Validity - Research studies involving a construct are valid only to the extent that the test or instrument used has been determined by SBR to actually measure the intended construct and not some unanticipated or intervening variable. • Test Content Validity - based on the relevance of test items and tasks to the domain of interest (in our case the NC Standard Course Of Study), and the representativeness with which items and tasks on the instrument “cover” the domain.
Content ↔Alignment ↔ Construct Alignment is a key issue in standardized tests, as much as it provides a means for establishing evidence for score interpretation. Test validity is not a static quality; it is an evolving property and a continuing process.