1 / 33

Conducting Scientifically-Based Research in Teaching with Technology, Part II

Conducting Scientifically-Based Research in Teaching with Technology, Part II. SITE Annual Meeting Symposium Atlanta, Georgia Gerald Knezek & Rhonda Christensen University of North Texas Charlotte Owens & Dale Magoun University of Louisiana at Monroe March 2, 2004.

cain
Download Presentation

Conducting Scientifically-Based Research in Teaching with Technology, Part II

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Conducting Scientifically-Based Research in Teaching with Technology, Part II SITE Annual Meeting Symposium Atlanta, Georgia Gerald Knezek & Rhonda Christensen University of North Texas Charlotte Owens & Dale Magoun University of Louisiana at Monroe March 2, 2004

  2. Scientifically-Based Research(Whose Definition?) • Methodology Issues • Randomization Issues • Instrumentation Issues • Determination of Impact • Analysis/Interpretation Issues

  3. Issues of Methodology • Quantitative • Currently in favor, heavy on analysis methodology • Qualitative • Rich analysis, takes longer • Mixed Methods • Seeing process in operation often necessary to find out ‘why’ in education • Theory Building vs. Theory Testing • Exploratory/Data Mining vs. Hypothesis Testing

  4. Issues of Randomization • Random assignment (currently emphasized) • For internal validity (fidelity of experiment) • Start with large group • Randomly assign 1/2 treatment, 1/2 control (Versus) • Random sampling • Drawing from larger population • For generalizability to larger population • External validity (Trust that this would work elsewhere) • Also very important

  5. Issues of Instrumentation • Much emphasis on standardized outcome measures as ultimate (valid) criteria • Less attention to reliability/accuracy of legislated tests and measures • Little attention to how/where/when (or numerous other holes in) the data gathered • Mistrust of teacher self appraisal/reflection

  6. Analysis of Impact • Hypothesis testing • Is the impact real (not due to chance?) • P = .05, .01, or none (vs. confidence intervals?) • Effect size as a yard stick of impact • Report p - level and ES

  7. What is Effect Size? • “… it is convenient to use the phrase ‘effect size’ to mean ‘the degree to which the phenomenon is present in the population,’ or ‘the degree to which the null hypothesis is false.” (Cohen, 1977, p. 9) • Effect size is a standardized measure of the strength (degree of impact) of a discriminating feature or intervention.

  8. How to Interpret Effect Size • Cohen’s d (1965, 1977, 1988) vs. other • Small (.2), medium (.5) vs. large (.8) .2: IQ difference between twins vs. non-twins height difference between 15 and 16 yr. old girls .5: large enough to be visible to naked eye height difference between 14 and 18 yr. Old girls .8: IQ difference between college freshman and Ph.D.s height difference between 13 and 18 yr. Old girls • Compare to other common effect sizes • “As a quick rule of thumb, an effect size of 0.30 or greater is considered to be important in studies of educational programs.” (NCREL) • For example .1 is one month learning (NCREL) SRI International. http://www.ncrel.org/tech/claims/measure.html

  9. Effect Sizes are Known for Many Interventions (Ex: Dede’s Optimal Areas of Information Acquisition, 1990) 1. Learners construct knowledge rather than passively ingest information: Acceleration (Study 1) 1.00 Acceleration (Study 2) .57 Individualized Instruction (Study 1) .32 Individualized Instruction (Study 2) A. Curriculum compacting .83 B. Credit for prior learning .5 2. Sophisticated information-gathering tools are used to stimulate the learner to focus on testing hypotheses rather than on plotting data: Higher Order Questions(Study 1) .34 Cognitive Processing (Study 3) .69 3. There is collaborative interaction with peers, similar to team-based approaches underlying today's science (Note that in constructivist methodology the teacher is considered a peer): Reinforcement (Study 1) 1.17 One to One Mentoring (Study 2) .57 Social Skills (Study 3) .47 Hancock, R. J. (2003). The Expanded Will, Skill, Tool Model: A Step Toward Developing Technology Tools That Work. Paper presented to EdMedia 2003, Honolulu, Hawaii.

  10. Issues of Analysis/Interpretation • Much attention to single ‘correct’ procedure • T-test of differences vs. Analysis of Covariance • Power estimates for hierarchically nested data • Little recognition of value of multiple views of data • Nonparametric techniques for small samples • Too much emphasis on accept/reject null and too little on strength of effect (ES/APA) • Tendency to use no data to make decisions rather than rely on less than perfect information

  11. It’s all About Confidence As shown in Figure 1, three of the measures’ 95% confidence intervals … are roughly 3/4 of a confidence interval band above … that is, no more than 1/4 of the 95% confidence interval range overlaps from the upper to the lower group. Differences in this range are as a rule-of-thumb “meaningful” according to emerging APA guidelines, and roughly comparable to a p = .05 level of significance (Cumming, 2003). The effect size for the combined upper three versus the lower two is approximately [((3.09+3.05+2.95)/3) – ((2.32+2.41)/2]/ ((1.34+1.33+1.40+1.00+1.05)/5) = (3.03 – 2.37) / 1.22 = .66 / 1.22 = .54, considerably larger than the .30 cutoff beyond which technology interventions are considered meaningful (Bialo & Sivin-Kachala, 1996). Teachers rated the ARTS to the Delta class as much more useful for promoting interest in music and creating a positive effect on students’ overall education experience that for improving reading and math skills.

  12. We’ve Long Been in the Credible Evidence Business • Research based on dissertation criteria • Quantitative to tell us what is happening; Qualitative to tell us why it is happening

  13. KIDS Project: Technology Innovation Challenge Grant • KIDS - Key Instructional Design Strategies • 9.2 million 1999-2004 TICG • Replication of successful model to 50 rural school districts • Major Project Components • Technology Integration Professional Development • Technology-enhanced Reading Instruction • External Evaluator: Univ. of North Texas • Research Agenda added to Project Evaluation

  14. 1999 - 2002 Findings • High integration teachers make a critical difference for students without computers at home. • Elementary school girls have equal or higher attitudes toward computers than boys. • Technology skills of high school students are often higher than their teachers. • Rural teachers and students fall between Allen ISD and Laredo on baseline measures of technology proficiency. • Technology-infused reading activities accounted for approximately 10% of reading achievement gains.

  15. Teacher Stage of Adoption vs. Home Access 2001(SITE 2003 Research Award)

  16. Trends in Computer Enjoyment - 2001 sample (Girls & Computers, NECC 2003)

  17. 2002-2003 Experimental Design • 7 randomly selected control districts • Compared with 18 treatment districts • Interventions: • Summer Institute (Eisenhower Model) • Tools to integrate into the classroom • New technology-enhanced reading program

  18. Technology Self Efficacy Gain(Pre-Post, 40-hour Summer Inst.)

  19. Technology Self Efficacy Gain(Pre-Post, 40-hour Summer Inst.)Technology Proficiency Self-Assessment by Ropp, 1999

  20. Is this good? • “In keeping with American Psychological Association reporting guidelines (APA, 2001, p. 25), and in accordance with standards established by other scholarly sources, the indications of this generalized approach to reporting gains in terms of standard deviation units are that the summer 2002 professional development institute had a moderately large effect on the technology integration skills of teachers (Cohen, 1969) and resulted in gains well beyond the .3 benchmark commonly regarded as indicative of educational significance (Bialo, 1996)”.

  21. 2002-2003 Student Achievement Findings • Treatment gained more than controls (p<.05) • Reading accuracy - Grade 1 and 2 • Reading comprehension - Grade 2 • Average Effect Size = .33 (range = .23 - .89) • Many reading programs were used in typical classrooms and some were very effective. • Students of teachers attending Institute gained more (2 times ES).

  22. Grade 1 vs. 2 Reading Gains

  23. Research Design Template • Evaluation Planned/Required: • Annual report to Dept. of Ed, 5-year summative • Research Question 1: • Is the KIDS Summer Inst. effective in promoting technology integration among teachers? • Research Question 2: • Is there a positive impact of the KIDS technology-based reading program on student achievement?

  24. Research Design Template (cont.) • Dependent Variable(s): • Gains in Level of Technology Integration (teachers) • Reading Achievement Gains (Grade 1-3 Students) • Independent Variables: • Teachers: Before vs. after training • Students: Treatment vs. Control classrooms/schools/districts • Randomization Possible/Control Group: • Yes, 7 districts randomly selected from pool of 150 matching treatment group membership criteria

  25. Instrumentation • For Teachers: • Several instruments for attitudes, skills, and level of technology integration capability • Reliabilities range from .78 to .95 for typical teachers • For Students: • Texas Primary Reading Inventory (TPRI) • Used by more than 90% of Texas districts in K-2 • Reliability reported as ‘high’ by creators • Story discontinuity and 2nd grade ceiling effect reported by Knezek, Christensen & Dunn-Rankin (2003).

  26. Data Analysis • T-tests of treatment vs. control gain scores • Reading accuracy • Reading comprehension • Analyis of covariance not carried out due to violations of assumptions • Effect Size computations carried out using Cohen’s d (in spite of paired data)

  27. Outlet for Findings • SITE 2004 ;-) • PT3 Leadership Institute Presentations • Electronic Newsletter/Web Site • 4th Annual Book on Project Findings nearing completion

  28. Significance/Implications of Findings • Technology - enhanced reading can be effective in promoting higher achievement in 1st and 2nd grade students • The average effect size is “educationally meaningful”, on the order of an additional 3 months of achievement gain over 1 school year. • The model has demonstrated “strong” evidence of effectiveness

  29. Has KIDS Established “Strong Evidence” of Effectiveness?

  30. For Additional Information • View KIDS Project findings at http://iittl.unt.edu • Contact: Gknezek@tenet.edu or • RhondaC@tenet.edu

More Related