Week 3

1. Week 3 – Collecting qualitative data – Analysis of qualitative data – Quality of data – Planning Action Research – Report writing

2. Action Research Model • Mertler’s AR four stages:

3. CHAPTER 4 Developing a Research Plan

4. Developing a research plan • Define purpose of study • Articulate your research questions • Describe participants • Describe the intervention, if appropriate 4

5. Articulating Research Questions • Intervention Studies – What action will be taken? – What is the desired outcome? • Research question – Primary – What is the effect of _______ (intervention) on ____ (outcome) among/of/in ___(participants)? 5

6. Articulating Research Questions • Intervention Studies – What action will be taken? – What is the desired outcome? • Research question – Secondary – What are teacher/parent/student perceptions of the ___________ (intervention)? 6

7. Writing Activity 4.1: Articulating Research Questions • Research paper activity: Write a paragraph on the purpose of your study followed by the primary and secondary questions that are the focus of your investigation. Label this section Purpose of Study (or similar heading). Place this section of the paper after the literature review. 7

8. Participants • Students—which students? • Teachers—which teachers? • Staff—which staff? • Others? • A participant is anyone who can contribute to your study • Researcher is a participant 8

9. Participants • Keep project manageable • Many participants requires a large research plan • For your first action research project—keep it small 9

10. Writing Activity 4.2: Participants • Provide a heading for the participants and place this section as a subsection in the Methods section. 10

11. Planning the Intervention • The intervention should… – Be based on the best practices from the review of literature – Clearly describe the intervention you will use – Be written in enough detail so that other teachers or administrators can read the plan and use it in their settings – Describe the activities to be used and length/intensity • How long will the intervention last? • How long are individual sessions? • How much time will be devoted to the activities in the intervention? 11

12. Planning the Intervention • Options for an Intervention Plan…. Week 1 Wednesday 9:00-9:55 Review of rubric description Monday 9:00-9:55 Tuesday 9:00-9:55 Thursday 9:00-9:55 Friday 9:00-9:55 Rubric description Review of rubric description review of student- developed rubric categoreis and scoring levels. Additional time for rubric discussion and rubric refinement, as necessary Students will study example rubrics Students will review example rubrics Review of student- determined rubric categories Students work in pairs to score last month's book reports using newly developed rubric. Students generate ideas about what to incorporate into book report rubric. Student review scoring levels on example rubrics. Rubric scoring discussion/questions. Rubric refinement. Students develop scoring levels for book report rubric. 12

13. Writing Activity 4.2: Planning the Intervention • Research paper activity: Describe the intervention you will use in your study. Write in sufficient detail so a colleague could use your plan in his or her own setting. Provide a heading for the intervention and place this section after the section on participants. This is not necessary for studies that do not focus on an intervention (e.g., descriptive studies). 13

14. CHAPTER 5 Collecting Data

15. Action Research, Craig Mertler Questions to Ask About Collecting Data (5-2) • Why am I collecting this data? • What exactly am I collecting? • Where am I going to collect the data and for how long? • When am I going to collect the data and for how long? • Who is going to collect the data? • How will the data be collected and displayed? 15

16. Collecting Multiple Forms of Data • Triangulation –Collection and analysis of multiple forms of data –Enhances the validity and credibility of a research study 16

17. Collect Data • Quantitative – Sources of data • Survey (Questionnaires, rating scales) • Checklists • Archival data (school records: Tests etc.) • Qualitative – Sources of data • Observations (field notes) • Interviews • Journals • Existing documents and records 17

18. Organizing/Analyzing Quantitative Data • Gather quantitative data sources & record info • Create graphical displays of the data • Examine displayed data to determine best ways to explain results • Share your analyses and graphical displays with others • Make notes in your journal about how your analyses can be used to answer your research questions 18

19. Writing Activity 6.1: Analysis of Quantitative Data • Write your primary and secondary research questions. • Gather all quantitative data sources that will be analyzed in your study. Match each quantitative data source to the primary or secondary research questions. (some data sources may be used to answer more than one question.) • Create records for each data source. • Display results for each data source. Share your displayed data with a peer and ask for feedback. • Produce a written description to explain displayed data. 19

20. CHAPTER 6 Report Results

21. Writer’s Workshop Study •Research question: What is the effect of Writer’s Workshop on third-graders’ writing achievement? •What’s the effect of the workshop on student writing confidence and writing attitudes? •Data collection: student essays, student surveys, teacher observations •Data analysis: 1. Writing Achievement. overall writing performance at baseline was compared with writing performance after implementing workshop. Writing component scores were also compared. individual student growth in writing was also analyzed. 21

22. Writer’s Workshop Study •Data analysis: 2.Writing confidence and attitudes. Initial survey responses were compared with responses obtained after writers’ workshop. responses to the survey items were averaged, and comparisons were made between responses on the baseline survey and responses on the final survey. 22

23. Sample Quantitative Results 23

24. Sample Quantitative Results • The average total score on the persuasive essay, which students wrote before we began the Writer’s Workshop intervention, was 6.4. Scores increased throughout the intervention phase of the study. On the second essay, average total scores rose to 7 points, and on the third essay, total scores rose once again to 8.1. On the final essay, students’ average total score was 9.5- just over a three point gain compared to the baseline data. 24

25. Sample Quantitative Results 25

26. Sample Quantitative Results • Throughout the Writer’s Workshop intervention, students improved in content, mechanics, and transitions. Their greatest gains, however, were in the area of transitions. Baseline data indicated that the average score for students in this area was 1.5, which was just below the basic level. Students scores gradually increased during the intervention, and on the last essay, students’ average score on transitions rose to 3.1, which is at the proficient level. 26

27. Sample Quantitative Results 27

28. Sample Quantitative Results • For each of the four closed-ended survey items, students’ confidence in their writing abilities and attitudes toward writing improved. On the baseline survey, most students strongly disagreed or disagreed that they know how to use correct grammar and transitions when writing essays. On the final essay, most students agreed or strongly agreed that they could use correct grammar and transitions when writing. There was also a change in students’ attitude toward writing. On the baseline survey, the average response to the prompt ‘I like to write’ was between strongly disagree and disagree. On the final survey, more students agreed that they like to write. Although students’ confidence in their writing abilities and their attitudes about writing increased, there was little change in students’ responses to the prompt ‘I am a good writer’. On both the baseline and final surveys, the average student response was between disagree and agree. 28

29. Conclusions • Conclusions are not the same as findings and should not simply be restatements of findings from Results section. • A conclusion should be broader and more encompassing than a specific finding, and several findings may be incorporated into one conclusion. • Specific findings are generally stated in the past tense (e.g., Only 10% were female and 5% were non-white in the sample of superintendents),while conclusions are stated in the present tense (e.g., Women and minorities are underrepresented in the position of public school superintendents). 29

30. Conclusions • Example • Results: Students writing improved by just over three points from the first essay to the final essay. Further analysis revealed that students writing improved in all areas assessed – content, mechanics, and transitions. The greatest improvement was in the area of transitions. • Conclusion: Writer’s Workshop has a positive impact on improving student essay writing. • Results: survey – refer to earlier slide • Conclusion: Writer’s Workshop improves students’ confidence in their writing abilities • Conclusion: Writer’s workshop improves students’ attitudes toward writing. 30

31. Analyze Quantitative Data using Pivot Table • An efficient tool when analyzing large data sets, e.g. building and district level data sets • Easily accessible – Excel component • Compatible with Google docs

32. Analysis of Qualitative Data: Process • Phase 1: Compiling – Compile data – Sort data – Transcribe interviews, focus groups, conferencing sessions (audio, video) • Phase 2: Disassembling – Break data down into smaller pieces or categories through a coding process – Coding evolves through review 32

33. Analysis of Qualitative Data: Process • Phase 3: Reassembling – Coded data are reassembled into themes • Phase 4: Interpreting – Described to tell the story revealed in the data • Phase 5: Concluding – Conclusions are drawn from the interpretation of the data 33

34. Writing Activity 6.2: Analysis of Qualitative Data • Write your primary and secondary research questions • Gather qualitative data sources. Transcribe any data from audiotapes or videotapes • Read text sources several times and over several days. Code data. • Create a codebook. Define codes and illustrate them with quotes or example from text sources. • Write the major findings from the analysis of the qualitative data in your study. Describe the patterns and themes specifically related to your research questions that emerged. • Refer to your responses to quantitative data analysis. Look for ways that the results of the different types of data you have collected support each other. 34

35. Sample Qualitative Study • Research question: What are major problems that need attention in the organization? • Data collection: open-ended survey question from 30 employees • Data analysis: Respondents’ comments were analyzed by a group of teacher researchers. Each teacher researcher first independently coded the comments and grouped the codes into broad themes. They then compared their coding and themes. Minor differences existed in coding and they were resolved through discussions. 35

36. Reporting Result for Qualitative Study • Specific problems that employees mentioned were grouped into four issue areas: communication, management, work environment, and training. • Eleven responses relating to issues in communication. One respondent mentioned clearly that ‘Communication needs improving’. Another mentioned ‘There is a lot of ‘us and them’ sentiment here’. • Twelve responses relating to issues in management. Some example responses were provided here: ‘We need consistent application of policy”, Two many signatures are required”, 36

37. Triangulating Data Sources & Drawing Conclusions • Triangulation: Corroborating multiple sources of data • Increases various types of validity • Supporting results from one data sources with the results from another data source • Example of how this triangulation would be written up (See next slide) • Conclusions are made about what was learned in the study • Results are used for ongoing and continuous reflective planning 37

38. Triangulating Data Sources & Drawing Conclusions • Example: Student achievement increased, which was evident when looking at their tests, quizzes, and written assignments. This increase in achievement is supported by observations that revealed higher student motivation, students working harder to complete classwork, and an increase in classroom participation during lessons. Further, students said that because of the new information we were studying, they were spending more time working on assignments and studying for the class. 38

39. CHAPTER 6 Quality relating to Study

40. Discussion • You did a study looking at if Writer’s Workshop intervention increases students’ achievement in writing. • Below are quantitative data you collected • Baseline student essay scores • Analysis: comparing baseline essay scores with post intervention scores • Findings: Student writing achievement increased following the intervention • Conclusion: intervention is working • Question: how much confidence do we have in the conclusion?

41. Major Research Methods • Quantitative research methods – Require numerical data – Utilize deductive reasoning (‘top-down’ approach) • Qualitative research methods – Require narrative data – Utilize inductive reasoning (‘bottom-up’ approach) • Mixed methods – Studies that combine both quantitative and qualitative data – Many individuals consider action research studies to be most similar to mixed-methods research (than purely quantitative or qualitative research)

42. Quantiative Research Design • Descriptive designs • purpose is to describe and make interpretations of current status of individuals, objects, conditions, or events. • Survey research • Correlational designs • purpose is to measure and describe statistical relationship between two or more variables • Typically use correlation coefficient • Group comparison designs • attempt to investigate cause-and-effect relationships by comparing two or more groups that differ on some characteristic • Experimental and non-experimental designs

43. Quantitative Data Collection Techniques • Quantitative data are numerical • Variety of techniques: – Surveys, questionnaires, rating scales—verbal or written administration of set of questions or statements to sample of people • Closed-response • Likert and Likert-type scales • Checklists • Follow guidelines and suggestions for developing instruments – Tests and other formal instruments

44. Quality of Quantitative Data • Validity—extent to which you actually measured what you intended to measure – Must be appropriate and accurate for your purposes – Seen as a unitary concept, combining content, concurrent, predictive, and construct validity (focus should be on evidence) Reliability—refers to consistency of collected data – Determined by correlating results with themselves or with another quantitative measure • – Three methods: • Test-retest reliability • Equivalent forms reliability • Internal consistency reliability • ‘A valid test is always reliable, but a reliable test is not always valid.’

45. Rigor of Quantitative Study • Can you trust the conclusions of the study? • Internal Validity: The extent to which the outcomes of the study result from the variables manipulated, measured or selected rather than from other variables not systematically managed. • External Validity: the extent to which the findings of a particular study can be generalized to people or situations other than those observed in the study.

46. Strategies to Increase Validity Type of Validity Focus Strategies Internal Validity Cause-effect Relationship Control Group Random Assignment Pretests Instrumentation External Validity Generalizability Sample size Effect size Replication

47. Best Practices in improving rigor of quantitative studies • Control Group: a group of subjects whose selection and treatment are exactly the same as those of the experimental group except that the control group does not receive the experimental treatment. Random Assignment: a method for assigning subjects to control and experimental groups. Not to be confused with random selection (a method for selecting a sample of subjects from a population). Pretests: When random assignment is impossible or undesirable, pretests can be used to examine prior existing differences between groups and to statistically adjust for these differences. Instrumentation: select valid and reliable instruments to minimize measurement errors. Sample size: Large sample size from representative sample is desired. Effect size: report effect size. Replication: perhaps the best way to ensure internal and external validity • • • • • •

48. Discussion • You did a study looking at if Writer’s Workshop intervention increases students’ achievement in writing. • Below are qualitative data you collected • Teacher interviews; student interviews; your observations • Analysis: triangulate data for themes • Findings: Teachers and students enjoyed the Writer’s Workshop. They think the Workshop motivates students to write. • Conclusion: intervention is working • Question: how much confidence do we have in the conclusion?

49. Qualitative Data Collection Techniques • Qualitative data are narrative • Variety of techniques: – Observations—carefully watching and systematically recording what you see and hear • Structured/semi-structured/unstructured observations • Recorded using field notes, videotapes • Should include observer’s comments in the field notes – Interviews—directly asking people questions (as opposed to watching them); conversations between researcher and participants • Prepare an interview guide (may be specific or general) • Several types: – Structured/semi-structured/unstructured interviews – Individual, phone, Focus group

50. Qualitative Data Collection Techniques – Journals—means of gathering data to provide insight into workings of a classroom • Types of data journals: – Student journals – Teacher journals – Class journals – Existing documents and records— schools are filled with existing sources of data… – Classroom artifacts—as are classrooms!