Understanding the Research Process Research Reports: Knowing What To Look for and Why

1. Understanding the Research Process & Research Reports: Knowing What To Look for�and Why!

3. Orientation to Session�s Materials & Methods Description, Content Outlines, & Performance Objectives for the Session--Handout Chapter 3/Unit 1 from Hymel (2006)--Handout Chapter 4/Unit 2 from Hymel (2006)--Handout PowerPoint Slides & Overhead Transparencies Didactic Presentation (with Attempts at Humor) Analyses of Three Published Massage Therapy Research Studies Ongoing Questions & Discussions Occasional �Exercises in Accountability�--Handouts Planning for Session Follow-Ups

4. A Few Guiding Thoughts . . . �I never faced a problem which was more than the eternal problem of finding order� (B. F. Skinner, in the context of discussing scientific practice). �Everything should be made as simple as possible, but not simpler� (Albert Einstein). �Live as if you were to die tomorrow; learn as if you were to live forever� (Gandhi). �Blessed are the flexible, for they shall not be bent out of shape� (Michael McGriffy, MD)

6. Introduction & Overview:Research Universe Quantitative Research Qualitative Research Integrative Research

7. Introduction & Overview:Research Continuum(General to Specific) Research Category (Most General) Research Strategy Research Method Research Design Research Procedure (Most Specific)

8. Introduction & Overview:Quantitative Research Category Difference-Oriented Research Strategy True Experimental or RCT Method Quasi-Experimental Method Single-Case Experimental Method Nonexperimental Comparative Groups Method Association-Oriented Research Strategy Correlational Method Predictive Method Descriptive-Oriented Research Strategy Single-Case Quantitative Analysis Method Survey method Naturalistic/Structured Observational Method Case Report Method

9. Introduction & Overview:Chapter 3/Unit 1� An Overview of the True Experimental/RCTResearch Method Chang, M. Y., Wang, S. Y., & Chen, C. H. (2002). Effects of massage on pain and anxiety during labor: A randomized controlled trial. Journal of Advanced Nursing, 38(1), 68-73.

10. Clarification of Terms Measurement Statistics Research Evaluation

12. True Experimental, orRandomized Controlled Trial,Research Method First Example from the Massage Therapy Research Literature: Chang, Wang, & Chen (2002) Abstract (p. 24 in Chapter 3 Handout)

14. �Jumping Ahead��Schematic Illustration of theChang et al. (2002) Study Flowchart of Sorts (p. 32 in Chapter 3 Handout) Reflecting Only Select Features of, & a Slight Modification in, the Study by Chang et al. (2002)

15. Schematic (cont.) of Chang et al. (2002) Study Research Question & Professional Literature Review Population & Sample Randomization: Random Selection & Random Assignment

16. Schematic (cont.) of the Chang et al. (2002) Study:Variables Independent Variable Dependent Variable Extraneous Variable Control Variable Confounding Variable Intervening Variable

18. Schematic (cont.) of the Chang et al. (2002) Study:Hypotheses Research Hypothesis Null (or Statistical) Hypothesis) Alternative Hypothesis

19. Schematic (cont.) of the Chang et al. (2002) Study:Parameter & Statistic Parameter = Characteristic of a Population Statistic = Characteristic of a Sample Parameter : Population :: Statistic : Sample Relationship to Research Hypothesis, Null Hypothesis, & Alternative Hypothesis

20. Schematic (cont.) of the Chang et al. (2002) Study:Statistical Analysis Statistical Testing & Inference Alpha (a) Level (or, Probability of a Type I Error) P Value (or, Level of Significance) Statistical Power (or, Power Analysis) Confidence Interval Effect size

21. �Off on a Tangent�:A Simple Example of What�s Happening in Null Hypothesis Significance Testing Claim: Hester Sue Has ESP Options: Take It on �Faith� or Empirically Investigate Null Hypothesis: Hester Sue Does Not Have ESP Research Hypothesis: Hester Sue Does Have ESP Task & Observations: Hester Sue Is Asked To Predict the Outcomes of 10 Flips of a Coin, & We Observe the Results Analysis: Compare Observed Results with Expected Probabilities (Jaccard & Becker, 2002, p.170, next slide) Decision: With a = .05, Reject Null Hypothesis If Observed Results Have a Probability of Happening < .05

23. Schematic (cont.) of the Chang et al. (2002) Study:Experimental Validity Internal Validity Issue: Nature of Relationship between Independent Variable & Dependent Variable Driven by Random Assignment & Manipulated Independent Variable External Validity Issue: Generalizability Driven by Random Selection

24. Schematic (cont.) of the Chang et al. (2002) Study:Experimental Research Design Notation See p. 31 in Chapter 3 Handout �Shorthand� for Identifying Research Category, Strategy, Method, Design, & Procedures

26. Measurement & Statistics As Research Tools Valid & Reliable Measuring Instruments: A Necessity Measurement Scale & Research Issues: Determinants of Statistical Usage Descriptive Statistics & Inferential Statistics: Distinctions

27. Major Ethical Issues in Behavioral & Health Science Research Considerations, e.g., in the Chang et al. (2002) Study Role of Institutional Review Board (IRB) Informed Consent Confidentiality Latitude To Exit Study without Prejudice Control Group Entitlement If Possible Debriefing

28. Where Do We Go from Here? Concepts, Principles, & Procedures Considered in Greater Detail Analyses of Two Additional Massage Therapy Research Studies

29. Introduction & Overview:Chapter 4/Unit 2� A More Focused View of the True Experimental/RCTResearch Method Van den Dolder, P. A., & Roberts, D. L. (2003) A trial into the effectiveness of soft tissue massage in the treatment of shoulder pain. Australian Journal of Physiotherapy, 49(3), 183-188. Preyde, M. (2000). Effectiveness of massage therapy for subacute low-back pain: A randomized controlled trial. Canadian Medical Association Journal, 162(13), 1815-1820.

30. True Experimental/RCT Method & the Research Category-to-Procedures Continuum Quantitative Research Category Difference-Oriented Research Strategy True Experimental/RCT Research Method Randomized, Control-Group, Posttest-Only Research Design (i.e., Chang et al., (2002) Random Assignment & Manipulated IV As Prerequisite Research Procedures for Possible Cause-and-Effect Conclusions

31. A �Closer Look� at the True Experimental/RCT Research Method Second Example from the Massage Therapy Research Literature: Van den Dolder & Roberts (2003) Abstract (p. 39 in Chapter 4 Handout)

32. A �Closer Look� at Van den Dolder & Roberts (2003) Research Question & Literature Review Population, Randomization, & Sample Variables Hypotheses Measurement & Statistics Research Design External Validity & Internal Validity Major Ethical considerations

33. A Framework for Reading a Research Article A Research Article�s Structure (�Anatomy�), Function (�Physiology�), & Implied Criteria for Evaluation A Research Article�s Sections Implying Structure & Function Preliminary Section: Title Page & Abstract Introductory Section Method Section Results Section Discussion Section Concluding Section: References & Beyond

34. Content of the Article�sPreliminary Section Title page suggesting type of study being reported, major variables investigated, and participants who were the focal point of the research effort Abstract that synthesizes the study�s introduction, method, results, & discussion sections

35. Content of the Article�sIntroductory Section General literature review of available research in the study�s problem area Specific literature review providing greater detail and depth of coverage Purpose statement via an explicit statement of the research question Rationale or justification for the research hypothesis Explicit statement of the research hypothesis

36. Content of the Article�sMethod Section Participants & sampling procedures, with emphasis on inclusion & exclusion criteria Research strategy, method, & design Variables operationalized Instrumentation used, with emphasis on reliability & validity issues

37. Content of the Article�sResults Section As implied, outcomes of the study are presented Reliance on descriptive statistics & inferential statistics Results of analyses conveyed in context of null hypothesis testing, with implications for the confirmation/disconfirmation of the research and alternative hypotheses

38. Content of the Article�sDiscussion Section Reflections on manner in which the study was conducted, inclusive of its limitations & delimitations (boundaries) Elaboration on interpretation of study�s findings begun in the results section Acknowledgment of study�s significance and its relationship to earlier research findings previously cited Theorizing as to why the study�s results are what they are Suggestions of recommended areas of further research as logical sequels to the study

39. Content of the Article�sConcluding Section References via bibliographic citations of works actually cited in the body of the report Possibly appendixes, author notes, and footnotes as needed

40. A Framework for Reading a Research Article (cont.) Implied Criteria for Critiquing a Research Article: Reflections of Structure & Function Critique Questions (n=29) To Ask in Each Section of a Research Article (pp. 48-49 in Chapter 4 Handout)

41. Criteria, or Critique Questions, for the Preliminary Section 1. Does the title of the study provide a basis for identifying the type of study, major variables, & participants? 2. Does the abstract synthesize the main body of the report (i.e., the introduction, method, results, & discussion) with a particular focus on the research question, research hypothesis, participants, research method & design, major variables, instruments, statistical techniques, principal findings, & conclusions?

42. Criteria, or Critique Questions, for the Introductory Section 3. Is the reader introduced to the relevant professional literature bearing on the study being reported by way of a general overview of the research problem area as well as a more specific coverage of individual studies? 4. Is the purpose of the study identified by way of the research question being formulated at an operational level?

43. Criteria, or Critique Questions, for the Introductory Section (cont.) 5. Is a rationale or justification, based on various features of the professional literature, presented as a context or framework for the study�s research hypothesis? 6. Do the authors state the study�s research hypothesis in such a way that the predicted answer to the study�s research question is clear & unambiguous?

44. Criteria, or Critique Questions, for the Method Section 7. Are the study�s participants clearly characterized along with the inclusion & exclusion criteria used for identifying them? 8. Did the researchers justify the number of participants constituting the sample size by way of a power analysis? 9. Was an accessible population of potential participants acknowledged along with an indication of how the sample was derived from such a population, be it through random selection or some other procedure?

45. Criteria, or Critique Questions, for the Method Section (cont.) 10.Did the authors specify the manner in which the participants were assigned to the two or more comparison groups, be it through random assignment or some other means? 11. Was any clarification provided as to how the ethical aspects of the study were governed, particularly in reference to the protection of the participants, the overall integrity of the research, & the earlier approval of the study by an IRB? 12. Was the nature of the research effort adequately characterized in terms of its position in the research continuum (i.e., its position regarding research category, strategy, method, design, & defining procedures)?

46. Criteria, or Critique Questions, for the Method Section (cont.) 13. Were the study�s variables operationalized in a comprehensive fashion so that their manipulation or measurement could be replicated? 14. Did the authors clearly specify the equipment or instruments used in the study for variable manipulation or measurement purposes, along with a documentation of the technical characteristics of such including validity & reliability?

47. Criteria, or Critique Questions, for the Results Section (cont.) 15. Were the data analysis techniques used in the study identified and justified? 16. Were the results of the study communicated by an appeal to descriptive and/or inferential statistics consistent with the nature of the research question as well as the research method & measurement scales used? 17. Were the results of the data analysis related to an appropriate decision regarding the study�s null or statistical hypothesis?

48. Criteria, or Critique Questions, for the Results Section (cont.) 18. Was the decision regarding the null hypothesis acknowledged as a basis for inferring decisions concerning the alternative & research hypotheses? 19. If hypothesis testing was performed, were the analyses augmented with other statistical techniques such as confidence interval estimation or effect size calculations? 20. Were tables & figures appropriately used so as to render the data analyses more comprehensible?

49. Criteria, or Critique Questions, for the Discussion Section (cont.) 21. Did the researchers reflect on the manner in which the study was designed and conducted regarding any limitations and/or delimitations (boundaries)? 22. Did the authors elaborate on the interpretation of the study�s findings beyond the interpretation that was begun in the results section? 23. Did the researchers address the significance of the study and its findings, particularly as they relate to earlier studies in the problem area investigated?

50. Criteria, or Critique Questions, for the Discussion Section (cont.) 24. Did the discussion section address possible intervening variables in the study that might explain why the results obtained were indeed forthcoming? 25. Were recommendations suggested to the reader regarding needed follow-up studies that might replicate, fully or partially, or at least augment, the study?

51. Criteria, or Critique Questions, for the Concluding Section 26. Does the list of references accurately reflect each of the sources cited in the research report, with a consistent bibliographic citation style used? 27. Does the research report contain any appendixes that augment in greater detail information provided earlier in the article?

52. Criteria, or Critique Questions, for the Concluding Section (cont.) 28. Is there any information in the form of author notes providing insight regarding funding support for the study, contact directives for communicating with the authors as a follow up, or collegial assistance in completing the study? 29. Are any footnotes provided that elaborate on one or more aspects of the study that would have been misplaced or distracting if they had been embedded in the main body of the report?

53. An Even �Closer Look� at the True Experimental/RCT Research Method Third Example from the Massage Therapy Research Literature: Preyde (2000) Full-Text Article (pp. 49-55 in Chapter 4 Handout) Detailed Analysis of Article Provided Via 29 Critique Questions (pp. 55-60)

54. Where Do We Go from Here?Other Options in theQuantitative Research Category Difference-Oriented Research Strategy True Experimental or RCT Method Quasi-Experimental Method Single-Case Experimental Method Nonexperimental Comparative Groups Method Association-Oriented Research Strategy Correlational Method Predictive Method Descriptive-Oriented Research Strategy Single-Case Quantitative Analysis Method Survey method Naturalistic/Structured Observational Method Case Report Method

55. Where Do We Go from Here? (cont.) Options in theQualitative Research Category Contextual/Interpretive-Oriented Research Strategy Case Study Method Phenomenological Method Grounded Theory Method Ethnographic Method Example (viz., of Case Study): Anderson, R. (1999). A case study in integrative medicine: Alternative theories & the language of biomedicine. The Journal of Alternative & Complementary Medicine, 5(2), 165-173.

56. Where Do We Go from Here? (cont.) Options in theIntegrative Research Category Synthesis-Oriented Research Strategy Traditional Narrative Review Method Critical Systematic Review Method Meta-Analytic Systematic Review Method Best-Evidence Synthesis Method Qualitative Systematic Review Methods Qualitative Meta-Synthesis method Qualitative Meta-Summary Method Example (viz., of Meta-Analysis): Moyer, C. A., Rounds, J., & Hannum, J. W. (2004). A meta-analysis of massage therapy research. Psychological Bulletin, 130(1), 3-18.

57. Where Do We Go from Here? (cont.) Examples of Most Critical Input Needed from Massage Therapists As Members of a Research Team Determination of principal anatomical structure(s) & physiological function(s) impacted by the client�s presenting signs & symptoms Formulation of a research-based & practice-sensitive treatment protocol appropriate to the client�s presenting condition and desired outcomes of the intervention Ensuring that the treatment protocol is specific & operational enough to allow replication (full or partial) in future studies and/or practice applications Translating or �transitioning� massage research findings into practice-relevant applications for the benefit of clients (i.e., translational research)

58. Follow-Ups to This Session Contact at hymel@loyno.edu, http://loyno.edu/~hymel/, or (504) 865-3257 Participation in Massage Research Education (MRE) LISTSERV via hymel@loyno.edu Reflections on Needed Curricular Content and Instructional Delivery Methods To Advance MRE Resources of the Massage Therapy Foundation: Massage Research Database at www.massagetherapyfoundation.org/researchdb.html Student & Practitioner Case Report Contests at www.massagetherapyfoundation.org/education.html Forthcoming (Summer 2008) International Journal of Therapeutic Massage & Bodywork: Research, Education, & Practice Resources of the National Center for Complementary & Alternative Medicine (NCCAM) at http://nccam.nih.gov/

59. A Parting Thought . . . .