1 / 42

Educational Research: Getting Started

Educational Research: Getting Started. Steven B. Goldin Chief of Hepatobiliary and Pancreatic Surgery Associate Professor of Surgery Vice Chair of Surgical Education Chief of Hepatobiliary and Pancreatic Surgery. Making Life Better. through Research, Education & Healthcare. TM.

Download Presentation

Educational Research: Getting Started

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.


Presentation Transcript

  1. Educational Research: Getting Started Steven B. Goldin Chief of Hepatobiliary and Pancreatic SurgeryAssociate Professor of SurgeryVice Chair of Surgical Education Chief of Hepatobiliary and Pancreatic Surgery Making Life Better through Research, Education & Healthcare TM

  2. NO DISCLOSURES EXCEPT: Yes, they really are my photos

  3. Outline 1. Research: Why do it? 2. What is educational research 3. Research: How to get started 4. Features of rigorous studies 5. Pitfalls – validity and reliability • Surveys • Ethics, funding, resources

  4. Research – why do it? ASE – “No discipline can call itself a profession that does not give top priority to research. More specifically, the status of a given profession depends on the adequacy of the research activities, and the application of research findings to routine professional activities.” (Mouly 1978)

  5. Research- why do it? Faculty advancement Teaching and research separate academic surgeons from those in community practice Develop a national prominence/reputation May be a funding source

  6. Introduction Importance of clinical productivity – how do you find the time to do a research project? Do research with your daily activities Data collection is common for all courses and rotations. medical students residents faculty Innovation – what makes an academic center academic. What additional data would need to be collected to begin a research project?

  7. Introduction Innovation and evaluation form the backbone of research ideas and projects Many research projects are done with students, residents, and faculty Steady supply of potential participants

  8. Introduction Traditionally empirical research (bench top) was viewed as the most valid method. Other forms of research (historical, phenomenological, introspective, qualitative, etc.) contribute to “educational research” Empirical research is characterized by direct observation, objectivity, falsification, replicability, parsimony and quantification (scientific method) Parsimony is defined as the adoption of the simplest assumption in the formulation of a theory or in the interpretation of data, especially in accordance with the rule of Ockham's razor

  9. Introduction Occam’s razor (Ockham’s razor) - “entities should not be multiplied unnecessarily.” The explanation of any phenomenon should make as few assumptions as possible, eliminating those that make no difference in the observable predictions of the explanatory hypothesis or theory When competing hypotheses are equal in other respects, select the hypothesis that introduces the fewest assumptions and postulates the fewest entities while sufficiently answering the question

  10. Evidence Based Medicine Evidence based medicine – interventions should be guided by data from rigorous studies was described at least 15 years ago Evidence-Based Medicine (EBM) – excellent framework for evaluating the quality of clinical research EBM is not focused on educational research, but provides useful criteria for different categories of research. (Treatment efficacy, accuracy of diagnostic tests, clinical guidelines, prognosis and risk, overviews and meta-analysis)

  11. Evidence Based Medicine Many EBM concepts are applicable to educational research and should be familiar to all educational researchers Medical educators, likewise, should teach based on evidence

  12. Introduction Educational research is driven by the process of documenting the steps of a project This process is the same for all studies including bench research, studying teaching methods, or student attitudes Educational research focuses on describing and documenting the transmission of knowledge, competency, and understanding about which teaching methods are best and can lead to better practice

  13. Making it Scholarly • All research involves – asking a question and obtaining an answer in a systematic, thoughtful way • Scholarly teaching involves routinely asking questions about daily educational activities • Research is answering these questions and scholarship is the dissemination of the research results

  14. Types of Studies Descriptive: study describes what is going on or what exists Relational: study looks at the relationships between two or more variables Causal: study is designed to determine whether one or more variables causes or affects one or more outcome variable

  15. Research Steps • Ask the question – state clear goals (hypothesis and alternate hypothesis) • Prepare (literature review, discussions, conferences) • Specify the techniques or interventions • Test methods and ways of measuring impact • Pilot test the process and refine process • Select the best method to answer the question

  16. Research Steps • Do the study • Capture and analyze data • Interpret findings and determine significance of the outcome • Negative results can be meaningful depending on the question • Reflective critique (thoughtful discussion, think about the implications of the findings and develop the next steps) • Disseminate the findings (article, abstract, presentation, workshop)

  17. Features of Rigorous Studies Large numbers of student participants Multi-institutional enrollment of participants Having a control or appropriate comparison group Measuring an objective outcome Measuring some outcome at least 1 month after the intervention Conduct the intervention more than once Estimate statistical power

  18. Features of Rigorous Studies Measure objective outcomes including any evaluation other than self-assessment by students including: Tests of knowledge Course grades Objective structured clinical examinations or observations of standardized patient Interaction assessment of performance with a real patient such as graded observation of patient interactions or patient feedback Objective clinical outcomes such as blood pressure control Performance with mannequins, computer simulations, or laboratory animals Psychological inventories

  19. Foundation Language Types of questions (descriptive, relational, causal) Time in research (cross-sectional vs. longitudinal with repeated measures or time series) Types of relationships (correlational vs. causal) Patterns of relationships (none, positive, negative) Variables and attributes (independent and dependent, exhaustive, mutually exclusive) Hypothesis and null hypothesis Types of data (quantitative vs. qualitative)

  20. The Hypothesis Theoretically significant research questions are different from other types of questions. A research question is not a question: of fact. (How many residents passed the in-service or their qualifying examinations?) about a specific sample of subjects. (Do my students perform better with open-ended rather than multiple choice questions?) about things that can not be measured. (Do lucky students have more success than unlucky students?)

  21. A good research question should: 1. seek new knowledge 2. use the scientific method 3. build upon existing scientific knowledge 4. pick up where existing theories and data end 5. lead to specific hypotheses which can be tested 6. have an answer deducible from the results and provide information applicable not only to the subjects studied, but to a broad, definable population of subjects and settings

  22. Reliability and Validity Education research quality is linked to reliability and validity, which are the two major criteria for the quality of the measurement Reliability involves the quality of measurement and the consistency or repeatability of the measures Theories of validity involve each step which are interdependent

  23. Reliability and Validity Studies may be reliable but not valid valid but not reliable neither reliable nor valid both reliable and valid Consulting with a research design and statistics expert to avoid issues that may impact study reliability or validity

  24. General Surgeon Shortage

  25. Surveys - References Sage Publications, Inc; 2nd edition (November 12, 2002), 10 volumes, 1434 pages Gall, Borg, and Gall. Educational Research: An Introduction (8th edition). Allyn and Bacon. 2006

  26. Surveys One of the most important areas of measurement in applied social research. Two types Questionnaires Interviews Construction of the Survey Types of questions Decisions about question content Decisions about question wording Decisions about response format Question placement and sequence

  27. Surveys Often done because users think they are simple and quick ways to publications Survey design has fallen into disrepute among editors, reviewers and consumers of medical research because they have been: poorly designed poorly developed poorly administered poorly analyzed Reinforces the reconceived notion that surveys are not worth doing Like experimental research, good survey research is very systematic and rigorous

  28. Surveys The purpose of the study must be supported by a review of the literature and understanding of the problem. Then develop the research question, which will guide the development of the content. If an item does not help you answer a research question, do not include it on the survey What information do you want from respondents? attitudes/beliefs? knowledge? behavior? attributes (demographic characteristics)?

  29. Surveys Structure the questions carefully Unless the sole purpose of the survey is to solicit suggestions, probe memories, clarify positions and or vent frustrations, limit the number of open-ended questions Open ended questions make it difficult to construct meaningful variables for statistical analysis, are difficult to collate, and difficult to decipher

  30. Surveys A questionnaire should: 1. look attractive/appealing to complete 2. have questions laid out and be organized in such a manner as to make them as easy to complete as possible 3. be organized with items appearing in some logical sequence 4. begin with those questions that the respondent is likely to see as useful to the purpose of the study, or with those that are easy to answer and non-threatening 5. place important items that are key to the study near the beginning Items that are interesting and clearly relevant to the study will increase response rate. Length has a small effect on response rate. Pilot the instrument

  31. Ethics Principle of voluntary participation Informed consent Risk of harm – physical or psychological Confidentiality Principle of anonymity Person’s right to service (no treatment control group) Institutional Review Board (IRB) Students and residents are considered a vulnerable population Social and Behavioral Institutional IRB vs. Biomedical IRB

  32. Funding Most surgical education research is conducted without the benefit of research grants ASE Foundation – CESERT program (www.surgicaleducation.com) National Board of Medical Examiners Emergency Medicine Foundation Agency for Health Care Research and Quality Sages American College of Surgeons (www.facs.org) lists funding agencies that may be willing to fund educational research

  33. Resources ASE Educational Research Committee Institutional Department of Medical Education Departments of psychology, education, sociology, and public health Biostatistician ASE – Surgical Education Research Fellowship Program (SERF)

  34. Conclusions Clinical demands are taking on larger levels of importance Protected time is a luxury Research is required to advance academically Educational research is a viable option even when faced with high clinical demands

  35. Conclusions Many educational research projects have significant limitations Insufficient sample sizes Lack of generalizability when interventions are assessed at only 1 institution or are conducted only once Lack of appropriate control groups Use of subjective and non-validated instruments to assess outcomes Assessing only short-term outcomes

  36. Conclusions Educational research challenges often relate to measurement issues, bias and individual differences An understanding of key educational concepts in psychology, sociology, anthropology, psychometrics, sociometrics, and statistics. Collaboration is key! Assuming a significant question has been clearly articulated, the quality of the study will depend upon the research design, data analysis, interpretation and results reporting

  37. References Research Methods Knowledge Base. William M.K. Trochim. http://www.socialresearchmethods.net. Free on the internet Surgical Educators’ Handbook. Curriculum Committee and Members of the Association for Surgical Education. ASE website. http://www.surgicaleducation.com/mc/page.do

  38. Educational Research: Getting Started Steven B. Goldin Chief of Hepatobiliary and Pancreatic SurgeryAssociate Professor of SurgeryVice Chair of Surgical Education Chief of Hepatobiliary and Pancreatic Surgery Making Life Better through Research, Education & Healthcare TM

More Related