IE 656B Modeling Healthcare Systems web.ics.purdue/~duffy/index656B

1. IE 656B Modeling Healthcare Systemshttp://web.ics.purdue.edu/~duffy/index656B Vincent G. Duffy, Ph.D. Associate Professor Regenstrief Center for Healthcare Engineering School of Industrial Engineering and Department of Agricultural & Biological Engineering Purdue University Monday, January 28, 2008 Evaluating Research – List of 10 Ways…

2. 10 ways to look at a research paper systematically for critical analysis • 1. research ideas/question -what is the purpose? • 2. what background support? - literature review • 3. theoretical basis for analyzing question/hypothesis? • 4. applicability-practical contribution? • 5. theoretical contribution? • 6. appropriate methodologies for carrying out study? (determining variables, data collection, method used to test hypotheses, validity of measures and reliability) • 7. appropriate statistical analyses and assumptions? • 8. presentation of results: what do they really mean? • 9. conclusions drawn: are they reasonable? • 10. future work/research directions: any possibilities?

3. 10 ways to look at a research paper systematically for critical analysis • 1. research ideas/question -what is the purpose? • Rather than assessment by oral exam (1st paragraph), research becomes a possible solution for the problem (behavior of team can not be assessed crisis). • Test effectiveness of training method. • Continuation. • Previously – recall of incidents. Prefer standard and measure. • Develop ratings. Validate that methodology (videotape) can be used. • Alternate methods for considering Inter-rater reliability considered

4. 10 ways to look at a research paper systematically for critical analysis • 2. what background support? - literature review • Important because educational/clinical experience/performance in critical incident – efficiency can affect overall health of patient. • (what are literature/refs. to support ‘why objective is important’) • p.9 measure two …. (ref. 18) • Difficult w/out simulating – need evaluation tools (p.9 ref. 10, 18-20) • Also similar in aviation. (see also refs. 12-17) • Other literature support could be to justify the methodologies used, or the theoretical foundations.

5. 10 ways to look at a research paper systematically for critical analysis • 3. theoretical basis for analyzing question/hypothesis? • That is the difference between lit review & theoretical foundation? • What are the hypotheses? • No definite technique to measure…“However, didn’t find why we need assessment…” • (based on p.16) Rationale may be related to application or may contribute to crisis management research. • What do you mean…theoretical foundations…? • Also similar in aviation & use of simulations – for video tapes – theoretical support for methodologies used. • Support may be embedded in the statement (col. 1 p.9) “…these crisis management behaviors in a ‘paradigm’ (model) we call ACRM (ref. 18-20).

6. 10 ways to look at a research paper systematically for critical analysis • 4. applicability-practical contribution? • Clearly this does fit as a practical contribution/application for the research. • “These performance assessment tools might be useful for educational research or for tracking a residents progress…” • “However, didn’t find why we need assessment…” • (based on p.16) Rationale may be related to application or may contribute to crisis management research. • ‘…may contribute’ is not yet a practical contribution (has potential to contribute) • As we read…let’s focus on ‘evidence’ that suggests the method or finding ‘adopted’ by someone or org. • What if authors adopted? • Partially ok… • Trouble…kind of like referring to own ‘paradigm’. • (good?) maybe….more information is needed… • Better when the theory or application can be externally tested or validated.

7. 10 ways to look at a research paper systematically for critical analysis • 5. theoretical contribution? • Maybe… • These performance assessment tools might be useful for educational research or for tracking a residents progress… • Depends on the nature of the contribution/how useful to ‘educational research’. • Can’t tell from abstract alone.. • Can look at ….conclusions/discussion to get further insight. • “However, didn’t find why we need assessment…” • (based on p.16) Rationale may be related to application or may contribute to crisis management research.

8. 10 ways to look at a research paper systematically for critical analysis • 6. appropriate methodologies for carrying out study? (determining variables, data collection, method used to test hypotheses, validity of measures and reliability) • p.11 variability between ratings can not be considered a true measure of the true interrater reliability of the rating scales. .. • Yes. Based on theoretical support for methodologies presented on p.9. • p.13 (top of discussion) Justifying method/videotape… for analysis…

9. 10 ways to look at a research paper systematically for critical analysis • 7. appropriate statistical analyses and assumptions? • Are these analyses appropriate (mapped well) to answer the questions/objective/hypotheses based on the data that is collected (for the variables that are identified/measured)? • ANOVA to check rate of variability as well as performance of crew on overall basis… • p.16 (last paragraph, left – simulators can be used to predict performance in crisis) • Simulators are good solution/later no. • Raises question about hypotheses – what were they, and are they shown to be supported by the data collected? • Not so directly stated? Authors should state hypotheses explicitly (related to 3.). • In results, they should tell whether the hypotheses were supported by considering the variables (that we can identify within those hypotheses) for which data was collected.

10. 10 ways to look at a research paper systematically for critical analysis • 8. presentation of results: what do they really mean? • Results/interpretation: ‘technical scoring system - reliability good. • Most teams technically good at handling two crisis scenarios, but interaction wasn’t up to the mark. • Level of agreement between raters was high.

11. 10 ways to look at a research paper systematically for critical analysis • 9. conclusions drawn: are they reasonable? • Although technical performance can be considered, perfect rating scale difficult to achieve – limits of the study. • Behavioral rating system needed some improvement. • Study not able to determine whether performance was related to experience. • Rating system needed more refinement, before determining competency/board certification.

12. 10 ways to look at a research paper systematically for critical analysis • 10. future work/research directions: any possibilities? • Rating system needed more refinement, before determining competency/board certification. • Other patient safety related initiatives in research…including predictive modeling. • Now that we’ve assessed methods of evaluation of training, can we identify models/predictive capability for determining which variables/and by how much in ‘training’ can improve/impact patient safety – where should additional resources be spent ? • Maybe more justification for the simulations is needed… • (if simulations improving learning was not yet supported)

13. 10 ways to look at a research paper systematically for critical analysis • 1. research ideas/question -what is the purpose? • 2. what background support? - literature review • 3. theoretical basis for analyzing question/hypothesis? • 4. applicability-practical contribution? • 5. theoretical contribution? • 6. appropriate methodologies for carrying out study? (determining variables, data collection, method used to test hypotheses, validity of measures and reliability) • 7. appropriate statistical analyses and assumptions? • 8. presentation of results: what do they really mean? • 9. conclusions drawn: are they reasonable? • 10. future work/research directions: any possibilities?

14. RCHE presentation-Fall ‘06 • http://video.e-enterprise.purdue.edu/regenstrief/240.wmv • Dr. Vin Sahney, Institute of Medicine & NAE • For more information see: http://weatherhead.case.edu/executive-education/about/instructors/vinod-sahney.cfm

15. For next week: • Week 5: February 4 • 9. (P2) Flynn, E.A., Barker, K.N., Gibson, J.T., Pearson, R.E., Smith, L.A., Berger, B.A., 1996. Relationships between ambient sounds and the accuracy of pharmacists’ prescription-filling performance, Human Factors, 38 (4) 614-622. (to be reviewed as a group next week - turn in initial one page summaries in class). • 10. (Book chapter) (P3 & P4) Groopman, J., 2007. How doctors think, Houghton-Mifflin Co.: Boston. Read 2 chapters- determined in class. • 11. (Book chapter) (P5) Kohn, L.T., Corrigan, J.M., Donaldson, M.S. (Eds.) 2000. To Err is Human: Building a Safer Health System, Ch.2 pp. 26-48 (to be reviewed together in class). • 12. Video/plenary talk by Don Berwick of IHI. • 13. Distribute semester project description

16. How doctors think • By Jerome Groopman, Houghton Mifflin, 2007 – from NYTimes Bestseller list • Consider ch.1-9 + epilogue & Introduction

17. re: semester project planning • Proposal writing workshop for Graduate Students; Jan.30, 7pm Room 214 ABCD, Stewart Center • Online registration at: • http://www.gradschool.purdue.edu/development