A Patient Care Digital Library Personalized Search and Summarization over Multimedia Information

1. A Patient Care Digital Library Personalized Search and Summarization over Multimedia Information Kathleen McKeown, Shih-Fu Chang, James Cimino, Paul Clayton, Judith Klavans Columbia University

2. Vision • Personalized access to distributed resources • information vetting • information fusion • information understanding • Provision of patient-specific information • at the point of patient care • in terms that can be understood • Benefits: awareness of new procedures, increased compliance

3. Features • Tailor search, presentation, and summarization of online medical literature • Exploit online patient record as user model • Patients and health care providers • Multimodal access and presentation over multimedia resources

4. Scope of Research “exciting,” “innovative,”“broad,” “ambitious,” “complex” • Computer Science and Medical Informatics • Links into libraries: CRIA • Suite of existing component technologies • Five year vision • Active partners

5. Continuum • Current: existing individual components for multiple applications • summarization of terrorist news, image search, categorization of images, multimedia briefings, PATCIS, patient mammography record extraction, video summarization of news, query translation • Future: DLI2 enables • unification and transformation of components • new level of capability • integrated in a single application

6. Outline • Desmond Jordan • Why this research is needed • Kathy McKeown • An example—integration of components • Scope, organization, management • Role of partners and new developments • Jim Cimino • Clinical information infrastructure • Kathy McKeown • Budget

7. Rounds • Patient-centric • Current: Access to clinical data • Missing: Access to literature that fits patient profile

8. Data follows the patient—why not literature?

9. Ongoing Collaboration: Generation of Multimedia Briefings Voice: . . . . Ms. Jones is an 80 year old, diabetic, hypertensive, female patient of Dr. Smith undergoing CABG. . . .

10. User Study Group • Speech, text, and graphics output were useful

11. Steve Feiner: If I recall correctly, I found the image at the bottom in a we page for patients with CHF. May have something to do with fluid buildup. Multimedia Video • Physicians: residents learn by comparison • Patients: • Video helps them understand their diagnosis • They like to investigate prognosis

13. Why should I prescribe amiodarone? Query Steve Feiner: This slide differs from the previous one in that it has a real graphical history, although not in the medical domain. It may be better, since it means that all of the material on the slide will be real. BTW, this is a “comment”. You can either delete it or make it invisible with the “View” menu “comments” command Lexical Navigation WebSeek History

14. Key Issues: User Interface • Query augmentation using the patient record • Congestive Heart Failure (CHF), male, Base Heart Rate (BHR) > 90, history of ischemia • Lucent: speech, IBM: query refinement, interface for image/video search

15. Distributed Search b Antiarrhythmic Drugs Documents NSF is planning to implement a new electronic project reporting system through which proposers will be able to produce the Results from Prior Support section of the proposal. This new system will be part of the NSF FastLane system and will permit updating of reports on each of the PI's NSF projects and will avoid re-en the normal mechanism for project reporting to NS Amiodarone Amiodarone profile profile profile Amiodarone Treatme CHF NSF is planning to implement a new electronic project reporting system through which proposers will be able to produce the Results from Prior Support section of the proposal. This new system will be part of the NSF FastLane system and will permit updating of reports on each of the PI's NSF projects and will avoid re-entry of information previously provided by the PI, either with the original proposal submission or in updates provided usingting process and become the normal mechanism for project reporting to NS Augmented query Search Echocardiogram Amiodarone CHF male BHR > 90 Echo 10/15/98 Echo 9/15/98 Patient heart sounds Echo 8/15/98 Echo 7/15/98 Databaseof Videos

16. Amiodarone CHF male BHR > 90 Journal Articles NSF is planning to implement a new electronic project reporting system through which proposers will be able to produce the Results from Prior Support section of the proposal. This new system will be part of the NSF FastLane system and will permit updating of reports on each of the PI's NSF projects and will avoid re-entry of information previously provided by the PI, either with the original proposal submission or in updates provided usingting process and become the normal mechanism for project reporting to NS Methods: All patients had advanced CHF with ... Prognosis Article Documents NSF is planning to implement a new electronic project reporting system through which proposers will be able to produce the Results from Prior Support section of the proposal. This new system will be part of the NSF FastLane system and will permit updating of reports on each of the PI's NSF projects and will avoid re-entry of information previously provided by the PI, either with the original proposal submission or in updates provided usingting process and become the normal mechanism for project reporting to NS Prognosis Article Amiodarone Amiodarone Categorization & Filtering Amiodarone Amiodarone NSF is planning to implement a new electronic project reporting system through which proposers will be able to produce the Results from Prior Support section of the proposal. This new system will be part of the NSF FastLane system and will permit updating of reports on each of the PI's NSF projects and will avoid re-entry of information previously provided by the PI, either with the original proposal submission or in updates provided usingting process and become the normal mechanism for project reporting to NS NSF is planning to implement a new electronic project reporting system through which proposers will be able to produce the Results from Prior Support section of the proposal. This new system will be part of the NSF FastLane system and will permit updating of reports on each of the PI's NSF projects and will avoid re-entry of information previously provided by the PI, either with the original proposal submission or in updates provided usingting process and become the normal mechanism for project reporting to NS Diagnosis Article . . .

17. Target Summaries Prognosis articles: • amiodarone patient, BHR > 90 A significant reduction in sudden death was observed in patients with a BHR > 90 beats/min treated with amiodarone • pacemaker patient, BHR  90 In patients with a BHR  90 beats/min, amiodarone-treated patients had a mortality rate similar to those not treated with amiodarone. One article found that pacing fails to improve hemodynamic function, while another showed a significantly better quality of life.

19. PERSIVAL Management Structure Technology Search Gravano User Interface Johnson Summarization McKeown Project Leader McKeown Clinical Infrastructure Cimino Evaluation Patel Industry Partners Klavans Content and Library Team Klavans

20. PERSIVAL Management Structure Technology Search Gravano User Interface Johnson Summarization McKeown Project Leader McKeown K. McKeown Clinical Infrastructure Cimino Evaluation Patel Industry Partners Klavans Content and Library Team Klavans

21. PERSIVAL Management Structure Technology Search Gravano User Interface Johnson Summarization McKeown Project Leader McKeown J. Klavans Clinical Infrastructure Cimino Evaluation Patel Industry Partners Klavans Content and Library Team Klavans

22. PERSIVAL Management Structure Technology Search Gravano L. Gravano User Interface Johnson Summarization McKeown S. Johnson Project Leader McKeown Clinical Infrastructure Cimino J. Cimino Evaluation Patel Industry Partners S.-F. Chang Klavans Content and Library Team Klavans

23. PERSIVAL Management Structure Technology Search Gravano S. Johnson User Interface Johnson S. Feiner Summarization McKeown Project Leader McKeown Clinical Infrastructure Cimino S.-F. Chang Evaluation Patel Industry Partners Klavans Content and Library Team Klavans

24. PERSIVAL Management Structure Technology Search Gravano User Interface Johnson K. McKeown Summarization McKeown Project Leader McKeown Clinical Infrastructure Cimino J. Klavans Evaluation Patel Industry Partners Klavans Content and Library Team Klavans V. Hatzivassiloglou

25. PERSIVAL Management Structure Technology Search Gravano J. Cimino User Interface Johnson Summarization McKeown D. Jordan Project Leader McKeown Clinical Infrastructure Cimino C. Friedman Evaluation Patel Industry Partners Klavans Content and Library Team Klavans S. Johnson

26. PERSIVAL Management Structure Technology Search Gravano User Interface Johnson V. Patel Summarization McKeown Project Leader McKeown Clinical Infrastructure Cimino J. Cimino Evaluation Patel Industry Partners Klavans Content and Library Team Klavans D. Jordan

27. PERSIVAL Management Structure Technology Search Gravano J. Klavans User Interface Johnson Summarization McKeown S. Jacobson Project Leader McKeown Clinical Infrastructure Cimino P. Molholt Evaluation Patel Industry Partners Klavans E. La Rue Content and Library Team Klavans

28. Role of Technology Partners • Lucent: Spoken Language Dialogue Systems (speaker independent) • IBM: Query refinement, evaluation, text summarization, interface for image/video search • GE: Concept based retrieval

29. New Partners and Changes • Siemens Corporate Research: grant for video summarization integrating text/image, plus researcher time • $68,680/year • AT&T: grant to build our networking infrastructure • $50,000 • Welch Allyn: information and equipment, Medical Director’s time, cash • $35,000 • GE Medical Systems: commitment to commercialization of results

30. Steve Feiner: IBM is more than just a software company; ditto Lucent, GE, etc. Why not just group companies together? Partners

31. Partners

32. Clinical Information Systems • Clinical repository of coded and text data • Medical Entities Dictionary server • Web-based access for clinicians (WebCIS) • Web-based access for patient (PatCIS) • Infrastructure handles security, application tracking, and evaluation • Architecture handles integration of components

33. Input for PERSIVAL • Coded data • Natural language processing • Medical Entities Dictionary • UMLS

38. Web/PatCIS and Digital Library Information Resource Infobutton WebCIS/ PatCIS CIS MED

39. Web/PatCIS and Digital Library Information Resource Infobutton WebCIS/ PatCIS Query Interface PERSIVAL CIS MED

40. Users • Clinical users • Columbia-Presbyterian Medical Center • New York Hospital • 37 affiliates • Patients • Heart failure • Home diabetes telemonitoring • High risk for heart attack • Cancer patients

41. Evaluation and User-Centered Design • Usability engineering • in situ and lab video recordings • Evaluation of system components: cognitive and quantitative • Subjective assessment of search plus recall, precision • Survivability • end-to-end evaluation, reduction of manual intervention, training, university support of applications

42. Summary • Tailor search, presentation, and summarization of online medical literature • Exploit online patient record as user model • Patients and health care providers • Multimodal access and presentation over multimedia resources