Developing medical software pitfalls and prophylactics
1 / 56

Developing Medical Software: Pitfalls and Prophylactics - PowerPoint PPT Presentation

  • Uploaded on

Developing Medical Software: Pitfalls and Prophylactics. Elliot Jaffe Seminar in Computer Assisted-Surgery, Medical Robots and Medical Imaging Fall 2002. Outline. Why should you be worried? Case Study: Therac-25 US Government Guidelines. What? Me worry?.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Developing Medical Software: Pitfalls and Prophylactics' - tevin

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Developing medical software pitfalls and prophylactics

Developing Medical Software: Pitfalls and Prophylactics

Elliot Jaffe

Seminar in Computer Assisted-Surgery, Medical Robots and Medical Imaging

Fall 2002


  • Why should you be worried?

  • Case Study: Therac-25

  • US Government Guidelines

What me worry
What? Me worry?

  • Software is used in medical devices

    • Monitoring

    • Planning

    • Surgery

    • Visualization

  • Software fails

Case study therac 25
Case Study: Therac-25

  • 1983 – 1987

    • AECL: Atomic energy of Canada Ltd.

    • 6 reported “accidents”

    • Changed the way software is developed and verified as part of a medical device

Medical linear accelerators
Medical Linear Accelerators

Linac: North Oakland Medical Center

Therac 25 genesis
Therac-25 genesis

  • Therac-6: 6MeV X-ray accelerator

  • Therac-20: 20MeV Dual Mode (Electron/X-ray) accelerator

  • Upgraded with Dec PDP-11 minicomputer for ease of use

  • Could be operated without computer

Therac 25

  • Dual Mode 25MeV accelerator

    • Electron/X-Ray

    • Can be operated ONLY through the computer

  • Computer controls and monitors system

    • Some hardware safety mechanisms and interlocks were replaced with software

  • First working prototype: 1976

  • First commercial product: 1982

Treatment goals
Treatment Goals

  • Deliver high energy radiation for the treatment of cancer

  • Radiation needs to be focused and controlled

  • Multiple energy levels

    • X-Ray

    • Electron

Therac 25 operation
Therac-25 Operation

  • Turntable to select from three modes

    • Visual

    • Electron

    • X-Ray

  • Turntable is moved mechanically

  • Software monitors position of turntable

Operator interface
Operator Interface

Cursor should be here during operation

Therac 25 error states
Therac-25: Error States

  • Treatment Suspend

    • Requires complete machine restart

  • Treatment Pause

    • Operator types “P” to proceed

Therac 25 error messages
Therac-25: Error Messages

  • HTILT, VTILT, etc.


    • 1 <= n <= 64

  • No documentation

  • No indication of severity

  • Occurred on average 40 times a day!

Therac 25 event 1
Therac-25: Event #1

  • June 1985: 10MeV electron treatment

  • Patient reported: “tremendous force of heat … this red-hot sensation”

  • Technician replied that it was impossible

  • AECL claimed it was impossible

  • Never reported to FDA

Therac 25 event 11
Therac-25: Event #1

  • Patient received severe radiation burn

  • Patient’s breast was removed

  • Shoulder and arm was paralyzed

  • AECL refused to believe that it was caused by Therac-25

  • Lawsuit settled out-of-court

Therac 25 event 2
Therac-25: Event #2

  • July 26, 1985

  • HTILT message, Treatment Pause

  • Operators resumed treatment

  • Repeated 5 times until machine stopped

  • Patient reported “electric tingling shock”

Therac 25 event 21
Therac-25: Event #2

  • Patient died of cancer

  • Autopsy revealed that a total-hip replacement would have been required due to radiation exposure

  • Reported to AECL, FDA

  • AECL believed it to be a hardware problem

Therac 25 event 22
Therac-25: Event #2

  • AECL could not reproduce the reported behavior

  • AECL modified turntable

    • “Fixed” potential error in 3-bit turntable location identifier

Therac 25 event 23
Therac-25: Event #2

  • AECL claimed:

    • “analysis of the hazard rate of the new solution indicates an improvement over the old system by at least 5 orders of magnitude”

Therac 25 event 3
Therac-25: Event #3

  • December 1985

    • After upgrade from event #2

  • Patient developed parallel striped pattern in treatment area

  • AECL reported: “Could not have been produced by any malfunction of the Therac-25 or by any operator error.”

  • Not reported to FDA

  • Patient required surgery to repair tissue damage

Therac 25 4
Therac-25: #4

  • March 21, 1986

  • Operator entered “x” instead of “e”

  • Moved cursor and corrected error

  • Began treatment


  • Continued Treatment


  • Machine shutdown

Therac 25 event 4
Therac-25: Event #4

  • Patient monitors: video and audio were broken

  • Patient received electric shock, started to get up and was then shocked in the arm

  • Patient pounded on treatment door

  • Patient sent home

  • Machine checked out ok

Therac 25 event 41
Therac-25: Event #4

  • Patient died of overdose 5 month later

  • AECL suggested an electrical problem in the area

  • Independent engineering firm checked and found no problem

Therac 25 event 5
Therac-25: Event #5

  • April 11, 1986

  • Same operator

  • Same editing


  • Audio monitor (now working) reported a loud sound from machine

  • Patient died May 1, 1986 (three weeks later) of acute high-dose radiation to his brain

Therac 25 event 51
Therac-25: Event #5

  • Physicist took machine out of service

  • Reported to AECL

  • Operator and Physicist were able to reproduce the failure

  • AECL still could not reproduce the failure

  • FDA declares system “defective”

Therac 25 event 5 cause
Therac-25: Event #5 - cause

  • Operating system was a hand-coded real-time system developed by one programmer in the 1970’s.

  • Problem was traced to race condition in the main loop

  • Result was that x-ray beam could be used through the electron magnet

Therac 25 event 6
Therac-25: Event #6

  • January 17, 1987

  • Operator set turntable to field light position

  • Gave command to system to “set” turntable to x-ray

  • Ran treatment

  • System reported “no dose or dose rate”

  • Re-ran treatment

  • Patient died in April, 1987 of problems related to overdose

  • AECL and FDA notified

Therac 25 event 6 cause
Therac-25: Event #6 - cause

  • Software bug

  • Register overflow

    • 8 bit register used for multiple purposes

    • Once or twice in each setup phase, the register overflows, allowing the system to think that the turntable was reset

Lessons learned
Lessons Learned

  • Studies reported 12 lessons learned

  • We will cover five of them

Overconfidence in software
Overconfidence in Software

  • First safety analysis did not include software, even though it was responsible for safety of the system

  • When problems did occur, it was assumed to be a hardware failure

Reliability vs safety
Reliability vs. Safety

  • Therac-25 ran for three years in production without a problem

  • Tens of Thousands of patients were treated before the first known overdose

  • Reliability leads to complacency

  • Reliability != Safety

Lack of defensive design
Lack of Defensive Design

  • Software was designed for small memory footprint

  • Self Checks, Error Detection, Error handling and Auditing was left out

Unrealistic risk assessment
Unrealistic risk assessment

  • First Risk Assessment did not include software

  • AECL claimed 5 orders of magnitude improvement from changing one microswitch

  • Software is harder to assess for failures than hardware

Inadequate software engineering practices
Inadequate Software Engineering Practices

  • Software specification was after-the-fact

  • Dangerous design/coding practices could have been avoided

  • Audit trails should be built into the production software

  • Software should be tested at the unit, module and system level

  • Regression testing on all changes

  • GUI should be designed, not implemented

Software reuse
Software Reuse

  • Therac-25 used software from T-20

  • Reliability != Safety

  • Assumptions and Preconditions may have changed

  • Sometimes its better to rewrite from scratch

Us government guidelines
US Government Guidelines

  • Significantly reduce the risk of death or injury

  • Impose standards and best practices to raise the overall level of the industry

  • Define minimum requirements for

    • New products

    • Derivative products

Level of concern
Level of Concern

  • Major: device directly affects the patient or operator and failure could result in death or serious injury

  • Moderate: device directly affects the patient and failure could result in non-serious injury

  • Minor: failures will not result in injury

Levels of concern
Levels of Concern

  • Does the software

    • Control life support device?

    • Control delivery of harmful energy?

    • Control treatment delivery?

    • Provide diagnosis as basis for treatment?

    • Monitor vital signs?

  • If no to all these questions, then concern is minor

Requirements for minor concern
Requirements for minor concern

  • Software Description

  • Device Hazard analysis

  • Software functional Requirements Specification

  • Architecture Design chart

  • Validation, Verification and Testing

  • Release Version Number

Requirements for moderate major concern
Requirements for Moderate/Major concern

  • Full Software Requirements Spec.

  • Design Specification

  • Traceability analysis

  • Development lifecycle documentation

    • Configuration management

    • Maintenance activities

  • Revision Level History

  • Unresolved Anomalies (bugs)

Software requirements spec
Software Requirements Spec

  • Hardware requirements

  • Programming languages

  • Interface requirements

  • Software functional requirements

  • Software performance requirements

Software requirements spec1
Software Requirements Spec

  • Algorithms for therapy, diagnosis, monitoring, alarms, analysis, interpretation (with supporting clinical data)

  • Device limitation due to software

  • Internal software tests and checks

  • Error and interrupt handling

Software requirements spec2
Software Requirements Spec

  • Fault detection, tolerance and recovery characteristics

  • Safety requirements

  • Timing and memory requirements

  • Use of off-the-shelf software

Risk hazard analysis tools
Risk/Hazard Analysis Tools

  • Fault Tree Analysis (FTA)

    • Used in initial design phase

  • Failure Modes Effect and Criticality Analysis (FMECA)

    • Used in design and development phase

  • Failure Reporting and Corrective Action System (FRACAS)

    • Used during product lifecycle

Fault tree analysis
Fault Tree Analysis

  • Identify a failure or safety hazard, then attempt to identify all possible ways to create that hazard

  • Answers the question:

    • How can event X occur?

  • Used in Military and Nuclear Industry since the 1970’s

Fault tree analysis example
Fault Tree Analysis: Example

Simplified fault tree diagram for an infusion pump

Fault tree analysis1
Fault Tree Analysis

  • Demonstrates that the system will not reach an unsafe state

  • Identifies areas for improvement

  • Provides a systematic hazard review

Developing medical software pitfalls and prophylactics

  • Assume a basic defect at the component level, assess the effect and identify potential solutions

  • Answer the question:

    • What happens if event X occurs?

  • Used in Automobile manufacturing

Fmea example
FMEA: Example


Subsystem/Name: DC motor P = Probabilities (chance) of Occurrences

Model Year/Vehicle(s): 2000/DC motor S = Seriousness of Failure to the Vehicle

D = Likelihood that the Defect will Reach the customer

R = Risk Priority Measure (P x S x D)

1 = very low or none 2 = low or minor

3 = moderate or significant 4 = high 5 = very high or catastrophic

Developing medical software pitfalls and prophylactics

  • Reveals unforeseen hazards

  • Does not consider multiple failures

  • Can be very time consuming


  • A process for tracking system reliability/safety

  • A set of procedures, policies and software tools

  • Used from beginning to end of the product lifecycle


  • Record events

  • Analyze failure modes

  • Verify corrective actions

  • Identify failure trends

  • Determine the failure contribution of individual parts

Risk hazard analysis tools1
Risk/Hazard Analysis Tools

  • Fault Tree Analysis

  • Failure Modes Effect and Criticality Analysis (FMECA)

  • Failure Reporting and Corrective Action System (FRACAS)

  • Failures occur, the question is: Did we prepare for them?


  • People depend on medical software with their lives!

  • Safety is part of the design and development process


  • Nancy Leveson, “Safeware: System Safety and Computers”, Addison-Wesley, 1995

  • Therac-25 case study

  • Guidance for the Content of Premarket Submissions for Software Contained in Medical Devices,

  • Laura M. Ippolito, Dolores R. Wallace. “A Study on Hazard Analysis in High Integrity Software Standards and Guidelines “, NISTIR 5589,

  • Daniel Kamm, P.E.,C.Q.A., “An Introduction to Risk/Hazard Analysis for Medical Devices”,