slide1 n.
Skip this Video
Loading SlideShow in 5 Seconds..
Alan Rector, Luigi Iannone, Robert Stevens PowerPoint Presentation
Download Presentation
Alan Rector, Luigi Iannone, Robert Stevens

Loading in 2 Seconds...

play fullscreen
1 / 25

Alan Rector, Luigi Iannone, Robert Stevens - PowerPoint PPT Presentation

  • Uploaded on

Quality Assurance of the Content of a Large DL-based Terminology using Mixed Lexical and Semantic Criteria: Experience with SNOMED CT. Alan Rector, Luigi Iannone, Robert Stevens “A report from the trenches”.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Alan Rector, Luigi Iannone, Robert Stevens' - miles

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Quality Assurance of the Content of a Large DL-based Terminology using Mixed Lexical and Semantic Criteria: Experience with SNOMED CT
  • Alan Rector, Luigi Iannone, Robert
a report from the trenches
“A report from the trenches”
  • SNOMED-CT - mandated terminology for electronic patient records in UK, US, & worldwide aspirations
    • The result of a merger of two other systems
      • SNOMED and Clinical Terms v3
      • Long history with much opportunity for error
    • Expressed in a Description Logic and now available in OWL
      • subset of EL++ without disjoint axioms
    • Has been resistant to independent analysis although many known problems
      • Despite several global QA attempts based on lexical criteria that have identified errors without explaining them
it s very big and classification matters
It’s very big - and classification matters
  • ~400,000 Concepts/Classes; >1,000,000 axioms
  • Much of richness only evident in classified for m
    • Most errors only present in classified form



an experiment of opportunity
An experiment of opportunity
  • The opportunities
    • Tried to use SNOMED for Commercial Collaboration on Clinical Systems
    • Tried to use SNOMED as contribution to WHO’s revsion of International Classification of Diseases (ICD-11)
    • Problems with both
    • Therefore, experiment if QA & repair were possible
      • Conventional wisdom said that it was not
  • However, we had new resources
    • Core Problem List Subset from NLM (8500 most used classes)
    • Software to extract “modules”
    • SNOROCKET Classifier for EL++
    • 4-8GB machines
step 1 cut it down find a classifier
Step 1: Cut it down & find a classifier
  • Find a subset
    • UMLS Core Problem List subset -
    • 8500 most used disease concepts
      • Collected by US National Library of Medicine by combining sets from 6 major institutions.
  • Extract a “Module” (built into OWL API v3)
    • Use core subset as “signature”
    • Guaranteed that all inferences amongst the classes in “signature” in whole will hold in module
    • 35,000 concepts - including most of anatomy
  • Find a classifier that can cope - at least two for checking
    • SNOROCKET (EL++) polynomial time subset of OWL (30 sec)
    • Pellet 2.1 (200 sec)
    • FaCT++ (250 sec)
step 2 pick some areas of interest to clinicians some with anomalies already spotted
Step 2: Pick some areas of interest to clinicians: some with anomalies already spotted
  • Myocardial Infarction (Heart attack)
    • Should be a kind of Ischemic Heart Disease, but wasn’t
  • Hypertension (High blood pressure)
    • Odd to find it a kind of Soft Tissue disorder
  • Diabetes
    • Odd to find it as a Disorder of the Abdomen
  • Allergies
    • Odd to find some but not all autoimmune disorders classified as Allergies.
look at classification most initial errors spotted looking upwards
Look at classification:Most initial errors spotted looking upwards
  • Look up hierarchy (with OWLViz)
    • Let clinicians find important concepts and check them
      • Face validity and then look up the hierarchy
    • Check any anomalies against the complete SNOMED in standard browser
      • Guard against artifacts in various transformations
    • Trace anomalies to their root
    • Decide which links to add or break
    • Decide how to break them
    • Edit, classify and check
      • Hierarchies
      • Usages
examine definition formulate solution
Examine definition & formulate solution

Disorder of blood vessel that

(Finding site some Systemic arterial structure) and (Has definitional manifestation some Increased blood pressure))

Disorder of blood vessel that

(Finding site some Cardiovascular system structure) and (Has definitional manifestation some Increased blood pressure)

Then check usages for unwanted results - anything that should relate to arteries instead of Cardiovascular system?
also look down hierarchy combine lexical semantic search
Also look down hierarchy:Combine lexical & semantic search
  • Hard to spot what is missing
    • Hypertensive disorders included some complications as well as kinds of hypertension. Did it contain them all?
  • Use OPPL combining lexical, owl semantics & queries
    • ?C:CLASS=MATCH(“.*[Hh]ypertensive.*”)  lexicalSELECT ?C SubClassOf Thing  open world OWL semanticsWHERE FAIL ?C SubClassOf “Hypertensive disorder”  closed world queryBEGIN ADD ?C SubClassOf Candidate_hypertensive END;  action
  • Classify and look at odd cases

look for regularities
Look for regularities
  • Of hypertensive complications
    • 1 linked to Hypertensive disorder by property due to
    • 1 linked to Hypertensive disorder by property associated with
    • 2 are subclasses of Hypertensive disorder
    • 2 not linked at all
  • No class for Hypertensive complication
    • Although there is a class for Diabetic complication
  • Regularise
    • Create classes for
      • Hypertension,
      • Hypertensive complication and
      • Hypertension AND/OR Hypertensive complication
    • Edit all complications to schema:Disorderdue to some Hypertension
which concept should carry the old id
Which concept should carry the old ID?
  • Look at usages of Hypertensive disorder
    • All fit Hypertension; none fit Hypertensive complication
    • Therefore, label original ID for Hypertensive disorder as Hypertension
      • New Hierarchy:
        • Hypertension AND/OR Hypertensive complication  new ID/concept Hypertension  old ID/concept …kinds of hypertension Hypertensive complication…  new ID/concept … kinds of hypertensive complication
looking down hierarchy analysis by categorisation
Looking down hierarchy:Analysis by categorisation
  • Break it up logically



Even short alphabetic lists are difficult to check

always trace errors to root to fix mish mash modelling
Always trace errors to root to fixmish mash modelling
  • Simple error
    • The axiom that Skin is a kind of Soft tissue was omitted
    • Therefore Injuries to skinare not listed as kinds of Soft tissue injuries
  • Authors have noticed some cases and tried to compensate
    • Cut of skin of footis a kind ofsoft tissue injury, butCut of the skin of lower limb was NOT a soft tissue injury
    • One axiom to fix it all: Skin subClassOf SoftTissue:
      • And then a script to find the redundant axioms
trace errors to their roots incomplete modelling example
Trace errors to their roots: Incomplete modelling: Example
  • Why is Myocardial Infarction not a kind of Ischemic Heart Disease?
      • Ischemia = “lack of blood supply”
      • Myocardium = “Heart muscle”
    • Infarction not fully defined in SNOMED. References say…
      • “Tissue death due to ischemia”
    • Ischemic heart disease not fully defined SNOMED, Refs say…
      • Heart disease due to ischemia
    • Ischemic disorderdoes not exist in SNOMED, Natural closure…
      • Disorder due to some Ischemia - NB always involves Cardiovascular system
  • Add definitions and Myocardial infarction classified correctly
    • Also discover a long list of Ischemic disease that have not been classified as cardiovascular
  • Check lexically for other uses of “ischemic”
    • None found in this subset
error in schema for anatomy conflates branches with parts
Error in schema for anatomy:Conflates branches with parts
  • Example
    • Injury to artery of the ankle is located in the pelvis and in the abdomen (as well as the ankle)!
  • Extends to all nerves & blood vessels
  • Requires a generic change
    • Simplest involves about 20 axioms for arteries
overgeneralisation explains many arguments
Overgeneralisation – explains many arguments
  • The dictionary says “Neuropathy” is a disease of nerves
    • But in practice it is a “dysfunction” of nerves
      • Doctors don’t consider tumors or injuries to nerves to be neuropathies
    • SNOMED often does not distinguish structural and functional disorders
      • Needs a consistent pattern:
naming issues
Naming issues
  • All SNOMED terms have at least two names
    • “Fully qualified name” & “Preferred name”
    • “Fully qualified names” should be consistent but…
  • Example - conflicting names
    • “Immune hypersensitivity disorder (disorder) = “Allergic disorder”
    • Structure nodes in SEP triples
      • “Structure of X”, “X Structure”, X
        • Leads to “Swelling of gums” is kind of “Swelling of face”
doing everything in a separate module insofar as possible
Doing everything in a separate module(insofar as possible)

Perform queries as “probes”

Perform queries as “probes”

Keep changes in Modules

Compromise: System of diffs and merges

summary qa of a large dl based ontology is possible
Summary: QA of a large DL-based ontology is possible!
  • Find a useful subset and use it as signature to extract a manageable module
  • Start with things that are important to your experts
    • Look upwards rather than downwards in the first instance
    • Follow up analogies and patterns
    • When looking downwards enrich categorization to reduce noise
      • Combine lexical and semantic techniques
  • Analysis by synthesis -
    • test alternative potential changes with classifier
    • as far as possible in a separate module; scripting where possible
  • Tooling gaps / weaknesses
    • Scripting tools need work
    • Combining filtering with imports
    • Diffs & change management – needed but don’t enough
  • Log everything!