Advanced Knowledge Technologies

1. Advanced Knowledge Technologies University of Aberdeen University of Edinburgh University of Sheffield Open University University of Southampton http://www.aktors.org MIAKT Oxford University King’s College, London University of Sheffield Open University University of Southampton CoAKTinG University of Edinburgh Open University University of Southampton

2. MIAKT: Medical Informatics and Knowledge Technologies Supporting triple-assessment (TA) (collaborative decision-making) for the diagnosis and treatment of breast cancer Oxford University Kings College Open University University of Sheffield University of Southampton

3. To support collaboration for the e-Scientist Intelligent meeting spaces: Decision rationale, group memory capture Planning, coordination support Instant messaging/presence Open University University of Edinburgh University of Southampton

4. Advanced Knowledge Technologies Representation and Reasoning Ontologies: domain and process models Interoperability Integration with databases for scalability (Semantic Web) Reasoning services – local and distributed Natural language processing Application domain – academic CS

5. People involved MIAKT Oxford University Mike Brady, Jon Whitely King’s College London David Hawkes, Christine Tanner, Yalin Zheng The Open University Enrico Motta, John Domingue, Liliana Cabral University of Sheffield Yorick Wilks, Fabio Ciravegna, Kalina Bontcheva University of Southampton Nigel Shadbolt, Srinandan Dasmahapatra, Paul Lewis, Bo Hu, Hugh Lewis CoAKTinG University of Edinburgh Austin Tate Stephen Potter Jessica Chen-burger Jeff Dalton Open University Marc Eisenstad Simon Buckingham Shum Jiri Komzak Michelle Bachler University of Southampton David De Roure Nigel Shadbolt Danius Michaelides Richard Beales Kevin Page Ben Juby

6. Breast Cancer – Statistics & Screening • EU: 24% of cancer cases 19% of cancer deaths 1 in 8 of women will develop breast cancer during the course of their lives 1 in 28 will die of the disease. • 5 year survival rate for localized breast cancer is 97% for early detection is 77% if the cancer has spread at diagnosis is 22% if distant metastases are found • Screening for ages 50+ • M Brady: MIAS (features), e-Diamond (priors) • Knowledge technology support

7. MIAKT: Patient Management --Triple Assessment

8. Imaging:Mammography/Ultrasound/MR Mammography (X-ray) Position breast on small flat plate, with X-ray plate under it. Flat plate above your breast. When machine is switched on, breast pressed down between plates by machine to get clearest picture. Two pictures are taken: from above and from the side. Ultrasound usually used for women under 35 (breasts too dense or solid to give a clear picture with mammography) It is also used to see if a breast lump is solid contains fluid (a cyst)

9. Histopathology Fine needle aspiration cytology Core biopsy With imaging guidance

10. MIAKT: Triple assessment

11. Triple Assessment Clinical and radiological opinions are used independently to decide upon further intervention The most suspicious opinion prevails (normal/definitely benign, probably benign, indeterminate, probably malignant) Needle biopsy is mandatory for all abnormalities classified as indeterminate or more suspicious Needle biopsy results are discussed in the context of imaging and clinical findings at multidisciplinary meetings

12. Radiology and Histopathology images: Different scales Same Patient: multiple descriptors

13. Ontology, Annotation, Language Generation Annotated images for retrieval in context Memory aids for specialists Report generation from annotations Ontologies for descriptive grounding Correlative reasoning across specialisation (across scales orders of magnitude apart) Distributed reasoning (grid?)

14. Knowledge Engineering Lexical and ontological issues Decision support Records – image and text

15. Diagnostic Mammography:different views To pinpoint exact size & location of breast abnormality and to image surrounding tissue and lymph nodes. mediolateral lateromedial Cranio-caudal (CC) & Mediolateral oblique (MLO) views

16. Diagnostic Mammography:BI-RADS Ontology - Masses 1. SHAPE 1.a. Round 1.b. Oval 1.c. Lobular 1.d. Irregular 2. MARGINS 2.a. Circumscribed 2.b. Microlobulated 2.c. Obscured 2.d. Indistinct 2.e. Spiculated 3. DENSITY: 3.a. High density 3.b. Equal density 3.c. Low density 3.d. Fat containing – radiolucent – oil cyst, lipoma, or galactocele as well as mixed lesions such as hamartoma or fibroadenolipoma. [and/or histologic terms] MASS: space occupying lesion seen in two different projections. If potential mass seen in single projection, called DENSITY until 3-D confirmation.

17. BI-RADS Ontology – Masses(details) MASS: space occupying lesion seen in two different projections. If potential mass seen in single projection, called DENSITY until 3-D confirmation. 1. SHAPE a. Round: spherical, ball-shaped, circular or globular b. Oval: elliptical or egg-shaped. c. Lobular: has contours with undulations. d. Irregular: none of the above. 2. MARGINS [modify the shape of the mass] a. Circumscribed Margins: abrupt transition between the lesion and the surrounding tissue. Without additional modifiers there is nothing to suggest infiltration. b. Microlobulated Margins: undulate with short cycles producing small undulations. c. Obscured Margins: hidden by superimposed or adjacent normal tissue; cannot be assessed any further. d. Indistinct Margins: poor definition of margins raises concern of infiltration by the lesion; not likely due to superimposed normal breast tissue. e. Spiculated Margins: lines radiating from margins of mass 3. DENSITY: x-ray attenuation of lesion relative to the expected attenuation of an equal volume of fibroglandular breast tissue; most cancers are of equal or higher density; never fat containing but may trap fat. a. High density b. Equal density c. Low density d. Fat containing – radiolucent - oil cyst, lipoma, or galactocele as well as mixed lesions such as hamartoma or fibroadenolipoma. [When appropriate, histologic terms may be included]

18. Microcalcification Microcalcifications: (<.5mm) specks of calcium in milk ducts. About half of the cancers detected by mammography appear as a cluster of microcalcifications. Microcalcifications are the most common mammographic sign of ductal carcinoma in situ P(micro-Ca | DCIS)=0.9

19. BI-RADS: Calcifications • Benign calcifications usually larger than malignant ones -- coarser, often round with smooth margins, more visible. • Malignant calcifications very small, require magnifying glass. • When specific aetiology not possible, description of calcifications should include their distribution and morphology. • Benign calcifications only reported if judged to be susceptible to misinterpretation.

20. Multiple views on domain • Structure, anatomy • Function, physiology • Pathology • Patient history

21. BI-RADS Ontology: Calcification Typically benign Intermediate concern Higher probability Distribution modifiers

22. BI-RADS Ontology: Calcification 2. INTERMEDIATE CONCERN 2.a. Amorphous or Indistinct Calcifications 1. TYPICALLY BENIGN 1.a. Skin Calcifications 1.b. Vascular Calcifications 1.c. Coarse Calcifications 1.d. Large Rod-Like Calcifications 1.e. Round Calcifications 1.f. Lucent-Centered Calcifications 1.g. Eggshell or Rim Calcifications 1.h. Milk of Calcium Calcifications 1.i. Suture Calcifications 1.j. Dystrophic Calcifications 1.k. Punctate Calcifications 4. DISTRIBUTION MODIFIERS 4.a. Grouped or Clustered 4.b. Linear 4.c. Segmental 4.d. Regional: 4.e.Diffuse/Scattered 3. HIGHER PROBABILITY OF MALIGNANCY 3.a. Pleomorphic or Heterogeneous Calcifications 3.b. Fine, Linear or Branching Calcifications

23. BI-RADS Ontology: Calcification (details) TYPES AND DISTRIBUTION OF CALCIFICATION: 1. TYPICALLY BENIGN - a. Skin Calcifications: typical lucent centered deposits that are pathognomonic. Atypical forms confirmed by tangential views to be in the skin. b. Vascular Calcifications: Parallel tracks, or linear tubular calcifications clearly associated with blood vessels. c. Coarse Calcifications: Classic calcifications produced by an involuting fibroadenoma. d. Large Rod-Like Calcifications: Continuous rods, occasionally branching, diameter > 1mm usually, may have lucent centers, if calcium surrounds rather than fills an ectactic duct. Found in secretory disease, "plasma cell mastitis", and duct ectasia. e. Round Calcifications: When multiple, of variable size. Considered benign and when small [under 1 mm], frequently formed in acini of lobules. Under 0.5 mm are termed punctate. f. Lucent-Centered Calcifications: Less that 1 mm to greater than 10 mm, smooth surfaces, round or oval, have lucent center. “Wall" thicker than the "rim or eggshell" type. Included are areas of fat necrosis, calcified debris in ducts, and occasional fibroadenomas. g. Eggshell or Rim Calcifications: Very thin, under 1mm thickness, appear as calcium deposited on the surface of a sphere. Although fat necrosis can produce these thin deposits, calcifications in the wall of cysts are the most common "rim" calcifications. h. Milk of Calcium Calcifications: Consistent with sedimented calcifications in cysts. Often less evident in craniocaudal image -- appear as fuzzy, round, amorphous deposits; sharply defined on 90° lateral -- semilunar, crescent shaped, curvilinear (concave up), or linear defining dependent portion of cysts. i. Suture Calcifications: Ca deposited on suture material, relatively common in post-irradiated breast, typically linear or tubular in appearance and knots are frequently visible. j. Dystrophic Calcifications: Usually form in irradiated breast or following trauma. Irregular in shape, usually > 0.5 mm, often have lucent centers. k. Punctate Calcifications: Round/oval, < 0.5 mm with well-defined margins.

24. Example Mammogram 2 cm mass (tumour) microcalcifications

25. Microcalcifications: Clinical Procedures (guideline) Microcalcifications Discharge! Further Ultrasound and Mammography benign suspicious Clustered heterogeneous Clinical exam + Needle core biopsy With Specimen radiography Normal or Definitely benign Equivocal result Significant abnormality Triple assessment Treatment

26. Histopathology • Fine needle aspiration cytology • Core biopsy • With imaging guidance F FNAC Descriptors when drawing sample: mm Sampling pattern for stereotactic FNAC

27. Histopathology slides • A histological slide has an immense amount of data, the closer you look, the more there is • Histological images are complex with challenges at both the segmentation and feature classification level • Think in terms of two scales – low-power, high-power

28. Histopathology

29. Histopathology slides:low power/high power

30. Histopathology slides: diagnostic criteria

31. Reporting guidelines

32. MIAKT: Technology Palette • MIAS – • Medical image registration (X-ray, MR) • Segmentation and feature extraction • Image Classification • AKT – • Ontology development • (Distributed) reasoning services • Image annotation against ontologies • Natural language generation • Decision support – belief nets?

33. MIAKT • Abstract away from the details of TA meeting • Collaborative problem solving/decision making • Possibly distributed, virtual presence • Well-defined goals, well-defined contributory skill sets • Structured protocol • Require recall of contents of events of past meeting • Report generation (audit trail)

34. Enhance Technologies • Ontologically annotated audio/video streams • Issue handling, tasking, planning and coordination • Collective sense-making and group memory capture • Enhanced presence management and visualisation • Adaptive information systems

35. Technology Integration Aim: To support e-Science collaboration by integrating and demonstrating the utility of: • intelligent task-orientated messaging, collaborative planning, issue, activity and constraint management (I-X Process Panels/<I-N-C-A>: Edinburgh) • peripheral awareness of the online presence, availability, attributes and location of colleagues, documents, and devices (BuddySpace: OU) • real time conversational mapping of meetings, providing shared visual focus and group memory capture (Compendium: OU) • multimedia meeting mark-up, replay and navigation (HyStream: Southampton)

36. Jabber • Jabber is a set of XML-based protocols for real-time messaging and presence notification • Communicates with other instant messaging services through gateways • Many clients available – see http://www.jabbercentral.org/

37. Jabber

38. Compendium • Provides a methodological framework, plus an evolving suite of tools, for collective sense-making and group memory. • Intersection of collaborative modelling, organisational memory, computer-supported argumentation and meeting facilitation. • Centres on face-to-face meetings, potentially the most pervasive knowledge-based activity in working life, but also one of the hardest to do well.

39. Compendium

40. BuddySpace • ‘Enhanced Presence Management for Collaborative Working, Messaging, Gaming and Beyond’ • The concept of presence is a rich combination of attributes that characterise an individual's… • physical and/or spatial location • work trajectory • time frame of reference • mental mood • goals and intentions http://kmi.open.ac.uk/projects/buddyspace/

41. BuddySpace Compendium

42. Process Panels • Based on notion of the representation of a product as a set of nodes making up the components of the product model, along with constraints on the relationship between those nodes and a set of outstanding issues • Investigates the use of shared models for task directed communication between human and computer agents who are jointly exploring a range of alternative options for activity.

43. Process Panels

44. HyStream

45. Smart spaces • Devices in the room enables us to capture continuous (real time, multi-way, multi-cast, ontologically informed, …) metadata • Other devices provide ‘presence’ information • Consider an experimental laboratory instead of a meeting room: • Instruments • Electronic log books • Visualisation