slide1 n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
11:45 – 12:30      From IHE Profiles to conformance testing , closing the implementation gap PowerPoint Presentation
Download Presentation
11:45 – 12:30      From IHE Profiles to conformance testing , closing the implementation gap

Loading in 2 Seconds...

  share
play fullscreen
1 / 102
yana

11:45 – 12:30      From IHE Profiles to conformance testing , closing the implementation gap - PowerPoint PPT Presentation

74 Views
Download Presentation
11:45 – 12:30      From IHE Profiles to conformance testing , closing the implementation gap
An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. 11:45 – 12:30      • From IHE Profiles to conformancetesting, closing the implementation gap • Helping the implementers, testingtools, connectathons • 12:30 – 13:30 Lunch Break • 13:30 - 15:00 • How to use IHE resources: hands on experience • TechnicalFrameworks: navigating, Q&A • Test tools: finding, using, configuring • Participating in the testingprocess

  2. IHE Resources Eric Poiseau, INRIA, IHE Europe technical manager Charles Parisot, GE, IHE Europe

  3. Connectathon

  4. History

  5. Connectathon • Started in 1998 in Chicago within the RSNA HQ • Europe started in 2001 • Japan in 2003 • China and Australia now also in the process

  6. Charenton le pont 2001 • 11 companies • 18 systems • 40 m2 • 30 participants Formation IHE France

  7. Paris 2002 • 33 companies • 57 systems • 130 m2 • 100 participants Formation IHE France

  8. Aachen 2003 • 43 companies • 74 systems • 350 m2 • 135 participants Formation IHE France

  9. Padova 2004 • 46 companies • 78 systems • 600 m2 • 180 participants Formation IHE France

  10. Noordwijkerhout 2005 • 75 companies • 99 systems • 800 m2 • 250 participants Formation IHE France

  11. Barcelona 2006 • 67 companies • 117 systems • 1500 m2 • +250 participants Formation IHE France

  12. Berlin 2007 • Companies • systems • 1500 m2 • +300 participants

  13. Oxford 2008 • 83 companies • 112 systems • 1500 m2 • 300 participants

  14. C.A.T Participation in Europe Berlin Oxford Barcelona Noordwijkerhout Padova Aachen Paris Paris

  15. Purpose • Test the implementation of the integration profile within product • Verify that the vendors did a good job • Verify that what the committees invented makes sense ! • Verify that the text is clear enough • Verify that that the committee did not miss anything • Build a community of …

  16. Computer geeks…

  17. …who like to enjoy local brewed beers

  18. From the vendor perspective • Unique Opportunity for vendors to test their implementations of the IHE integration profiles • Controlled environment • Customer is not present ! • Not in a clinical production environment • Specialists available • From SDO • From the peer companies • Bugs are identified and most of the time fixed !!!! • Connectathon Result Matrix • http://sumo.irisa.fr/con_result

  19. But… • Testing is sub-optimal • Only a part of all the possible tests are performed • A system successful at the connectathon is not guaranteed to be error free !!!! • We do not do certification !

  20. From the IHE perspective • Feedback from the vendor community • Did the committee do a good job ? • Did the developed integration profile respond to a demand of the vendors ?

  21. European C.A.T • We have reach now our cruise speed • NA and EU C.A.T are very alike • C.A.T used as an IHE promoting tool • Workshop in parallel to the C.A.T • Berlin : ITEG • Oxford • Vienna

  22. C.A.T. Model

  23. The IHE testing process Users Testing Results Deployed Systems Testing Tools Sponsors:Project Management Team Develop Testing Tools Approves Test Logs Connectathon Product +IntegrationStatement Implement Profile Actors In-House Testing Vendors Sponsors: Exhibits Demonstration IHE Technical Framework (Profiles Specification) Projet IHE-Dev Inria Rennes

  24. Pre-connectathon

  25. Pre-connectathon • Registration • See what can be tested • Exchange of configuration parameters • IP addresses • AE Title • Assigning authorities • OID • Certificates • Affinity domain specification

  26. Pre-connectathon • Mesa testing • In-house testing for vendors to get ready • Vendors return logs • Upon log return participation to C.A.T is accepted

  27. Atconnectathon

  28. Connectathon Testing • 3 types of test to be performed • No peer tests • Peer to peer tests • Workflow tests Participant Workshop

  29. No Peer Tests • Calibration Tests -CPI : • screen calibration • Printer calibration • Scrutiny Tests • Verifythat the objectscreated are « valid » • Providepeerswithsamples Participant Workshop

  30. Peer To Peer Tests (P2P) • Test subsections of a workflowbetween 2 vendors • Preparation to workflow test • Vendor chose when to runthem • Vendor select theirpeer. • Not to berunwithothersystemsfromsamecompany Participant Workshop

  31. Workflow Tests • Test an entire workflow that may combined more than one integration profile • We have a schedule, vendors need to be ready at the time of the test. • We have a list of difficulties to check. • Some test can run in 15 minutes • Some will require more than an hour • No second chance test Participant Workshop

  32. 5 days • Monday morning till 11 am • Set up time • Till Friday noon : • Free peer to peer and no peer testing • From Wednesday till Friday noon : • Directed workflow testing

  33. Monitors • Volunteers • Independent from vendors • Standard specialist • Verify tests • Act as moderator between vendors

  34. Results • Failure are not reported • To be successful • Each peer to peer test needs to be verified with at least 3 peers • There are some exceptions • A vendor may fail for an actor but pass for the others

  35. Connectathon Results • IHE does not report failure • Public resultsonlyat the companylevel • IHE willnever tell youwhat system participated to the connetathon • Vendors have access to theirown test results. Formation IHE France

  36. Connect-a-thon Results Browser Formation IHE France

  37. ConnectathonResults Browser Formation IHE France

  38. ConnectathonResults Browser Formation IHE France

  39. What does it mean ? • The Companywassuccessfulat the connectathon for the actor/integration profile combination • Results do not guarantyproductconformity • This is the role of the « IHE integrationstatements » Formation IHE France

  40. IHE Integration Statement Formation IHE France

  41. Participation Fees • First System € 2750 • Othersystems € 2850 • Per domain €   750  • Covers : • Infrastructure : room, power, monitors, internet… • Lunch and coffee breaks for 2 engineers during 5 days

  42. Next Connectathon • Where : Remise, Vienna, Austria • http://www.koop-kundenweb.at/remise/ • When : Monday 20th April to Friday 24th April 2009 • Registration : November 1st – January 7th 2009 • Announcement to bereleasedsoon

  43. CAT : conclusion

  44. C.A.T : Conclusion • It’s not a certification process • Unique opportunity for vendor to test and discuss • Seems to beusefull as proved by increased participation over the years • Sure, needsimprovement… • … but, we are working on it

  45. Testing

  46. Beforewestart • Impossible to test everything • Whatwe do not test • Design • Performance (Load) • Whatwe are looking for • interoperability • conformity Projet IHE-Dev Inria Rennes

  47. Conformance / Interoperability Specifications/Standards Conformance testing ImplementationA ImplementationB Interoperability testing VendorA Vendor B Projet IHE-Dev Inria Rennes

  48. Conformance Testing (1/2) • Is unit testing • Tests a single ‘part’ of a device • Tests against well-specified requirements • For conformance to the requirements of specified and the referenced standards • Usually limited to one requirement per test. • Tests at a 'low' level • At the protocol (message/behaviour) level. • Requires a test system (and executable test cases) • Can be expensive, tests performed under ideal conditions

  49. Conformance Testing (2/2) • High control and observability • Means we can explicitly test error behaviour • Can provoke and test non-normal (but legitimate) scenarios • Can be extended to include robustness tests • Can be automated and tests are repeatable • Conformance Testing is DEEP and NARROW • Thorough and accurate but limited in scope • Gives a high-level of confidence that key components of a device or system are working as they were specified and designed to do

  50. Limitations of Conformance Testing • Does not prove end-to-end functionality (interoperability) between communicating systems • Conformance tested implementations may still not interoperate • This is often a specification problem rather than a testing problem! Need minimum requirements or profiles • Does not test a complete system • Tests individual system components, not the whole • A system is often greater than the sum of its parts! • Does not test functionality • Does not test the user’s ‘perception’ of the system • Standardised conformance tests do not include proprietary ‘aspects’ • Though this may well be done by a manufacturer with own conformance tests for proprietary requirements