1 / 21

Benchmarking the Interoperability of Ontology Development Tools

I nteroperability W orking D ays October 10th-11th, 2005. Benchmarking the Interoperability of Ontology Development Tools. Raúl García-Castro <rgarcia@fi.upm.es > October 10th, 2005. ¿=?. Ontologies are available in internet. Conceptualize. Evaluate. Extend. Specify. Specialize.

alvin-riley
Download Presentation

Benchmarking the Interoperability of Ontology Development Tools

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Interoperability Working Days October 10th-11th, 2005 Benchmarking the Interoperability of Ontology Development Tools Raúl García-Castro <rgarcia@fi.upm.es> October 10th, 2005

  2. ¿=? Ontologies are available in internet Conceptualize Evaluate Extend Specify Specialize Import Prune Evolution Identify Differences O3 O2 O1 Integrate Export Alignment Reasoning Merge Anotate Document ... +

  3. Interoperability problem • It appears due to ontology reuse. Tool 2 Tool 4 Ontology development tools Tool 1 Tool 3 Tool 5 Potential functionalities Real functionalities

  4. Interoperability problem • Why is it difficult? • Different KR formalisms • frames description logics conceptual graphs • first order logic semantic networks • Different modelling components inside the same KR formalism The great diversity in languages and tools originates two problems: • The translation problem: • How can we translate an ontology between two different languages without losing knowledge? • The interoperability problem: • How can two tools interchange ontologies or parts of them without losing knowledge? • How can a tool use ontologies or parts of them from other tool?

  5. O. Corcho. A Layered Declarative Approach to Ontology Translation with Knowledge Preservation Frontiers in Artificial Intelligence and Applications, Volume 116, January 2005 Interoperability problem • Some results: • It is difficult to preserve the semantics and the intended meaning of the ontology • Interoperability decisions… • At many different levels • Usually hidden in the programming code of ontology exporters/importers

  6. Do Tools interoperate? Ontology development tools Evaluation tools ICS-FORTH Validating RDF Parser, RDF Validation Service Ontolingua DAML+OIL RDF(S) RDF(S) OWL ... OWL Ontology Validator, OWL Validator Loom DAML Validator, DAML+OIL Ontology Checker OWL Interoperability is crucial in the semantic web RDF(S) OIL OWL SilRI RIL TRIPLE Reasoners FACT RACER RQL RDQL SeRQL SPARQL OntoBroker Flora, Flora2 FORID Query systems OIL RDF(S) Flora, Flora2 Flogic RDF(S) DAML+ OIL DAML+OIL OWL Flogic Loom FACT RACER TRIPLE JTP DQL Loom Classifier FACT RACER TRIPLE BOR OWL OWL-QL

  7. T 1 T 2 T 3 T 4 T 5 T 6 Interoperability approaches Interoperability through an interchange language Tool i Alternative 1. Ontology interchange using an interchange language Interchange language Tool j Interoperability through application program interfaces (APIs) Alternative 2. Direct interchange between each two Alternative 3. Common API T 1 T 2 T 3 Common API T 4 T 5 T 6

  8. Knowledge models comparison Partitions Subproperty-of Concept groups RDF(S) Class attributes Containers Disjoint decompositions Exhaustive decompositions Collections Relation properties Statements Bibliographic references Template Slots/properties/ instance attributes Constants Literals Abbreviations Subclass-of Synonyms Data types Instances Metaclasses Classes Own slots

  9. Translation strategies RDF(S) EXPORT IMPORT Minimize knowledge loss Partial loss Thesis Thesis subclass subclass Thesis MSc Thesis PhD Thesis MSc Thesis PhD Thesis Disjoint-subclass Don’t export Total loss MSc Thesis PhD Thesis Insert ad-hoc RDF(S) <rdfsClass rdf:about=“#Thesis”> <a:disjoint rdf:Resource=“#MsC Thesis”> <a:disjoint rdf:Resource=“#PhD Thesis”> </rdfs:Class> Doesn’t import Thesis Thesis

  10. Interoperability using an interchange language Ontolingua RDF(S) OWL ... Loom RDF(S) OIL OWL Interoperability is not guaranteed with the existence of importers and exporters from the tools to the interchange language. Interoperability using an interchange language depends on the correct working of the translators… …and these translators don’t work properly.

  11. How can we improve this situation? Benchmarking: Goal 1: • To assess and improve the interoperability of ontology development tools using RDF(S) for ontology exchange. • Evaluation of the interoperability • Improvement of the interoperability • Collection of best practices • Goal 2: • To identify the subset of RDF(S) elements that ontology development tools can use to correctly interoperate. • Goal 3: • Next step: OWL.

  12. García-Castro, Maynard, Wache, Foxvog and González-Cabero. Knowledge Web Deliverable 2.1.4 Specification of a methodology, general criteria, and benchmark suites for benchmarking ontology tools. December 2004. PLAN PHASE 1. B. goals identification 2. B. subject identification 3. Participant identification 4. B. proposal writing 5. Management involvement 6. B. partner selection 7. B. planning and resource allocation EXPERIMENT PHASE 8. Experiment definition 9. Experiment execution 10. Experiment results analysis IMPROVE PHASE 11. B. report writing 12. B. findings communication 13. Improvement planning 14. Improvement 15. Monitor General framework for benchmarking BENCHMARKING ITERATION Recalibration task • General evaluation criteria: • Interoperability • Scalability • Robustness • Benchmark suites for: • Interoperability • Scalability • Robustness • Benchmarking supporting tools: • Testing frameworks • Workload generators • Monitoring tools • Statistical packages

  13. B.P. Plan phase RDF(S) import and export capabilities Improve the interoperability of ontol. development tools Organisation's tools Identify ontology components exported/imported Need for benchmarking Benchmarking subject, tool functionalities, evaluation criteria Benchmarking goals, benefits, costs Benchmarking goals identification Benchmarking subject identification Participant identification List of involved members, benchmarking team Organisation goals and strategies Benchmarking proposal writing Organisation planning Benchmarking proposal Benchmarking partners, updated benchmarking proposal Benchmarking planning and resourceallocation Management support Management involvement Benchmarking partner selection Tools from outside the organisation Benchmarking planning

  14. test 1 • test 2 • test 3 • ... • test 1 • test 2 • test 3 • ... • test 1 • test 2 • test 3 • ... • test 1 • test 2 • test 3 • ... • test 1 • test 2 • test 3 • ... • test 1 • test 2 • test 3 • ... ... NO OK OK OK NO OK OK OK NO NO OK OK OK NO OK OK OK NO E.R. Experiment phase RDF(S) Import benchmark suites RDF(S) Export benchmark suites Benchmarking proposal Experiment definition, experimentation planning Experiment report Experiment results Experiment definition Experiment execution Experiment analysis Benchmarking planning

  15. Comparative analysis • Compliance with standards • Weaknesses • Recommendations on tools • Recommendations on practices Improve phase Updated benchmarking proposal Experiment report Benchmarking report Benchmarking findings communication Benchmarking report writing Organisation support Updated benchmarking report Improvement planning Improvement Necessary changes, improvement planning, improvement forecast Improved tool Monitorisation report Monitor

  16. Q1: Which are the elements of the knowledge model of a tool that are interchanged with another tool without knowledge loss? Q2: Which are the elements of the knowledge model of a tool that are interchanged with another tool with knowledge loss? Q3: Which are the elements of the knowledge model of a tool that are not interchanged with another tool? M4: Knowledge loss when importing from RDF(S) into another tool an element of the knowledge model of a tool M5: Elements of the knowledge model of a tool that are not exported to RDF(S) M6: Elements of the knowledge model of a tool that are not imported from RDF(S) intro another tool M1: Elements of the knowledge model of a tool that are exported to RDF(S) M2: Knowledge loss when exporting to RDF(S) an element of the knowledge model of a tool M3: Elements of the knowledge model of a tool that are imported from RDF(S) intro another tool Identification of the evaluation elements and metrics GQM paradigm: Any software measurement activity should be preceded by: 1.- The identification of a software engineering goal ... 2.- ... which leads to questions ... 3.- ... which in turn lead to actual metrics. To improve the interoperability of ontology development tools using RDF(S) as an interchange language

  17. Export experiments Import experiments Evaluation infrastructure Ontology Comparer Ontology Repository Ontology Loader Export Benchmark Suite Executor Ontology Storer Ontology Development Tool Export Benchmark Suite Import Benchmark Suite Executor Import Benchmark Suite RDF(S) File Repository RDF(S) Comparer

  18. Concept 1 Concept 2 Concept 3 Concept 4 <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"> <rdfs:Class rdf:about="http://www.pru.com/ontology#concept1“ /> <rdfs:Class rdf:about="http://www. pru.com/ontology#concept2"> <rdfs:subClassOf rdf:resource="http://www. pru.com/ontology#concept1"/> </rdfs:Class> <rdfs:Class rdf:about="http://www. pru.com/ontology#concept3"> <rdfs:subClassOf rdf:resource="http://www. pru.com/ontology#concept1"/> </rdfs:Class> </rdf:RDF> YES = ? <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"> <rdfs:Class rdf:about="http://www.pru.com/ontology#concept1“ /> <rdfs:Class rdf:about="http://www. pru.com/ontology#concept2"> <rdfs:subClassOf rdf:resource="http://www. pru.com/ontology#concept1"/> </rdfs:Class> <rdfs:Class rdf:about="http://www. pru.com/ontology#concept3"> <rdfs:subClassOf rdf:resource="http://www. pru.com/ontology#concept1"/> </rdfs:Class> </rdf:RDF> <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"> <rdfs:Class rdf:about="http://www.pru.com/ontology#concept1“ /> <rdfs:Class rdf:about="http://www. pru.com/ontology#concept2"> <rdfs:subClassOf rdf:resource="http://www. pru.com/ontology#concept1"/> </rdfs:Class> <rdfs:Class rdf:about="http://www. pru.com/ontology#concept3"> <rdfs:subClassOf rdf:resource="http://www. pru.com/ontology#concept1"/> </rdfs:Class> </rdf:RDF> NO Export experiments Ontology Repository Ontology Loader Ontology Development Tool Export Benchmark Suite RDF(S) Comparer RDF(S) File Repository

  19. YES = ? Concept 1 NO Concept 1 Concept 1 Concept 2 Concept 3 Concept 4 Concept 2 Concept 2 Concept 3 Concept 3 Concept 4 Concept 4 <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"> <rdfs:Class rdf:about="http://www.pru.com/ontology#concept1“ /> <rdfs:Class rdf:about="http://www. pru.com/ontology#concept2"> <rdfs:subClassOf rdf:resource="http://www. pru.com/ontology#concept1"/> </rdfs:Class> <rdfs:Class rdf:about="http://www. pru.com/ontology#concept3"> <rdfs:subClassOf rdf:resource="http://www. pru.com/ontology#concept1"/> </rdfs:Class> </rdf:RDF> Import experiments Ontology Comparer Ontology Repository Ontology Storer Ontology Development Tool Import Benchmark Suite RDF(S) File Repository

  20. WE ARE HERE Experimentation in the benchmarking • Agreement phase.The quality of the benchmark suites is essential for the results. • Evaluation phase 1.The RDF(S) importers and exporters of the ontology development tools are be evaluated. • Evaluation phase 2.The ontology exchange between ontology development tools is evaluated.

  21. Interoperability Working Days October 10th-11th, 2005 Benchmarking the Interoperability of Ontology Development Tools Raúl García-Castro <rgarcia@fi.upm.es> October 10th, 2005

More Related