Ontology Assessment – Proposed Framework and Methodology

1. Ontology Assessment – Proposed Framework and Methodology

2. Biological Classification Scheme AMS Classification Scheme NASA Thesaurus Library of Congress Subject Headings Dublin Core Metadata Scheme Organizational Chart ISO Country List Metadata repository scheme Master Data Repository Content architecture models (OO models) SCORM XML Schema for Directory Records Classification Scheme Social Network Representation Folksomony Domain Knowledge Map Visual representation of concept clusters Financial ratios Economic indicators Mathematical formula XML structured electronic journal issue WordNet Which one is an ontology? Why/not? Why do we care whether one is or isn’t an ontology? What’s the point?

3. Personal Recommendation • We need to distinguish between an ontology and applications that use ontologies • We need to suspend our heavy reliance on different domain terminologies that describe applications that use ontologies and adopt a neutral mental model • Minimize our references to dictionaries, synonym rings, taxonomies, thesauri, class schemes, knowledge maps, content schema, etc. • Rather, compare the specific applications and standards to a neutral framework – this will facilitate more intelligent conversations, and will also help us to better communicate with others outside the field • Framework must reflect the multi-dimensionality of ontologies, though – a single, linear representation of applications does not serve as a framework

4. Goals of the Framework • Before we can do this, though, we need to explicitly agree on the ‘end game’ of a the framework • Do we agree that the goal is to develop a neutral, well defined, quantifiable, multidimensional framework against which any ‘thing’ that any one is calling an ontology could be evaluated? • Anyone who has anything they’re calling an ontology should be able to use the framework to judge: • whether it is or is not an ontology • Which essential components it is missing • Where it ranks on a scale of informal to formal ontologies • What they can do to improve or enhance it • Enable assessment of any proposed ontology for the purpose of • informing users about an ontology • Providing developers with methodology for comparison and improvement • Enable definition of: • Minimum standards for what is/is not an ontology • Thresholds for formal and informal ontologies

5. First Steps • If we agree on the goal, we need to start by defining the basic dimensions of an ontology. • Dimensionality proposed in the framework includes: • Structure • Expressiveness • representational granularity • intended use • automated reasoning • descriptive/prescriptive • design methodology • Do the proposed framework dimensions accomplish this goal? • Are these theoretical or practical dimensions? • Do they work at a representational or on an analytical level? • How easy would it be for people who are developing ontologies to understand them? • Do they allow everyone who is working in an ontology space to play, or do they automatically exclude some? • Do they support the ‘end game’ of communication and use?

6. Framework Recommendations • I would suggest that the framework still requires both simplification and elaboration • Simplification in terms of how it groups factors, and elaboration in terms of coverage of the factors that matter to those who are developing or using ontologies • ‘Pre-tested’ the framework with some colleagues in different areas of responsibility – none of them could understand the framework because it was too theoretical • Need to bring it to a practical level • Need to describe the dimensions in terms of formal lists of factors and concrete definitions for those factors • As we develop the framework, we must also define the analytical methods • The test of the framework is our ability to leverage it as an analysis method that allows us to neutrally characterize any ‘thing’ as an ontology and to be able to explain the characterization so that anyone can understand • Suggest that we should consider using a simple factor analysis for representation and analysis

7. Simplify the Framework • I would suggest that the following framework accomplishes this ‘end game’ more effectively • Concepts – the nature of the content or values that are delivered or accessed through the ontology such as type, granularity, etc. • Relationships – nature, type, extent, specification of relationships, logic associated with relationships • Context – the context for which the ontology was developed and in which it may be used, including knowledge domain, application domain, • Governance – control and management of the concepts, relationships and context exercised by the developer or current user • Dimensions are orthogonal but yet sufficiently well defined that they allow us to include factors which are important to different kinds of ontology applications

8. Factor Analysis • Statistical method used to describe variability of factors in which the factors are modeled as linear combinations. • A single factor in the model would represent a set of ‘like’ variables which otherwise would be too complex to model • Factor analysis might help us to synthesize a set of variables into a single factor – to represent in this case a dimension of an ontology • Challenges: • Agree on dimensions (synthesis of factors) • Develop a method for quantifying factors appropriate to the dimension • Define the method of factor analysis • Advantages: • it might help us to focus our discussions on actual factors and away from argumentation • Allows everyone and anyone to play in the ontology space • Allows everyone and anyone to characterize their ontology as a starting point for conversations and interoperability • We can keep the analysis simple since we are only using this to ‘characterize’ and ‘communicate’ – not to predict or to explain factors and

9. Example of Factor Analysis Methodology is currently used to calculate and visually display factors which Contribute to the development or knowledge economies. Helps economists to compare and define knowledge economies.

10. Sample List of Innovation Factors • Let’s take as an example the ICT factor as it relates to knowledge economies • What factors might define the ICT Dimension? • Access to computers • Telecommunications development • Level of education achieved • Investment in technology development (Tech R&D)

11. Proposed Ontology Assessment Methodology • Factor analysis for ontologies would involve … • defining the essential dimensions of an ontology • defining those factors which characterize each dimension • quantifying the factors • analyzing the factors for any given application (factor analysis) or comparison • visually representing the analysis for a single ‘ontology’ and/or for comparisons of ‘ontologies’ • Let me explain how factor analysis might be used • If we can define the dimensions of an ontology, each dimension could then be represented as a composite measure • The composite measure is made up of scores for a set of factors that define that dimension • Having a composite score for each dimension would allow us to use a very simple analytical method that would characterize or compare specific ontology applications

12. Representation of Ontological Assessments Another Dimensionality Framework Dimension 1 Index of Factors Dimension 3 Dimension 2 Index of Factors Index of Factors Index of Factors Dimension 4 Methodology could be used to generate a factor index for ontologies, to rank and compare ontologies.

13. Factor Analysis • Factor analysis could be conducted: • At the component level on that subset of factors • At the ontology level, across all factors • Developers or users could determine what the optimal dimensionality was for their particular use • Summit members and the Ontology community could identify minimun factor scores that define what is/is not an ontology, and what constitutes a full, formal ontology • Ultimately, this may provide us with an ecumenical vs. evangelical approach to ontological standards development and assessment

14. Representation of Ontological Assessments Dimensionality Suggested in the Framework Paper Structure Expressiveness Intended Use Represetational Granularity Use of Automated Reasoning Descriptive vs. Prescriptive Critical Question: Are these dimensions orthogonal, mutually exclusive and clean enough for analysis?

15. Representation of Ontological Assessments Another Dimensionality Framework Relationships Concepts Context Governance Methodology could be used to generate an ontological factor index for ontologies, and to rank and compare ontologies.

16. Representation of Ontological Assessments Sample assessment of a folksonomy Relationships Context Concepts Governance Methodology could be used to generate an ontological factor index for ‘ontological things’, and to rank and compare ontologies.

17. Representation of Ontological Assessments Sample assessment of a medical disease classification scheme Relationships Context Concepts Governance

18. Representation of Ontological Assessments Sample assessment of an institutional records classification scheme Relationships Context Concepts Governance

19. Defining and Quantifying Factors • For each component an orthogonal, independent set of factors must be defined • Factors must be independent of any particular pre-existing ontology (neutral) • Each factor must have a quantifiable method of representation that lends itself to ‘scoring’, analysis and comparison • Factors must have agreed upon definitions, be easily interpreted by people and machines, and be inclusive in their coverage of values/conditions • To illustrate the idea, selected examples are presented in following slides

20. Concept types Data/numbers Calculation/ratios Words Grammatical fragment Logical statement Rule expression Engineering equations Degree of ambiguity Context sensitivity/insensitivity of definition Representational form Usable encoding method Availability of representational specifications (Strings vs. syntax) Degree of conceptualization/ specification Theoretical to commital What else…? Selected Examples of Concept Factors

21. Simple expressive form of relationships Grammatical Mathematical Logical Relationship behavior Membership dependence Representation or instance Equivalence Causal dependence Derivational dependence Degree of Relationship Validation/Rigor Fully Subjective Grammatical validation Mathematical validation Logical rigor/validation What else? Selected Examples of Relationship Factors

22. Knowledge Context Formal vs. informal knowledge domain Application Context System vs. human application/ consumption Managed/standardized application vs. home grown Functional context Search Mathematical or statistical analysis Logical inference Classification Dynamic clustering Metadata representation Concept indexing What else…? Selected Examples of Context Factors

23. Standards Availability Published formal vs. guidelines vs. ad hoc concepts Published formal vs. guidelines vs. ad hoc relationships Prescriptive vs. Descriptive Governance Enforcement of standards Design Guidelines Top-down (model) vs. Bottom-up (empirical) Extensibility Degree to which others can add to or extend either the concepts or the relationships Currency Degree to which the concepts and/or relationships represent our current view or knowledge of the context What else…? Selected Examples of Governance Factors