Storing evaluation models, criteria, and metrics

1. Storing evaluation models, criteria, and metrics CTSA Ontology Workshop, Orlando, FL 2013-02-11 Dagobert Soergel Department of Library and Information Studies Graduate School of Education University at Buffalo

2. Prolog.Integrated Information Infrastructure Information system to support many CTSA functions • support for discoveringspecific expertise, collaboration opportunities, data sets, specimens, apparatus, software, . . . ; • support for research: managing clinical studies, data analysis • providing a program income system for resources and servicesand grant budget management • managing the many interdependent activities and deadlines (project management); • tracking, assessment, and evaluation.

3. Prolog.Integrated Information Infrastructure Needs many types of data, just some examples • many kinds of people, their expertise, projects, publications, and groups to which they belong; • resources, such as animal models, reagents, cell lines, data sets, core facilities; • types of research supported, instances of research supported, users, publications; • training opportunities, their objectives, content, trainee participants, mentors, evaluation.

4. Prolog.Integrated Information Infrastructure Communicating Oracle database and triple store Open: Tables, triples, queries can be added freely Key: Everyone must use defined ontologies – entity types, relationship types, entity values Oracle tables Oracle queries Triple store SPARQL queries

5. Storing a logic model A logic model is a set of assertions of the form Factor/variable/construct A <hasEffectOn> (Factor/variable/construct B, EffectDirectionAndSize) Example UsingStandardOntologies<hasEffectOn> (DataIntegrability, +5) DataIntegrability <hasEffectOn> (DrugDiscovery, +3)

6. Storing milestones Data to be stored about milestones. Examples Milestone <occursOnDate> Date (Date can be recurring) Milestone <hasCriterion> (CM, Value) CM = Construct or Metric Milestone <hasResponsiblePerson> Person Milestone <hasNotificationSpan> Duration

7. Storing data on criteria and metrics In evaluation it may make sense to distinguish between • evaluation criteria or constructs, for example research productivity and • metrics that operationalize these constructs, for example% proposals funded as is commonly done in the social sciences. On the other hand, the boundary may not be sharp.I will use CMto designate any entity that might be considered a construct or a metric.

8. Storing data on criteria and metrics A CM is a property of some entity. The value of a CM can be derived from data through a specified procedure and/or computed from other CMs. Values can be given on a nominal, ordinal, ratio, or interval scale or they may be simply a list of accomplishments, such as a list of three drugs discovered in a program over the last three years with some report on the use or potential use of theses drugs.

9. Storing data on criteria and metrics The next slide shows some examples of the data to be stored. One application is an inventory of all evaluation criteria/measures used in the CTSA consortium

11. S T • T

12. Data needed to compute CMs 1 Research integrated with patient care Community engaged in research • # studies in which clinical and research institutions collaborate • Research priorities stated through community engagement Bridging basic►pre-clinical►clinical • Case studies of research collaboration across the entire pipelineGenomics & imaging to patient care Research collaboration across disciplines and institutions. Team Science • Social Network analysis: Co-researchers, co-authors, members on review and study support teams • Multicenter trials • Institutional & corporate collaborations

13. Data needed to compute CMs 2 Research productivity and quality • % proposals funded • # pilot studies to funded studies • % studies meeting accrual goals • Usage of services, by type • # % of researchers using services • # publications and their impact Timeliness of research. Time from • IRB submission ► approval • grant award ► study opening • study completion ► publication • study completion ► application • #studies moving from one phase to next

14. Data needed to compute CMs 3 Improved data sharing, reusability, and security • % studies with excellent data security • % studies using standard ontologies Trainees educated for research careers • # participants in educational events • Career trajectory (i.e., career choices and success of trainees) • publications and funding of trainees

15. Data needed to compute CMs 4 Research findings relevant to prevention and treatment • Inventory of research findings and their translation potential Changes to practice guidelines • Inventory of changes based on findings of BTC research and their significance New or modified diagnostics, therapeutics, and prevention and their actual use: - new drugs / # NDAs / # INDs, - improved/new uses of drugs, - new devices, - patents - new procedures • Inventory with indication of significance Improved health in Buffalo region • Public health measures, especially as applied to underrepresented groups

16. The end

17. Storing data on criteria and metrics Definitions (sort of) In the context of survey research, a construct is the abstract idea, underlying theme, or subject matter that one wishes to measure using survey questions. Some constructs are relatively simple (like political party affiliation) and can be measured using only one or a few questions, while other constructs are more complex (such as employee satisfaction) and may require a whole battery of questions to fully operationalize the construct to suit the end user's needs.