semantic models in healthcare education what is it and how it can improve formative assessments n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Semantic models in healthcare education What is it and how it can improve formative assessments PowerPoint Presentation
Download Presentation
Semantic models in healthcare education What is it and how it can improve formative assessments

Loading in 2 Seconds...

play fullscreen
1 / 32

Semantic models in healthcare education What is it and how it can improve formative assessments - PowerPoint PPT Presentation


  • 85 Views
  • Uploaded on

Semantic models in healthcare education What is it and how it can improve formative assessments. MedBiquitous Annual Conference 2012 May 2-4 2012 - Baltimore, MD Muriel Foulonneau Younes Djaghloul Raynald Jadoul Nabil Zary. 1. Context. 2. OAT approach. 3. Experimentation. Challenges

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Semantic models in healthcare education What is it and how it can improve formative assessments' - zeph-hill


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
semantic models in healthcare education what is it and how it can improve formative assessments

Semantic modelsin healthcare educationWhat is it and how it can improve formative assessments

MedBiquitous Annual Conference 2012

May 2-4 2012 - Baltimore, MD

Muriel FoulonneauYounes DjaghloulRaynald JadoulNabil Zary

slide2

1

Context

2

OAT approach

3

Experimentation

  • Challenges
  • Research questions
  • OVACS
  • AIGLE
  • TAO
  • The ontology of carieswith Karolinska institutet
slide3

1

Context

2

OAT approach

3

Experimentation

  • Challenges
  • Research questions
  • OVACS
  • AIGLE
  • TAO
  • The ontology of carieswith Karolinska institutet
challenges
Challenges
  • Two main challenges:
  • Item variability in an assessment
    • generate items from a model in order to avoid repeating items
    • save time and resources, as assessment resource creation is a time and resource consuming activity
  • Learning adaptivity
    • adapting question forms or assessment path in formative assessment according to candidate answers or profile
  • We strive toward: Efficient Approach to Automate/Assist the generation of assessment resources.

OVACS-AIGLE-TAO

how the challenges are addressed
How the challenges are addressed
  • Knowledge sources
    • Expert
    • Social / Crowd sourcing
    • Repository
    • Textual

How ?

  • Question generation
    • Keep the initial semantic
    • Semantic Inference
    • Adaptivity

Assessment resources

OVACS-AIGLE-TAO

in summary the goal was to
In summary the goal was to
  • Enable the automatic generation assessment questions based on formal models of knowledge
  • Knowledge oriented approach based on semantic technologies:
    • The creation of a streamline exploring the use of semantic technologies for e-assessment
    • Semantic for model checking
    • Semantic for inference ( to discover knowledge)
  • Needs to have models with formal representation (such as RDF)
  • Four questions
    • How to build a domain model?
    • How to validate the proposed model by non IT expert ?
    • How to generate assessment questions from the refined model?
    • How to build a flexible delivery environment for these questions?

OVACS-AIGLE-TAO

slide7

1

Context

2

OAT approach

3

Experimentation

  • Problem statement
  • Research questions
  • OVACS
  • AIGLE
  • TAO
  • The ontology of carieswith Karolinska institutet
slide8

The vision

OVACS-AIGLE-TAO

overview on approach
Overview on approach

Experts, repositories, social media

The Final test

Knowledge: Informal models

  • 4.Delivery strategy
  • TAO Delivery Module
  • TAO QTI viewer

1.Model building: data mining, human methodology

Formal but not validated

Validate questions

Experts for question validation

Knowledge: Formal models

List of assessment questions

  • 2.Model validation
  • Experts for model validation
  • OVACS : to assist experts and to hide the complexity of he formalism (OWL, description logic )

Final

Ontology

  • 3.Question generation
  • AIGLE tool, Automatic QTI based questions generation
  • Semantic similarity techniques

OVACS-AIGLE-TAO

the process
The process

OVACS-AIGLE-TAO

origins of oat
Origins of OAT

OVACS-AIGLE-TAO

slide12

OVACS

  • Ontology VAlidation for Common uSers
  • How to validate formal knowledge model by questions

OVACS-AIGLE-TAO

ovacs what
OVACS: what ?
  • Question based strategy for validation
    • Question to for the validation of the domain not for the assessment
    • Generate question based on existed knowledge element ( automatic)
    • More simple for the expert than modifying formal model ( OWL )
  • Four ontological components (OC) to validate (RDF schema)
    • Instance of
    • All property value
    • Sub class
    • Property of a class
  • 12 types of feedback
    • For each OC Accept, remove, don’t Know
  • Templates for textual question
    • Generic (Subject, Predicate, Object)
    • Dedicated

OVACS-AIGLE-TAO

ovacs architecture
OVACS architecture

Source Ontology

OWL

Validated ontology

OVACS Engine

(Semantic web technologies)

Evaluated ontology

Ontology of management

feedback

Generated Question

(Web based)

  • Manage history
  • Get past questions

Expert feedbacks

OVACS-AIGLE-TAO

ovacs interface
OVACS interface

http://crpovacscaries.elasticbeanstalk.com/

OVACS-AIGLE-TAO

slide16

AIGLE

  • AssessmentItemGeneratorinLearningEnvironment

OVACS-AIGLE-TAO

aigle assessment item generator
AIGLE – Assessment item generator
  • Security issue (variability)
  • Adding variability to an item
  • no expected variation of the construct
  • Model-based learning (adaptivity)
  • Generating items from knowledge represented as a model
  • the construct is modified for each item

Stem variables

Auxiliary information

Options

Key

OVACS-AIGLE-TAO

ims qti item generation process
IMS-QTI item generation process
  • Generating items from Web data sources

OVACS-AIGLE-TAO

calculating the semantic similarity between distractors and the correct answer
Calculating the semantic similarity between distractors and the correct answer

Gabon -- Libreville

No SemSim

With SemSim

Adapted 3 semantic similarity strategies to large scale semantic graphs

OVACS-AIGLE-TAO

results of user test
Results of user test
  • Clear decrease of performance in the population when using SemSim (optimizing the similarity between the correct answer and the distractors)

OVACS-AIGLE-TAO

slide22

TAO

  • TestingAssisté parOrdinateur
  • (Computer-Aided Testing)

OVACS-AIGLE-TAO

tao assessment and feedback loop
TAO – assessment and feedback loop
  • The TAO platform is based on semantic web paradigm, i.e. it manages question items decorated with any needed ad-hoc properties
  • The TAO platform delivers questionnaires that can also be featured with any desiderated extra semantic properties
  • The TAO collects all answers and behaviors of the test-takers
  • If extra properties like the “provenance” (i.e. the source model built with OVACS and used by AIGLE) are attached to the question items or to the questionnaire, these properties are stored in tests results
  • The analysis of the tests results will enforce be used by as feedback loop for a validation process impacting the AIGLE & OVACS phases.

OVACS

AIGLE

TAO

OVACS-AIGLE-TAO

slide24

1

Context

2

OAT approach

3

Experimentation

  • Problem statement
  • Research questions
  • OVACS
  • AIGLE
  • TAO
  • The ontology of carieswith Karolinska institutet
original hypothesis
Original hypothesis
  • The creation of the domain ontology can use semi-automatic strategies, or third party encoders, or a collaborative work: can we ask an expert to validate the assertions in the ontology?
  • What is lost in the expert’s speech when creating the ontology?
  • Does the expert understand automatically generation questions?
  • Does the expert flag the errors?

OVACS-AIGLE-TAO

creating the ontology
Creating the ontology
  • An ontology of the caries
  • A one hour interview where the teacher explained the caries, their description, their causes, how to handle them, how to prevent them, how to set a diagnostic
  • Definition of a list of concepts / keywords
  • Creation of classes, instances, and properties
  • Creation of the OWL ontology

OVACS-AIGLE-TAO

test set up
Test set up
  • Labels on stand alone
  • Selected a subset of the ontology to keep the test short:instanceOf (13 items) and subClassOf (11 items)
  • Only Boolean questions + “I do not know” option
  • 24 questions
  • 2 intentional mistakes: on the content (causes of caries) and spelling (emanel instead of enamel)
  • Objective:
  • verify whether the teacher would find the validation mechanism usable
  • Verify whether errors would be detected and corrected
  • Video recording of the teacher

OVACS-AIGLE-TAO

ovacs interface1
OVACS interface

http://crpovacscaries.elasticbeanstalk.com/

OVACS-AIGLE-TAO

test conclusions
Test conclusions
  • Confusion between
  • the role of domain expert validating knowledgeandthe role of teacher who prepares questions for students
  •  Objective was not well understood  rework experiment conditions
  • According the comments of our expert:
  • “Difficulty level of the generated questions is generally low”
  • “But with very different variations in the difficulty level”
  • The OVACS validation questionnaire led to:
  • 6 removals (2 subClassOf, 4 instanceOf)
  • 16 accept (9 for subclassOf, 7 for instanceOf)
  • 2 answers “I do not know” for subclassOf meant not relevant

OVACS-AIGLE-TAO

next steps
Next steps
  • OVACS
  • Enrich collaborative features
  • AIGLE
  • Ensure a validation / feedback on the generated items
  • AIGLE generates distractors from an open model (large dataset from the Web) using semantic similarity, but needs to identify relevant distractors in the case of a bounded model (in this case a model for caries)
  • Predicting item difficulty? Initial test for general culture questions using a Web mining approach. Would need to be tested for medical knowledge.

OVACS-AIGLE-TAO

slide32

Thanks a lot!

http://tao.lu

OVACS-AIGLE-TAO