its data collection framework n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
ITS Data Collection Framework PowerPoint Presentation
Download Presentation
ITS Data Collection Framework

Loading in 2 Seconds...

play fullscreen
1 / 29

ITS Data Collection Framework - PowerPoint PPT Presentation


  • 94 Views
  • Uploaded on

ITS Data Collection Framework. Capturing data based on agent communication standard Olga Medvedeva , Center for Pathology Informatics, University of Pittsburgh. Outline. Need for communication standard for Intelligent Tutoring Systems Existing standard for multi-agent communication

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'ITS Data Collection Framework' - lamont


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
its data collection framework

ITS Data Collection Framework

Capturing data based on agent communication standard

Olga Medvedeva,

Center for Pathology Informatics,

University of Pittsburgh

Educational Data Mining Workshop

20th AAAI-05 Conference

outline
Outline
  • Need for communication standard for Intelligent Tutoring Systems
  • Existing standard for multi-agent communication
  • Implementation in SlideTutor
    • Communication protocol
    • Data collection
    • Database query tool
    • Lessons learned
  • Comparison with recent standardization effort
  • Advantages of using the the existing standard

Educational Data Mining Workshop

20th AAAI-05 Conference

intelligent learning environment common base
Intelligent Learning Environment Common Base
  • Underlying theory
    • Cognitive tutors (Anderson et al. 1995)
    • Adaptive hypermedia (Brusilovsky et al. 1996)
    • Constraint-based (Mitrovic et al. 2001)
  • Modules
    • Expert, Student, Interface, Pedagogic
  • “Single-purpose” development approach

Educational Data Mining Workshop

20th AAAI-05 Conference

keystone communication standard
Keystone – communication standard
  • Previous efforts:
    • Inter-tutor communication (Ritter, Koedinger 1996; Brusilovsky et al. 1997) one-to-one translators, strict channel, no real protocol
    • Shared resources (Koedinger et al. 1999) – limited use: lack of standard
    • DORMIN protocol (developed at CMU) – used in commercial product
  • Our approach
    • Multi-agent technology
    • Use existing inter-agent communication standard

Educational Data Mining Workshop

20th AAAI-05 Conference

foundation for intelligent physical agents fipa
Foundation for Intelligent Physical Agents (FIPA)

FIPA (www.fipa.org) - collection of standards for

inter-agent communication:

  • Agent Management System – manages an agent life-cycle, maintains a registry with unique Agent Identifier (AID)
  • Transport – describes message exchange protocol: transport type and specific address for an agent
  • Agent Communication Language (ACL) –communication specifications

FIPA was officially accepted by the IEEE as one of its standards committees on 8 June 2005

Educational Data Mining Workshop

20th AAAI-05 Conference

fipa design principals
FIPA Design Principals
  • Forms abstract basis for concrete architecture
  • Sets minimum required elements
  • Permits introduction of new elements
  • Permits arbitrary content language, uses Abstract Content Representation (ACR) for ACL as key-value pairs

Envelope:

Sender (locator)

Receiver (locator)

Timestamp

Message (ACL):

Sender (AID)

Receiver (AID)

Performative (String)

Content: ( ACR)

Reply-to(Message ID)

Message (ACL):

Sender (AID)

Receiver (AID)

Performative (String)

Content: ( ACR)

Reply-to (Message ID)

Educational Data Mining Workshop

20th AAAI-05 Conference

fipa acl message structure
FIPA ACL Message Structure

:sender – identity of the sender

:receiver – identity of the recipient

:content – the object of the action

:performative – the type of the communicative act

Optional:

:reply-with:replay-to:in-replay-to:replay-by– replay constraints

:language – encoding schema of the content of the message

:encoding – encoding identifier

:ontology – is used to give a meaning to symbols/concepts in the content

:protocol – gives additional context for the interpretation of the message

:conversation-id – identifies the ongoing sequence of communicative act, manages the conversation strategies

Educational Data Mining Workshop

20th AAAI-05 Conference

fipa performatives
Accept-proposal

Agree

Cancel

Call-for-proposal

Confirm

Disconfirm

Failure

Inform

Inform-if

Inform-ref

Not-understood

Propagate

Propose

Proxy

Query-if

Query-ref

Refuse

Reject-proposal

Request

Request-when

Request-whenever

Subscribe

FIPA Performatives

Educational Data Mining Workshop

20th AAAI-05 Conference

fipa implementation in java
FIPA Implementation in Java
  • Java Agent Services (JAS) (www.jcp.org) defines a set of objects and service interfaces to support the deployment and operation of the agents.
  • Contains interfaces for building messages, directory services and a factory for message transfer services.
  • JAS is a base for multi-agent communication in our system

Educational Data Mining Workshop

20th AAAI-05 Conference

slidetutor architecture http slidetutor upmc edu
SlideTutor Architecturehttp://slidetutor.upmc.edu

SlideTutor - an agent-based model tracing ITS for visual classification problem solving insurgical pathology

Educational Data Mining Workshop

20th AAAI-05 Conference

slide11
Educational Data Mining Workshop

20th AAAI-05 Conference

generic representation of problem solving space
Generic Representation of Problem-Solving Space

Educational Data Mining Workshop

20th AAAI-05 Conference

collected data
Collected Data
  • InterfaceEvent – low-level human-computer interactions
  • ClientEvent – collection of InterfaceEvents that represents an elementary subgoal, understood by tutor
  • TutorResponse – system response to a ClientEvent

Educational Data Mining Workshop

20th AAAI-05 Conference

message example
Message Example

ClientEvent

  • Envelope indicates the locators of client and protocol agents
  • 4 required key-value pairs for a message
  • Performative defines a type of communicative act
  • List of preceding InterfaceEvent Ids:
    • click on Finding button
    • Click on image
    • Selecting 3 times down a tree of findings

Envelope:

Sender: Client_1

Receiver: PROTOCOL

TimeStamp = 1114444377783

Message:

Sender: Concept2

Receiver: PROTOCOL

Performative: X-Created

In-reply-with: 1114444378242

Content:

Type = Finding

Label = blister

Id = Concept2

ObjectDescription = Finding.blister.Concept2

Parent = null

Input:

name = text value = blister

name = y value = 11808

name = x value = 38048

name = z value = 0.03

InterfaceEventIDS = [1114444374333, 1114444375546, 1114444376304,

1114444376798, 1114444377444]

Educational Data Mining Workshop

20th AAAI-05 Conference

message in depth
Message in Depth

ClientEvent

  • Widget object (agent) description parameters
    • Type(“Button”, “Finding”)
    • Label(“Next”, “Blister”)
    • Id – unique within a session
    • ObjectDescription – combination of Type+Label+Id (“Finding.blister.Concept2)
    • Parent – list of all parent ObjectDescriptions for hierarchical structures
  • Common for ITS user action triplet
    • Action = Performative
    • Selection = ObjectDescription+Parent
    • Input = list form Content Input
  • Message encoded in XML is easy to translate into other languages (RDF, KIF, SL, etc.)

Envelope:

Sender: Client_1

Receiver: PROTOCOL

TimeStamp = 1114444377783

Message:

Sender: Concept2

Receiver: PROTOCOL

Performative: X-Created

In-reply-with: 1114444378242

Content:

Type = Finding

Label = blister

Id = Concept2

ObjectDescription = Finding.blister.Concept2

Parent = null

Input:

name = text value = blister

name = y value = 11808

name = x value = 38048

name = z value = 0.03

InterfaceEventIDS = [1114444374333, 1114444375546, 1114444376304,

1114444376798, 1114444377444]

Educational Data Mining Workshop

20th AAAI-05 Conference

tutorresponse example
TutorResponse Example
  • Student performance data
    • Performative: FAILURE – user took incorrect step
    • ErrorCode = 15 – user incorrectly located existing finding
    • Input: - contains a description of an error message to be presented to user
  • Tutor performance data
    • Best possible next step – action expert model would take in this problem state

Envelope

Sender: TutorEngine0

Receiver: PROTOCOL

TimeStamp: 1114444379378

Message:

Sender: TutorEngine0

Receiver: PROTOCOL

Performative: FAILURE

Conversation_ID: 1114444378242

Content:

ErrorCode = 15

NextStepType = Evidence

NextStepLabel = blister

NextStepID = Concept2

NextStepParent = null

NextStepAction = DELETE

Input:

name = Messages

value = "[TEXT:There is BLISTER present, but not where

you have pointed in the image. See if you can find

where. POINTERS:[PointTo:Concept2

IsPermanent:false Method:setFlash Args:[true]]]“

name= TutorAction

value = "PointTo:Concept2 IsPermanent:false

Method:setBackgroundColor Args:[RED]"

Educational Data Mining Workshop

20th AAAI-05 Conference

database schema
Database Schema
  • High-level static tables similar to Mostow et al. 2002 contains
  • Experiment, CaseList, Student, etc.
  • Low-level tables for captured events, including start/end of problem and
  • session closely follow the FIPA standard, generic with any number of
  • event parameters stored in corresponding Input tables

Educational Data Mining Workshop

20th AAAI-05 Conference

web based protocol query tool
Web-Based Protocol Query Tool
  • Allows the user to obtain data sets specific to a wide range of constraints
  • Outputs to HTML file that can be transferred to Excel
  • Query can be saved and viewed in SQL
  • Interface, Client and Tutor events data can be joined in different ways

Educational Data Mining Workshop

20th AAAI-05 Conference

query tool results for identifying blister
Query Tool Results for Identifying Blister

InterfaceEvents

ClientEvents

TutorResponses

Educational Data Mining Workshop

20th AAAI-05 Conference

advantages of event based data representation
Advantages of Event-Based Data Representation
  • Usability Perspective: InterfaceEvents linked to ClientEvents (Saadawi et al. 2005)
    • How many actions were performed
    • How much time was required to achieve a particular subgoal, such as identification of Blister
    • How many InterfaceEvents were unrelated to any ClientEvent
  • Student Performance over time: ClientEvents linked to TutorResponses
    • Number of hints requested
    • Depth of hints
    • Error frequency and distribution
  • Tutor Performance:NextStep fields in TutorResponses
    • Compare next student actions to those predicted by tutor

Educational Data Mining Workshop

20th AAAI-05 Conference

slidetutor data sharing limitation
SlideTutor Data Sharing Limitation
  • This paper and presentation have been approved by Institutional Review Board (IRB)
  • Researcher needs to sign a Limited Use Agreement
  • There might be one agreement with consortiums

Educational Data Mining Workshop

20th AAAI-05 Conference

lessons learned
Lessons Learned

For the past year our data collection framework was used in 4 small HCI studies and one large experiment with a total of 50 students.

  • Keep data clean: ended up maintaining ‘raw’ and ‘clean’ copies of database
  • Granularity of captured data: capturing of detailed data slows the system
  • Separate database for assessment: no explicit mapping of performance on tests and in the tutoring system

Educational Data Mining Workshop

20th AAAI-05 Conference

data collection framework advantages
Data Collection Framework Advantages
  • Advantages of relational database (Mostow et al. 2002)
    • Eases the analysis of the enormous volume of complex data
  • Generic framework that might be adapted to other model-tracing ITS
    • Adapted in the extension of SlideTutor – ReportTutor that teaches how to write the pathology reports
  • Flexibility of FIPA-based communication protocol
    • Flexibility to describe interaction events
    • Extendable set of performatives
    • Multiple messages in one envelope, unrestricted number of input parameters
    • Potential to reference ontologies within the message
    • Can be easily reused in the Data Shop

Educational Data Mining Workshop

20th AAAI-05 Conference

data shop project pittsburgh science of learning center http www learnlab org
Data Shop Project, Pittsburgh Science of Learning Center (http://www.learnlab.org )
  • Logging and Analysis: Tools and reports to aid PSLC researchers and course developers
    • Log the activities of the experiments to a database
    • Provide the reports and queries on that experiments
  • Goal: Standardize the messaging format among tools, tutoring translators and agents
    • Message types: tool_message, tutor_message, curriculum_message, message
  • Data Shop Tutor Logging v3 released in June 2005

Educational Data Mining Workshop

20th AAAI-05 Conference

data shop tool message and slidetutor interface client events

meta (0 or 1)

user_id

session_id

time

time_zone

tool_message

attempt_id

event_descriptor (0+)

event_id

selection (0+)

id

type

action (0+)

id

problem_name (0 or 1)

semantic_event

id

semantic_event_id

name

trigger

input (0+)

id

(1+)

step (0+)

probability

ui_event

id

Data Shop Tool Message and SlideTutor Interface/Client events

Educational Data Mining Workshop

20th AAAI-05 Conference

data shop tutor message and slidetutor tutorresponse

meta (0 or 1)

user_id

session_id

time

time_zone

problem_name (0 or 1)

semantic_event

id

semantic_event_id

name

trigger

ui_event

id

action_evaluation (0+)

current_hint_number

total_hints_available

classification

event_descriptor (0+)

event_id

step (0+)

probability

tutor_advice (0+)

selection (0+)

id

type

skill (0+)

probability

production (0+)

step_interpretation (0+)

action (0+)

id

input (0+)

id

custom_field (0+)

name (1)

name (1)

value (1)

value (1)

Data Shop Tutor Message and SlideTutor TutorResponse

Educational Data Mining Workshop

20th AAAI-05 Conference

fipa advantages
FIPA Advantages
  • FIPA as a information exchange underlying standard
    • Develop a set of performatives – a controlled vocabulary for ITS communication
    • Create sharable ontologies for domain knowledge, hint content, error categories and use ‘:ontology’ FIPA parameter to give a meaning to the message content
    • Use ‘:protocol’ parameter to identify the translator and to preserve the internal component structure
  • Syntactically aligned systems
    • Ease meta-analysis for tutors with the identical performatives
    • Reuse data for simulations
    • Shared services for real-time interoperability
      • Identifying particular help-seeking behavior
      • Calculating knowledge tracing probabilities

Educational Data Mining Workshop

20th AAAI-05 Conference

acknowledgements
Acknowledgements

Grants:

  • National Library of Medicine
  • National Cancer Institute

People:

  • Rebecca Crowley
  • Girish Chavan
  • Eugene Tseytlin
  • Elizabeth Legowski
  • Katsura Fujita
  • Maria Bond

Educational Data Mining Workshop

20th AAAI-05 Conference

references
References
  • Anderson JR, Corbett AT, Koedinger KR, and Pelletier R. Cognitive Tutors: Lessons learned. Journal of the Learning Sciences 4(2): 167-207, 1995
  • Brusilovsky, P., Kommers, P. & Streitz, N. (Eds.) (1996) Multimedia, Hypermedia, and Virtual Reality (LNCS Vol. 1077). Berlin: Springer-Verlag, 1996
  • Mitrovic A, Mayo M, Suraweera, P and Martin, B. Constraint-Based Tutors: A Success Story. In Monostori, L. and Vancza, J. (Eds). Proceedings of the 14th International Conference on Industrial & Engineering Applications of Artificial Intelligence and Expert Systems, Budapest, Hungary, Springer, pp. 931-940, 2001
  • Ritter, S. and Koedinger, K. R. (1996). An architecture for plug-in tutor agents. Journal of Artificial Intelligence in Education, 7, 315-347
  • Brusilovsky, P., Ritter, S., & Schwarz, E. Distributed intelligent tutoring on the Web, Proceedings of AIEDâ97, the Eighth World Conference on Artificial Intelligence in Education. 1997
  • Koedinger KR, Suthers DD, & Forbus KD.  Component-based construction of a science learning space: A model and feasibility demonstration.  International Journal of Artificial Intelligence in Education: 10, 392-31, 1999
  • Mostow J, Beck J, Chalasani R, Cuneo A, and Jia P.Viewing and Analyzing Multimodal Human-computer Tutorial Dialogue: A Database Approach. Proceedings of the ITS 2002 Workshop on Empirical Methods for Tutorial Dialogue Systems, 75-84
  • Saadawi G, Legowski E, Medvedeva O, Chavan G, and Crowley RS. A method for automated detection of usability problems from client user interface events. Accepted to Proceedings of the American Medical Informatics Association Symposium 2005

Educational Data Mining Workshop

20th AAAI-05 Conference