Mobile Multilingual Maintenance Man

1. 4 M Mobile Multilingual Maintenance Man

2. 4M Assists a Service Person

3. Problem situation Possible solution Problem solved 4M supports Problem Solving Plan Consult Do it Report • Multilinguality • Speech recognition • Dialogue anagement • Support for reporting • Accumulation of knowledge • Message understanding Service Center • Situational recognition • Search for information • Presentation of instructions • Creation of basic knowledge

4. 4M Clients INPUT ANALYZER OUTPUT GENERATOR ONTOLOGY SERVICES DIALOGUE MANAGER Ontology repository 4M Architecture Desktop clients (text, speech, graphics) Mobile clients (menus, speech) Speech recognition on either client or server side Speech synthesis on either client or server side 1 12 4M Server Interface Conceptual system response 2 11 9 3 4 10 5 8 Conceptualized user input Verbalized system response FACTS REASONER 6 7 Model-based Tool Case-based Tool Human Assistance Information Retrieval Asynchronic query to tools Selected tool response

5. Text (documentation, emails etc.) INFORMATION PROCESSING DIALOGUE RDF RULES ONTOLOGY PROCESSING Segmentation Annotation Designer Compiler TriplEd INPUT ANALYZER FACT FINDING & REASONING TOOLS ONTOLOGY SERVICES OUTPUT GENERATOR DIALOGUE MANAGER 4M System Dialogue Planner Input Module RDF/XML Generator Parser Response Evaluator Dialogue Context Report Generator Concept Analysis Discourse Memory Reasoner Interface Ontology repository Information Retrieval … Model-based Human Assistance Case-based DATA SOURCES External systems 4M data Text/XML files External systems and data 4M Knowledge Accumulation

6. CoGKS assists a Service Community with 4M

7. The Cognitive Guidance and Knowledge System (CoGKS) CoGKS Clients Desktop clients (text, speech, graphics) Mobile clients (text, speech) Speech, text CoGKS Server Establish rooms Access manager Query and copy rooms Room manager Room (Job) Invite participants Summarization of cases Chat Summary Attached data Import supporting material 4M System Active or passive assistance mode Consult participants and solve problems in cooperation Variety of reasoning: Model, Case, IR, Human, … Cumulative knowledge base Consult 4M Summarization of cases

8. 4M Ontology: a multilingual ontology matrix company instance bases

9. 4M Dialogue Manager Reasoner Data Interfaces in Uniform RDF • Blackboard Data Architecture. • Reasoner Abstraction • Broadcast Messages Constructive Dialogue Management (CDM) Input Analysis and Natural Language Generation in 4M Multilingual Agenda Markup Language. Agent Based Dialogue Model • Specifying Query Selection • Weighted Rules for Output Responses • Dialogue Planning Utilizing Dialogue Objects

10. Constructive Dialogue Management theory (CDM) (Jokinen, 1996) Human and Machine Participate in Ideal Cooperation (Allwood, 1976) strive to achieve the same purposes cognitively and ethically consider each other in trying to achieve these purposes trust each other to act according to the above principles 4M Dialogue Manager

11. Agent Based Dialogue Model • Specifying Query Selection • Weighted Rules for Output Responses Q1 “Is the power switch off?” DM Q2 Q3 Question Pool Sorted by Probability: • 0.21: [pwr1] [may_be] [off] • 0.08: [inet1] [may_be] [down] • 0.04: [hub1] [may_be] [broken] 4M Dialogue Manager

12. Reasoner Data Interfaces in Uniform RDF • Blackboard Data Architecture – All current data is in messages. • Reasoner Abstraction – New reasoners can be added easily. • Broadcast Messages – Messages are sent to all reasoners at once. Collected facts: • [network] [is_not] [accessible] • [eth_cable] [is_not] [loose] • [pwr1] [may_be] [off] • [hub1] [may_be] [broken] DM R1 4M Dialogue Manager R3 R2 • Model Based Reasoning • Case Based Reasoning • Information Retrieval

13. Input Analysis and Natural Language Generation in 4M • Multilingual Agenda Markup Language DM “Is the power switch turned off?” “Cable is not loose.” IA Text > RDF NLG RDF > Text 4M Dialogue Manager

14. Code switching with a multilingual grammar

15. Commutative graph rewriting Hello

16. Concept Analyser Connects dialogue references with the things they refer to: Keeps track of what the current conversation is about: Ontology This conversation is about: 1. Calendars 2. Mobile phones 3. Humans 4. Poodles 5. Computers Mr. Vertti Hiiri User said: “Vertti cannot access his calendar.” Vertti's calendar program

17. industry project company factory/office extendable data schema base ontology extended ontology instance database case base common query-language shared database platform dialogue- manager life-cycle supporting tasks concept analysis planning planning diagnostics common protocol extendable set of reasoners ontology- reasoner model based reasoner case based reasoner Reasoning methods and techiques Reasoners

18. Ontology based diagnostics • Loads in the conceptual ontology and the instance base describing the system being diagnosed. • Used domain independent search methods for localizing the faults in the system. • Uses the incremental set of observations reported by users of the functionality of the system.

19. Stepwise fault recognition • Observation O1: Mary can't access team calendar. • O2: Jack can't access project directory. • Can Jack access team calendar? • O3: Yes • Fault candidates (O1 and O2 but not O3 )

20. ... U: m4:Internet, m4:Broken S: m4:Mouse U: m4:Access, m4:Net S: m4:Mouse, ... ... Identify parts like subject, verb and object in dialogue turns. Identify known concepts: solve ambiguities and differences in terminology, cross the language barrier Sometimes you may encounter problems with Internet access, or with input devices like keyboards and mice – or with just about anything. By simply following these trouble- shooting instructions, Sometimes you may encounter <c id = "m4:Problem"> problems </> with <c id = "m4:Internet"> Internet </> <c id = "m4:Access"> access </>, or with <c id = "m4:Input"> Information Retrieval Unannotated dialogue Annotated dialogue Input analysis ... User: Internet on rikki System: Toimiiko hiiri? User: En pääse nettiin System: Pääseekö hiiri nettiin? User: ... ... Information retrieval Ontology Build a query out of the concepts in the dialogue. Retrieve best matching documents List known concepts and their relations to each other and to terms in each language Mark occurrences of known concepts with BRIEFS ontology matching Unannotated documents Document analysis Annotated documents

21. Case Based Reasoning Tool • Input: a problem description as a set of RDF triplets • Target state • Observations • Case base: example problems with their solutions as xml documents • The problem is matched against the cases in the case base • The cases with similarity measure over a similarity threshold are retrieved, and their solutions are returned in similarity order • If none of the library cases is similar enough compared to the current case, the solutions of the most similar cases are used in creating further questions to the user • The similarity calculation between two cases uses ontology to determine semantic similarity

22. Human Assistance Tool • The Human Assistance Tool finds experts, administrators, and contact information according to ontology and instance base • Explicit user queries and queries initiated by Dialogue Manager: "Who is the expert of Windows", "Who is the administrator of server S1" • Passive queries representing the cumulative topic of conversation (the concepts found in the discussion) used for team building • locate all the persons in the instance base with some kind of knowledge in the conversation topic, both experts and administrators are considered • search for the people who are closest matches to the minimum spanning tree representing the topic

23. CPSL Ontology RoolTool rules Briefs Annotated document Document 4M: Annotating documents to improve IR