1 / 33

Information processing

Information processing. Michal Laclav ík , Ladislav Hluch ý (Email research, information extraction, information retrieval, contextual recommendation). Primary Research Team & Capabilities. URL: http://ikt.ui.sav.sk. Dept. of Parallel and Distributed Computing

wruth
Download Presentation

Information processing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Information processing Michal Laclavík, Ladislav Hluchý (Email research, information extraction, information retrieval, contextual recommendation)

  2. Primary Research Team & Capabilities URL: http://ikt.ui.sav.sk Dept. of Parallel and Distributed Computing Research and Development Areas: • Large-scale HPCN, Grid and MapReduce applications • Intelligent and Knowledge oriented Technologies Experience from IST: • 3 project in FP5: ANFAS, CrosGRID, Pellucid • 6 project in FP6: EGEE II, K-Wf Grid, DEGREE (coordinator),EGEE, int.eu.grid, MEDIGRID • 4 projects in FP7: Commius, Admire, Secricom, EGEE III Several National Projects (SPVV, VEGA, APVT) IKT Group Focus: • Information Processing (Large Scale) • Graph Processing • Information Extraction and Retrieval • Semantic Web • Knowledge oriented Technologies • Parallel and Distributed Information Processing Solutions: • SGDB: Simple Graph Database • gSemSearch: Graph based Semantic Search • Ontea: Pattern-based Semantic Annotation • ACoMA: KM tool in Email • EMBET: Recommendation System • Experts on MapReduce and IR (Nutch, Solr, Lucene) Director & leader of PDC: Dr. Dipl. Ing. Ladislav Hluchý Bratislava, 10th November 2011

  3. Approach and Solutions

  4. Large scale Text and Graph data processing Underlined are the technologies developed by IISAS Core Technology • Web crawling • Nutch + plugins • Full text indexing and search • lucene, Sorl • Information Extraction • Ontea, GATE • All above large scale • Hadoop, S4 • Graph processing and Querying • Simple Graph Database (SGDB) • gSemSearch • Neo4j • Blueprints Bratislava, 10th November 2011

  5. Ontea: Information Extraction Tool http://ontea.sf.net • Regex patterns • Gazetteers • Resuls • Key-value pairs • Structured into trees • graphs • Transformers, Configuration • Automatic loading of extractors • Visual Annotation Tool • Integration with external tools • GATE, Stemers, Hadoop … • Multilingual tests • English, Slovak, Spanish, Italian Bratislava, 10th November 2011

  6. Email Search Prototype • Use of Social Network from email • Includes extracted objects • Full text of extracted objects • Related objects discovered and ordered by spread activation on social network graph • Faceted search, navigation Bratislava, 10th November 2011

  7. gSemSearch: Graph based Semantic Search • Graph/Network of interacting (interconnected) entities • Discovering relation in the Graph (network) using spread of activation algorithm • Showing relations of concrete type, e.g. telephone numbers related to a person • Navigation over related entities • Full-text search of the entities • User interface for search • User interaction with data (merging, deleting entities) with immediate impact on discovered relations • Tested on Email Enron Corpus • Email Social Network Search • http://ikt.ui.sav.sk/esns/ Bratislava, 10th November 2011

  8. SGDB: Simple Graph Database • Storage for graphs • Optimized for graph traversing and spread of activation • Faster then Neo4j for graph traversing operations • Supports Blueprints API • https://simplegdb.svn.sourceforge.net/svnroot/simplegdb/Sgdb3 • Graph Database Benchmarks • Graph Traversal Benchmark for Graph Databases • http://ups.savba.sk/~marek/gbench.html • Blueprints API - possibility to test compliant Graph databases Bratislava, 10th November 2011

  9. Future Direction: Relations Discovery in Large Graph Data • Motivation • Graph/Network data are everywhere: social networks, web, LinkedData, transactions, communication (email, phone). • Also text can be converted to graph. • Interconnecting graph data and searching for relations is crucial. • Approach • Forming semantic trees and graphs from text, web, communication, databases and LinkedData • User interaction with graph data in order to achieve integration and data cleansing • Users will do it, if user effort have immediate impact on search results Bratislava, 10th November 2011

  10. Ontea: Pattern based information extraction and semantic annotation Text processing

  11. Ontea: Information Extraction (Features) • Regex patterns • Visual Annotation Tool • Integration with external tools • GATE, Stemers, Hadoop … • Gazetteers • IE System configuration • Automatic loading of extractors • Patterns • Multilingual tests • Spanish • Slovak • English • Italian Bratislava, 10th November 2011

  12. Information Extraction Model Address and product patterns 3 words macro Extraction ZIP macro Street number macro Street name macro City name macro Country macro Processing Address patterns Bratislava, 10th November 2011

  13. Segmentation • Sentences • Paragraphs • Objects (Address, Product ..) Bratislava, 10th November 2011

  14. Information Extraction: Gazetteers configuration Gazetteer Can extract information, which cannot be properly extracted by regular expression patterns (like given names, product names, etc.) Gazetteer extraction approach is combined with regular expressions based extrac-tion. For example personal full names can be extracted with higher precision. Gazetteer is easy to update, because it is configured by simple text files. Gazetteer configuration simple text file with <list file>:<IE result type> Gazetteer lists simple text files with keywords Information extractor rules Bratislava, 10th November 2011

  15. Information Extraction: Rules configuration IE System configuration • IE dynamically loads and run its components (XMLRegexExtractor, Gazetteer, RuleTransformer) according to setting in IE rules file • IE Components are executing consecutively and operate on a set of information extraction results Regex based IE component Gazetteer IE component Modified IE result set IE result set IE component Result set transformer IE component Information extractor rules file Bratislava, 10th November 2011

  16. Semantic Annotation Theconcept • InformationExtractor - IE produces a set of extraction results • SemanticAnnotator - SA consumes the IE result set and builds a trees convertible to Ontology instances or objects according to XML schema e.g. Core Components • SA first builds an intermediate tree of IE results on which it operates • The tree is upon its creation not compliant to Core Components specification and needs to be transformed • Therefore we have tree transformers which transform the IE result tree to a trees Bratislava, 10th November 2011

  17. Semantic Annotation Tree transformers Input is a tree of IE results and output is the modified tree of IE results Tree transformers are executing consecutively and operate on a tree of information extraction results Tree transformers, which delete, create,rename, move, switch and order nodesare configured in the SA rules file Treetransformer Bratislava, 10th November 2011

  18. Social Networks Social nework reconstruction: • probabilistic inference using spreading activation • relies on the output of the information extractor (IE) in the form of complex objects Preliminary results on a set of 50 Spanish emails (phone/name): • Precision 60% (due to lower recall in IE) • Precision 85% (achievable with better IE) • self-healing (with new incoming emails) Bratislava, 10th November 2011

  19. Social Networks Results as XML or HTML: (via XSL Transformations)Future: • DataSource for Search for Partner module • Improve the recall of Information Extractor • Exploit multi-pass algorithm and named entity recognition: things learned in the first pass will be used in the next, e.g. possible names with initials, etc. • Build an enhanced statistical reasoning procedure on top of the present Social Network Extractor/Correlator Bratislava, 10th November 2011

  20. Email Research Acoma

  21. Acoma Architecture • Connected to email protocols on desktop or server • No need to change working practices • Emails are received and send as before • Received email is processed by Acoma and enriched with useful information • Extensible with OSGi modules Bratislava, 10th November 2011

  22. Web Connector Key-value Meta-Connector SpreadSheet Connector Transformed Key-value Database Connector System Connectors • Connection of Acoma to existing systems • Document Archives • Internet or Intranet Systems • Databases • Access or import of data • Key-value pair transformation Bratislava, 10th November 2011

  23. Acoma architecture : Message Post Processing • Useful hints with links are included in enriched email • Links lead to internal or external systems (Internet, Intranet) Bratislava, 10th November 2011

  24. Study on 6 organizations show: Objects can be identified by patterns and gazeteers It is possible to define set of common objects Objects identified: Organization: org:Name, org:RegNo, org:TaxNo Person: person:Name, person:Function Contact: contact:Phone, contact:Email, contact:Webpage Address: address:ZIP, address:Street, address:Settlement Product: product:Name, product:Module, product:Component, product:BOID Document: doc:Invoice, doc:Order, doc:Contract, doc:ChangeRequest Inventory: inventory:ResID, inventory:ResType Other business object ID: BOID Business objects in Emails Bratislava, 10th November 2011

  25. Social Networks and Graph Data • Relations among objects • Support for search Bratislava, 10th November 2011

  26. Email Search Prototype • Use of Social Network from email • Includes extracted objects • Full text of extracted objects • Related objects discovered and ordered by spread activation on social network graph • Faceted search, navigation Bratislava, 10th November 2011

  27. Context based Recommendation, Knowledge Sharing EMBET, Acoma

  28. EMBET: proactive information and knowledge provision • Collaboration among users • Knowledge sharing • Active knowledge provision • Reuse of knowledge: notes and other resources Objective: Recommend and provide user information or knowledge in context http://ups.savba.sk/kwfgrid/uaa/ Bratislava, 10th November 2011

  29. Software with following functionality User Problem description Displaying Knowledge Adding Knowledge Knowledge Reuse Permanent Notes Storage Voting on Notes EMBET architecture: Core, GUI Context detection Context Matching to display information & knowledge Plain text analysis using Advanced Semantic Annotation Algorithms – OnTeA Theory of different context matching algorithms EMBET: Achievements Bratislava, 10th November 2011

  30. Acoma: Hint Recommendation Bratislava, 10th November 2011

  31. Information Retrieval and Information Extraction lectures

  32. IR Lectures • Introduction to Information Retrieval • Text Operations, Text Analysis, stemming • Crawling, link processing • IR Models, Indexing techniques • IR Software libraries and systems • Ranking by Graph Algorithms (PageRank, HITS, …) and Searching • Information Extraction • Regular Expressions • Large Scale Data Processing on MapReduce Architecture • Multimedia Information Retrieval • Evaluation Techniques, Precision, Recall • Google • Semantics and IR, Semantic Web Standards Bratislava, 10th November 2011

  33. Lectures conditions • Every students gets project focused on • Crawling • Indexing • Ranking • Information Extraction • Large Scale information Processing • They have to consult project 3 times during semester • Availability of data from day one • Lectures are available at: • http://vi.ikt.ui.sav.sk/Témy_prednášok Bratislava, 10th November 2011

More Related