1 / 22

Mining Document Collections to Facilitate Accurate Approximate Entity Matching

Mining Document Collections to Facilitate Accurate Approximate Entity Matching. Presented By Harshda Vabale. Introduction. Typical product analytics and reporting system. Identification of entities in queries submitted to query engine. Reference entity database. ID Entity Name

Download Presentation

Mining Document Collections to Facilitate Accurate Approximate Entity Matching

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Mining Document Collections to Facilitate AccurateApproximate Entity Matching Presented By Harshda Vabale

  2. Introduction • Typical product analytics and reporting system. • Identification of entities in queries submitted to query engine. • Reference entity database. • ID Entity Name • E1 Canon EOS Digital Rebel XTI SLR Camera • e2 Lenovo ThinkPad X61 Notebook • e3 Sony Vaio F150 Laptop Example Entities

  3. Alternative method • Mine variations of entities in reference table. • Expanding the reference table with variations. • Propose an architecture which scales to a very large number of documents and reference entities.

  4. Architectural Overview

  5. Problem Definition • Let E denote the set of entities in a reference table. For each e belongs E, let Tok(e) denote the set of tokens in e. • For simplicity, we loosely use e to denote Tok(e). We say that an entity e contains a token set T if T subset Tok(e). • The frequency of a token set T with respect to E is the number of entities in E which contain T . We use the notation Te to denote a • subset of tokens of e. That is, Te subset e. • DisTokenSet

  6. Architecture • The first candidate IDTokenSet generation (CIG) phase generates the (candidate IDTokenSet, entityID) pairs. Specifically, a candidate IDTokenSet is a substring of some documents, and also a DisTokenSet of the entity. • The second per-document score computation (PSC) phase computes the per-document correlation (using either g1 or g2) between each candidate IDTokenSet and the entity it may identify for each of the documents. • The third score aggregation and thresholding (SAT) phase aggregates these scores, and outputs the candidate IDTokenSets whose score is above the given threshold.

  7. Subsequence –Subset Join Baseline Solutions: • Substring Extraction • Subset Check The solution of this problem goes as follows: Filtering Document substrings: Properties: Linear Time Document Processing High Selectivity Compactness

  8. Subsequence –Subset Join • Batching Subset Checks : • Candidate substrings across documents contain many common token sets; thus, checking these common token sets once across all documents results in significant savings. • We also “order" the subset-checks for each of these token sets so that we can reuse partial computation across different token sets. • Since we cache the intermediate results, we need to discard results that are not required for future subset checks.

  9. Filtering Document Substrings • CT filter Procedure for identifying Core Token Sets CT Filter • Generating Hit Sequence for a document Hit Sequence Generation Core Token Set

  10. Algorithm For Hit Sequence Generation

  11. Batching Subset Checks • CT Index • Suffix Tree based Scheduling

  12. Candidate IDTokenSet Generation Algorithm

  13. Complete Architecture • Pre Document Score Computation: • Score Aggregation And Thresholding • Handling Large Intermediate Results

  14. Alternative Implementations • One Doc Scan Approach: The one DocScan implementation leverages the fact that when the number of entities is not large, the CT Index can be loaded into memory. In this case, we can run CIG on b documents, where b is the batch size determined by the space budget. • Map – Reduce Framework Map reduce is a software framework that supports parallel computations over large data sets on clusters of computers. This framework consists of a “map" step and a “reduce" step. In the map step, the master node chops the original problem up into smaller sub-problems, and distributes those to multiple worker nodes. The worker node processes that smaller problem, and passes the answer to the reduce step. In the reduce step, the master node combines the answers to all the sub-problems and gets the answer to the original problem.

  15. Performance Study

  16. Performance Study • High quality IDTokenSets: the document-based measure performs significantly better than a representative string-based similarity measure in determining IDTokenSets. To begin with, we show some example IDTokenSets generated by our framework in Table. • Scalable IDTokenSets generation: our algorithms efficiently generate IDTokenSets from large collections of entities and documents. Speci¯cally: (a) CT Filter and CT Index improve the baseline byorders of magnitudes. (b) Candidate IDTokenSet generation using suffix tree is significantly faster than the baseline. (c) Two DocScan is scalable to a very large number of entities.

  17. Quality of IDTokenSets • First describe the experimental setup, and then compare the precision-recall with respect to different similarity measures. Also examine the effect by using different window sizes in determining the context, and by allowing gaps in extracting candidate IDTokenSets.

  18. Quality of IDTokenSets • Experimental Setup • Comparison with String Similarity • Effect of Window Size • Effect of Gapped Mentions

  19. Scalability • Overall Performance: • Set K = 5 for DisTokenSet. In order to examine the computational and space complexity of the CT Filter and CT Index, we first vary K0 and L values on the 1M product entities. The number of core token sets, the average length of idlists per core token set and the execution time to compute the core token sets are reported in Table

  20. Scalability • Effect of Core Token Sets • Effect of Batched Subset Check • Two DocScan vs One DocScan

  21. ???Questions????

  22. Thank You

More Related