1 / 51

SEEDEEP: A System for Exploring and Querying Deep Web Data Sources

SEEDEEP: A System for Exploring and Querying Deep Web Data Sources. Gagan Agrawal Fan Wang, Tantan Liu Ohio State University. The Deep Web. The definition of “the deep web” from Wikipedia.

oleg-bass
Download Presentation

SEEDEEP: A System for Exploring and Querying Deep Web Data Sources

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SEEDEEP: A System for Exploring and Querying Deep Web Data Sources Gagan Agrawal Fan Wang, Tantan Liu Ohio State University

  2. The Deep Web • The definition of “the deep web” from Wikipedia The deep Web refers to World Wide Web content that is not part of the surface web, which is indexed by standard search engines.

  3. The Deep Web is Huge • 500 times larger than the surface web • 7500 terabytes of information (19 terabytes in the surface web) • 550 billion documents (1 billion in the surface web) • More than 200,000 deep web sites • Relevant to every domain: scientific, e-commerce, market

  4. The Deep Web is Informative • Deeper content than surface web • Surface web: text format • Deep web: specific and relational information • More than half of the deep web content in topic-specific databases • Biology, Chemistry, Medical, Travel, Business, Academia, and many more… • 95 percent of the deep web is publicly accessible

  5. Hard to Use the Deep Web • Challenges for Integration • Self-maintained and created • Heterogeneous and hidden metadata • Dynamically updated metadata • Challenges for Searching • Standard input format • Data redundancy and data source ranking • Data source dependency • Challenges for Performance • Network latency and caching mechanism • Fault tolerance issue

  6. Motivating Example (1) Biologists have identified the gene X and protein Y are contributors of a disease. They want to examine the SNPs (Single Nucleotide Polymorphisms) located in the genes that share the same functions as either X or Y. Particularly, for all SNPs located in each such gene functions similar to either X or Y, and those have a heterozygosity value greater than 0.01, biologists want to know the maximal SNP frequency in the Asian population.

  7. Motivating Example (2) The gene has the same functions as X The gene has the same functions as Y The frequency information of the SNPs located in these genes and filtered by heterozygosity values

  8. Motivating Example (3) How do you know NCBI Gene could provide gene function information given the gene name? Three data sources, dbSNP, Alfred, and Seattle, could provide SNP frequency data, why do you choose dbSNP? What if SNP500Cancer data source is unavailable? Do NCBI Gene and GO data source both use “function” to represent the meaning of “gene function”? I cannot filter SNP by heterozygosity values on dbSNP A path clearly guides the search

  9. Our Contribution: SEEDEEP System Discover data source metadata Discover data source inter-dependency Generate query plans for search Fault Tolerance mechanism Query caching mechanism

  10. Outline • Introduction and Motivation • System Core • Query planning problem • Query planning algorithms • Other system components • Query caching • Fault tolerance • Schema mining • Other Issues

  11. What queries does our system support? They want to examine the SNPslocated in the genes that share the same functions as either X or Y. Particularly, for all SNPs located in each such gene functions similar to either X or Y, and those have a heterozygosity value greater than 0.01, biologists want to know the maximal SNP frequency in the Asian population. Selection-Projection-Join (SPJ) queries Aggregation-Groupby queries Nested queries: Condition and Entity

  12. Data Source Model (1): Single Data Source • Each data source is a virtual relational table • Virtual relational data elements • MI: must fill-in input attributes • OI: optional fill-in input attributes • O: output attributes • C: inherent data source constraints

  13. Data Source Model (2): Correlated Sources • Hyper-graph dependency model • Multi-source dependency • Dependency relations for data sources D1 and D2 • Type 1: D1 provides must fill-in inputs for D2 • Type 2: D1 provides optional fill-in inputs for D2

  14. Planning Algorithm Overview Tree representation of user query • Each node represents a simple query Query Types: 2. A divide-and-conquer approach 1. Aggregation query 3. A final combination step generates the final query plan 2. Nested entity sub-query 3. Ordinary query

  15. Query Planning Problem for Ordinary Query • Ordinary query format • Entity keywords, attribute keywords, comparison predicates • Standard select-project-join SQL query style • Formulation • Sub-graph set cover problem, NP-hard Target data source Starting data source

  16. Bidirectional Query Planning Algorithm (1) • Heuristic algorithm based on the algorithm introduced by Kacholia et al. • Algorithm overview • Starting nodes • Target nodes • Bidirectional graph traversal

  17. Bidirectional Query Planning Algorithm (2) • How to find minimal sub-graph • Find the shortest paths from starting nodes to target nodes • Dijkstra’s shortest path algorithm • Benefit function • Data source coverage • Data source data quality, ontology based • User constraints matching

  18. Query Planning Problem for Aggregation Query • Node connection property The aggregation data source(s) must be directly or indirectly connected with the grouping data source. • Formulation • Sub-graph set cover problem with node connection property constraint • NP-hard

  19. Center-spread Query Planning Algorithm (1) • Algorithm initialization • Starting nodes • Target nodes • Center nodes: aggregation data source nodes • Algorithm overview • Graph traversal starts from the center nodes • Gradually add center nodes’ neighbors adhering to node connection property

  20. Center-spread Query Planning Algorithm (2) Grouping data source Grouping data source

  21. Query Planning Problem for Nested Entity Query(1) “SNP_Frequency, Gene {Function, X}” Find the genes which have the same functions as X {Gene, Function, X} Find the entities specified by b that have the same value on attribute a as the entities that are specified by e1,…,ek

  22. Query Planning Problem for Nested Entity Query(2) • Node linking property “Gene {Function, Protein X}” b a e The linking data source, which is the data source covering keyword a, must be topologically before the data source covering the entity keyword b • Formulation • Sub-graph set cover problem with node linking property constraint • NP-hard

  23. Plan Combination Receiving nodes Ending nodes Ending nodes

  24. Plan Merging • Query plans for sub-queries can be similar • Reduce the network transmission cost of a query plan • Two edges and can be merged if the used input and output of paired data sources is the same • Mergeable edges weights • Optimal Merging • Compatibility graph CG • Maximal node weighted clique in CG • Modified reactive local search (tabu search) algorithm

  25. Query Execution Optimization: Pipelined Aggregation • Performing aggregation in a pipelined manner • Reduce transmission cost by early pruning • Grouping-first query plans

  26. Query Execution Optimization: Moving Partial Grouping Forward • Aggregation-first query plans • Conditions • Aggregation data source AD covers a term pga • 1 to 1 relation between the entity specified by pga and the entity specifed by the grouping attribute • N to 1 relation between the entity specified by the aggregation attribute and the entity specified by pga

  27. Query Planning Evaluation (1) • Cost model evaluation: query plan size

  28. Query Planning Evaluation (2) • Planning Algorithm Scalability • 0.03% query planning overhead

  29. Query Planning Evaluation (3) • Optimization techniques • NO: No optimization technique used • Merging: Only perform plan merging • Grouping: Only perform two grouping optimizations • M+G: Perform both merging and grouping

  30. Outline • Introduction and Motivation • System Core • Query planning problem • Query planning algorithms • Other system components • Query caching • Fault tolerance • Schema mining • Proposed work

  31. Query Caching: Motivation • High response time for deep web queries • Motivating observations • Data source redundancy • Data sources return answers in a All-In-One fashion • Users issue similar queries in one session • Query-Plan-Driven query caching method • Not only cache previous data, also query plans • Caching query plans increases the possibility of data reuse

  32. Query Caching: Strategy Overview • We are given a list of n previous issued queries, each of which has a query plan Pi • Given a new query q, we want to generate a query plan for q in the following way • Define a reusability metric to identify the previous query plans that is beneficial to reuse • Select a set of reusable previous queries and query plans • Use a selection function to obtain the sub-query plans we will like to reuse • Use a modified query planning algorithm to generate query plan for the new query based on reusable plan templates

  33. Query Caching: Evaluation • Three mechanisms compared • NC: No Caching • DDC: Data Driven Caching • PDC: Plan Driven Caching

  34. Fault Tolerance: Motivation • Remote data sources are vulnerable to unavailability or inaccessibility • Data redundancy across multiple data sources, partial redundancy • Use similar data sources to hide unavailable or inaccessible data sources • Data redundancy based incremental query processing • Not generate new plan from scratch • Inaccessible part is suspended • Incrementally generate a new part to replace the inaccessible part

  35. Fault Tolerance: Strategy Overview • System Model: data redundancy graph model • Nodes: data sources • Edges: redundancy usage between data source pair • Given a query plan P and a set of unavailable data sources UDS, find the minimal impacted sub-plan MISubP • Impacted sub-plan: the sub-plan of the original plan P which is rooted at unavailable data sources UDS • Minimal impacted sub-plan: an impacted sub-plan with no usable data sources • Generate the maximal fixable sub-query of the minimal impacted sub-plan • Maximal fixable sub-query doesn’t contain any dead attributes which are covered by the minimal impacted sub-plan • Generate a query plan for the maximal fixable sub-query as the new partial query plan

  36. Fault Tolerance: Evaluation • Query plan execution time • Generate new plan from scratch • Our incremental query processing strategy

  37. Schema Mining: Motivation • Data source metadata reveals data source coverage information • Metadata: input and output attributes • Data sources only return a partial set of output attributes in response to a query • the ones have non-NULL values for the input • Find approximate complete output attribute set

  38. Schema Mining: Strategy Overview • Sampling based method • A modest sized sample could discover most deep web data source output schema • Rejection sampling method to choose the sample • A sample size estimator is constructed • Mixture model method • Sample is not enough • Output attributes could be shared among different data sources • Data source: probabilistic data source model generates output attributes with certain probability • Borrowability among data sources: an output attribute is generated from a mixture of different probabilistic data source models

  39. Schema Mining: Evaluation • Four methods compared • SamplePC: Sampling + Perfect label classifier • SampleRC: Sampling + Real label classifier • Mixture: Mixture model method • Mixture + Sample: SampleRC + Mixture

  40. Outline • Introduction and Motivation • System Core • Query planning problem • Query planning algorithms • Other system components • Query caching • Fault tolerance • Schema mining • Proposed work

  41. Answering Relationship Search over Deep Web Data Sources: Motivation and Formulation • Knowledge is only useful when it is related • Linked web data • Deep web data sources are ideal sources for linked data • Supported by backend relational databases • Data on output pages are related • Deep web data sources are correlated, input and output relation • Deep web data source output pages are hyperlinked with output pages from other data sources • Problem Formulation • A relationship query RQ={ke1,ke2} • Find the terms relate ke1 with ke2

  42. Relationship Query: Proposed Method 1 • Use correlation among data sources Q={MSMB, RET} Find the relation between these two genes Connect the data source taking one gene as input and another data source taking the other gene as output Connect the data sources taking two genes as input • A modified query planning algorithm introduced in the current work

  43. Relationship Query: Proposed Method 2 • Use hyperlinks among different output pages to build relation • Two-level source-object graph model • Sampled output pages • Extract objects (entities) represented as (data source, object name) pair • Extract hyperlinks on output pages, pointing from one object to another object in different output pages • Data source nodes and object nodes • Data source virtual link edges connect correlated data sources • Hyperlink edges connects hyperlinked object nodes or connects data source node with its corresponding object nodes • Edges are weighted

  44. Relationship Query: Graph Model Data source node Data source virtual link edge Hyperlink edge Edge weight Hyperlink edge object node

  45. Relationship Query: Method 2 Algorithm • Identify two nodes in the graph as path ends • Path weight: multiplication of edge weights • Shortest N paths: NP-hard problem Shortest Paths

  46. Quality-Aware Data Source Selection based on Functional Dependency Analysis: Motivation • Current data source selection method • Coverage • Overlap • relevance • Quality-aware data source selection • Data richness • Both sources A and B provide information genes and their encoded proteins • A only considers one encoding schema, but B considers two • B is better than A, but how to detect? Can we find the information we need? Which one is better?

  47. Quality-Aware Data Source Selection: Proposed Method (1) • Functional dependency • A functional dependency any two tuples t1 and t2 that have must have • The previous example • Data source A • Data source B • Extract functional dependencies • Sampling: data tuples from deep web data sources • Discover functional dependencies

  48. Quality-Aware Data Source Selection: Proposed Method (2) • A set of data sources • Each has a set of functional dependencies • Functional dependency lattice • An attribute set • Data source has functional dependency set on

  49. Optimized Query Answering over Deep Web Data Sources: Motivation and Formulation • Current technique: minimize the number of data sources with benefit function • A more interested aspect: minimized the total query plan execution time • Optimization problem 1: single query • Minimize response time (ERT), maximize plan quality (RS) • Maximize the plan gain per execution unit • Optimization problem 2: multiple queries • Minimize total response time for multiple queries • Scheduling problem, don’t assume similarity among queries

  50. Optimized Query Answering: Proposed Methods • Optimization for single query • Tabu search framework to find the optimal plan • Optimization for multiple queries • Query as a job with a list of tasks • Data sources as machines • Dependencies among task, each task can be performed on a set of machines • Data source response time as machine working time • Job scheduling problem

More Related