approximate substring matching over uncertain strings
Download
Skip this Video
Download Presentation
Approximate Substring Matching over Uncertain Strings

Loading in 2 Seconds...

play fullscreen
1 / 37

Approximate Substring Matching over Uncertain Strings - PowerPoint PPT Presentation


  • 78 Views
  • Uploaded on

Approximate Substring Matching over Uncertain Strings. Tingjian Ge Zheng Li University of Kentucky. Motivation. As a consequence of the burgeoning growth of data and the cost/technology constraints against producing completely clean data , there is much uncertainty in the data itself. .

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Approximate Substring Matching over Uncertain Strings' - louis


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
approximate substring matching over uncertain strings

Approximate Substring Matching over Uncertain Strings

TingjianGeZheng Li

University of Kentucky

motivation
Motivation
  • As a consequence of the burgeoning growth of data and the cost/technology constraints against producing completely clean data, there is much uncertainty in the data itself.

Text retrieval

Computational biology

Signal processing

motivation1
Motivation
  • The amount of text data increases in an unprecedented rate. Managing the sheer amount of (often noisy) text data has become more challenging than ever.
  • Approximate substring matching has many applications.
      • The deterministic case is well studied.
      • But approximate pattern matching over uncertain texts is largely an unexplored problem in the past.
example pattern matching in dna sequence
Example : Pattern Matching in DNA Sequence
  • Suppose a certain DNA pattern, say AAATTT and it’s variations ( with small edit distance ) are known to be the cause of a low or nonfunctioning protein. Thus we want to do approximate pattern matching in DNA sequences.
  • Challenges are :
    • A single DNA sequence can be a few million to a few hundred million characters.
    • It has uncertainty due to a number of factors in the high-throughput sequencing technologies.
example holter monitor application
Example : Holter Monitor Application
  • For each heartbeat, the annotation software gives a symbol such as N (Normal beat), L (Left bundle branch block beat), etc. Quite often, the ECG signal of each beat may have ambiguity.
  • A doctor might be interested in locating a pattern such as “NNAV”, in order to verify a specific diagnosis.
outline
Outline
  • Motivation
  • Preliminaries
  • A new semantics
  • Multilevel filtering index
  • Two verification algorithms
  • Experiments
q gram index
Q-gram Index
  • The most frequently used indexing method for approximate matching is the q-gram indexes.
  • To use the q-gram index, partition pattern into k+1pieces, where k is the edit distance threshold. Since the number of errors is no more than k, it must be true that at least one piece must have an exact match in the text.

Position list of q-gram1

k+1 pieces

verification algorithm
Verification Algorithm
  • A DP algorithm that computes edit distance

t h i s

Ins.

h a s

Sub.

The edit distance between “has” and “This” is 2 in this example.

The value in each cell is the edit distance between the corresponding pattern and text characters read so far.

outline1
Outline
  • Motivation
  • Preliminaries
  • A new semantics
  • Multilevel filtering index
  • Two verification algorithms
  • Experiments
k matching query
(k, τ)-Matching Query
  • We propose (k, τ)-matching query, which is based on a pattern string p, a set of uncertain text strings {Xi} (1 ≤ i ≤ r), and threshold parameters k, τ, and asks for all substrings X of Xi’s such that Pr[d(p, X) ≤ k] > τ.
  • An other semantics is the EED ( expected edit distance ), which computes the expected edit distance between p and X and check if it’s < k’.
semantics
Semantics
  • (k, τ)-matching query v.s. EED
    • EED first summarizes possible worlds andthen apply the threshold, hence many algorithms developed for the deterministic case are inapplicable.
    • More importantly, EED may either miss real matches or have an unduly big threshold so that many false positives may mix in.
example semantics
Example : Semantics

Consider this approximate pattern matching in DNA sequence :

Pattern p, Length = 20

Substring X1 has an exact match, but 9 characters are uncertain. EED(p, X1) = 6

Substring X2 has no exact match, with 5 errors and 2 uncertain characters. EED(p,X2)=6

To find X1, with EED, the threshold should be at least 6, which means we can’t avoid X2.

But with (k, τ)-matching query, we can use a ( 2, 0 )-matching query toselect X1 and avoid X2at the same time.

outline2
Outline
  • Motivation
  • Preliminaries
  • Semantics
  • Multilevel filtering Index
  • Two verification algorithms
  • Experiments
left right signature of a q gram
Left/Right Signature of a Q-gram
  • The edit distance of signatures is no more than the edit distance of original strings if we treat each two-bit value in the signature as a character when computing the edit distance.

left signature

right signature

q-gram

RATGS

Text string x

Hash

pos

pos+q

Left signature and right signature of a q-gram.

The hash function maps a character to a two-bit value.

multilevel filtering index
Multilevel Filtering Index

28

72

short position list

4-byte tag

4-byte tag in the figure contains left/right signature of the q-gram.

4-byte tag

. . . . .

. . . . .

next level directory

4-byte tag

99

short position list

A multilevel filtering Index for a q-gram based on measuring signature distance.

best matching prefix
Best Matching Prefix

x

G

P

P

G

A

p

1

2

3

4

5

0

G

0

1

1

2

3

4

1

A

2

1

1

2

3

2

1

2

2

3

2

P

“1” is the smallest in the last row. Hence, 1 is x’sprefix distance from p and GGAP is the best matching prefix of x for p.

dynamic programing verification using signatures
Dynamic Programing Verification Using Signatures
  • The value on each diagonal in the DP table form a non-decreasing sequence, as shown in the figure.
  • Let be prefix

distance from , The

verification requires :

We need to do this mapping

to accommodate the certain

length of signatures in the index

tag.

x’[1]

x’[l]

x’[m’]

p’[1]

p’[l−k]

p’[l]

p’[l+k]

p’[m’]

x’[m’−k]

x’[m’+k]

dynamic programming verification using signatures
Dynamic Programming Verification Using Signatures
  • When pattern is exhausted. We do mapping as the following.

x’[l]

p’[m’]

x’[m’+k]

x’[m’−k]

example using the index
Example : Using the Index

Q-gram “BEST” matches. We then use this q-gram’s multi-level signature index for filtering.

  • Input:

X: … … [DC][CB][BEST][RO][DO]… …

P: [DC][AB][BEST][NR][OB]

K = 2

Text

Text

Pattern

Pattern

example using the index1
Example : Using the Index

Use the first level signatures in the index tag.

  • Input:

Need to expand to next level of the index.

X: … … [DC][CB][BEST][RO][DO]… …

P: [DC][AB][BEST][NR][OB]

K = 2

Text

Text

B’

C’

R’

O’

Pattern

Pattern

0

1

2

0

1

2

B’

1

0

1

N’

1

1

2

A’

2

1

1

R’

2

1

2

C’

O’

2

1

2

1

3

3

D’

B’

4

3

2

4

3

2

using the index example
Using the Index-Example

Use the second level signatures in the index tag.

  • Input:

This candidate position is filtered out.

X: … … [DC][CB][BEST][RO][DO]… …

P: [DC][AB][BEST][NR][OB]

Pattern is exhausted in this case.

K = 2

Text

Text

B

C

C

D

R

O

D

O

Pattern

Pattern

0

1

2

3

4

0

1

2

3

4

B

1

0

1

2

3

N

1

1

2

3

4

A

2

1

1

2

3

R

2

1

2

3

4

C

1

O

2

1

2

2

1

2

3

3

3

D

B

4

3

2

2

1

4

3

2

2

3

outline3
Outline
  • Motivation
  • Preliminaries
  • Semantics
  • Multilevel filtering index
  • Two verification algorithms
    • Bounds based on CDF
    • Bounds based on local perturbation
  • Experiments
verification algorithms
Verification Algorithms
  • The goal of verification is to conclude whether a candidate position selected by an index is a true match.
  • We present two algorithms, each of which gives an upper and a lower bound of the probability that .
outline4
Outline
  • Motivation
  • Preliminaries
  • Semantics
  • Multilevel filtering index
  • Two verification algorithms
    • Bounds based on CDF
    • Bounds based on local perturbation
  • Experiments
bounds based on cumulative distribution functions cdf
Bounds Based on Cumulative Distribution Functions (CDF)
  • The basic verification consists of two symmetric runs of a DP algorithm. We describe how we change such a DP algorithm to accommodate uncertain characters.
  • Our key idea is to compute (at most) k+1 pairs of values in each cell, i.e., where D denotes the edit distance.
a basic step
A Basic Step

Consider a basic step: how do we get D from its 3 neighbors?

//probability of a match at cell D

Pick one fixed neighbor cell to use.

Use the union of upper bounds from the three neighbors

argminiDi returns the index value i that minimizes Di; the minimization is defined as the Di (1 ≤ i ≤ 3) that has the greatest ( i.e. the one that has a small distance value with the highest prob. ).

example bounds based on cdf
Example : Bounds Based on CDF
  • Consider p = “CAT” and X is “C” followed by four characters, each of which has the same distribution G.1A.4T.5, denoting that it is G (A, T) with probability 0.1 (0.4, 0.5). K = 2.
  • Take the cell at the 3rd row and the 4th column as an example. How do we compute Fl[j] & Fu[j]?

argminiDi is D3

Mixture of the upper bound in D1 and the union of those in all three neighbours.

outline5
Outline
  • Motivation
  • Preliminaries
  • Semantics
  • Multilevel filtering technique based on measuring signature distance
  • Two verification algorithms
    • Bounds based on CDF
    • Bounds based on local perturbation
  • Experiments
bounds based on local perturbation
Bounds Based on Local Perturbation
  • Adjacent and remote possible worlds: Give a (k, τ)-matching query on a pattern p and on an uncertain text X, we say that a p.w. w of X, denoted x(w) is adjacent to pif . We say that it is remote to p if .

Perturbation

An initial adjacent\remote p.w.

More adjacent\remote p.w. after perturbation.

how to get initial adjacent remote possible world
How to Get Initial Adjacent/Remote Possible World.

K=1

x

Get a closest p.w. ( an adjacent p.w. with the smallest distance ) on the optimal path in DP table.

C

G.1A.4T.5

G.1A.4T.5

G.1A.4T.5

p

C

A

T

If D1 and D contain the same distance value, the corresponding variable in the test string is called a crucial variable of this p.w.

Use a randomized algorithm to get a remote p.w. :

perturbation how
Perturbation – How?

Let be the difference between k and the edit distance of this p.w and the pattern.

Text string x

For an closest p.w. :

Pr(at most crucial variables ( of the total c curial variables ) change their values as in the optimal path ) is a lower bound of .

Suppose there’re c crucial variables in this closest/remote p.w.

For a remotep.w. :

We could get a upper bound similarly.

outline6
Outline
  • Motivation
  • Preliminaries
  • Semantics
  • Multilevel filtering index
  • Two verification algorithms
  • Experiments
setup of experiment
Setup of Experiment
  • We examine the behaviors of (k, τ)-pattern matching query using signature filtering and verification bounds with the following dataset.
  • The DNA dataset.
    • Raw datasets of sequencing runs of Escherichia coli 536 from the NCBI SRA (Sequence Read Archive) database.
    • Use Bowtie to align the short DNA sequences with the complete Escherichia coli genome reference. The mapping reports output by Bowtie show positions within the DNA that have more than one possible value.
  • The protein dataset.
  • Synthetic datasets.
    • We generate a few synthetic datasets based on the two real datasets above.
    • Vary the parameter values of data (such as the uncertainty ratio θ) or the size of the data.
experiments
Experiments

Running time for various settings.

Varying θ.

Breakdown of I/O & CPU costs.

Using larger synthetic data.

experiments1
Experiments

Varying |p| (protein).

Varying threshold k.

# of positions to be verified.

Varying |p| (DNA).

conclusions and future work
Conclusions and Future Work
  • We study a real and unsolved problem of approximate substring matching over uncertain texts.
    • Proposed a novel semantics and demonstrate its advantages over an alternative one introduced by previous work.
    • Developed a q-gram based index to handle uncertain texts.
    • Proposed two efficient verification algorithms.
  • As future work, we plan to study the matching problem under correlated uncertainty to address a wider range of applications.
thank you
Thank You!
  • Questions?
ad