Approximate Substring Matching over Uncertain Strings

1. Approximate Substring Matching over Uncertain Strings TingjianGeZheng Li University of Kentucky

2. Motivation • As a consequence of the burgeoning growth of data and the cost/technology constraints against producing completely clean data, there is much uncertainty in the data itself. Text retrieval Computational biology Signal processing

3. Motivation • The amount of text data increases in an unprecedented rate. Managing the sheer amount of (often noisy) text data has become more challenging than ever. • Approximate substring matching has many applications. • The deterministic case is well studied. • But approximate pattern matching over uncertain texts is largely an unexplored problem in the past.

4. Example : Pattern Matching in DNA Sequence • Suppose a certain DNA pattern, say AAATTT and it’s variations ( with small edit distance ) are known to be the cause of a low or nonfunctioning protein. Thus we want to do approximate pattern matching in DNA sequences. • Challenges are : • A single DNA sequence can be a few million to a few hundred million characters. • It has uncertainty due to a number of factors in the high-throughput sequencing technologies.

5. Example : Holter Monitor Application • For each heartbeat, the annotation software gives a symbol such as N (Normal beat), L (Left bundle branch block beat), etc. Quite often, the ECG signal of each beat may have ambiguity. • A doctor might be interested in locating a pattern such as “NNAV”, in order to verify a specific diagnosis.

6. Outline • Motivation • Preliminaries • A new semantics • Multilevel filtering index • Two verification algorithms • Experiments

7. Q-gram Index • The most frequently used indexing method for approximate matching is the q-gram indexes. • To use the q-gram index, partition pattern into k+1pieces, where k is the edit distance threshold. Since the number of errors is no more than k, it must be true that at least one piece must have an exact match in the text. Position list of q-gram1 k+1 pieces

8. Verification Algorithm • A DP algorithm that computes edit distance t h i s Ins. h a s Sub. The edit distance between “has” and “This” is 2 in this example. The value in each cell is the edit distance between the corresponding pattern and text characters read so far.

9. Outline • Motivation • Preliminaries • A new semantics • Multilevel filtering index • Two verification algorithms • Experiments

10. (k, τ)-Matching Query • We propose (k, τ)-matching query, which is based on a pattern string p, a set of uncertain text strings {Xi} (1 ≤ i ≤ r), and threshold parameters k, τ, and asks for all substrings X of Xi’s such that Pr[d(p, X) ≤ k] > τ. • An other semantics is the EED ( expected edit distance ), which computes the expected edit distance between p and X and check if it’s < k’.

11. Semantics • (k, τ)-matching query v.s. EED • EED first summarizes possible worlds andthen apply the threshold, hence many algorithms developed for the deterministic case are inapplicable. • More importantly, EED may either miss real matches or have an unduly big threshold so that many false positives may mix in.

12. Example : Semantics Consider this approximate pattern matching in DNA sequence : Pattern p, Length = 20 Substring X1 has an exact match, but 9 characters are uncertain. EED(p, X1) = 6 Substring X2 has no exact match, with 5 errors and 2 uncertain characters. EED(p,X2)=6 To find X1, with EED, the threshold should be at least 6, which means we can’t avoid X2. But with (k, τ)-matching query, we can use a ( 2, 0 )-matching query toselect X1 and avoid X2at the same time.

13. Outline • Motivation • Preliminaries • Semantics • Multilevel filtering Index • Two verification algorithms • Experiments

14. Left/Right Signature of a Q-gram • The edit distance of signatures is no more than the edit distance of original strings if we treat each two-bit value in the signature as a character when computing the edit distance. left signature right signature q-gram RATGS Text string x Hash pos pos+q Left signature and right signature of a q-gram. The hash function maps a character to a two-bit value.

15. Multilevel Filtering Index 28 72 short position list 4-byte tag 4-byte tag in the figure contains left/right signature of the q-gram. 4-byte tag . . . . . . . . . . next level directory 4-byte tag 99 short position list A multilevel filtering Index for a q-gram based on measuring signature distance.

16. Best Matching Prefix x G P P G A p 1 2 3 4 5 0 G 0 1 1 2 3 4 1 A 2 1 1 2 3 2 1 2 2 3 2 P “1” is the smallest in the last row. Hence, 1 is x’sprefix distance from p and GGAP is the best matching prefix of x for p.

17. Dynamic Programing Verification Using Signatures • The value on each diagonal in the DP table form a non-decreasing sequence, as shown in the figure. • Let be prefix distance from , The verification requires : We need to do this mapping to accommodate the certain length of signatures in the index tag. x’[1] x’[l] x’[m’] p’[1] p’[l−k] p’[l] p’[l+k] p’[m’] x’[m’−k] x’[m’+k]

18. Dynamic Programming Verification Using Signatures • When pattern is exhausted. We do mapping as the following. x’[l] p’[m’] x’[m’+k] x’[m’−k]

19. Example : Using the Index Q-gram “BEST” matches. We then use this q-gram’s multi-level signature index for filtering. • Input: X: … … [DC][CB][BEST][RO][DO]… … P: [DC][AB][BEST][NR][OB] K = 2 Text Text Pattern Pattern

20. Example : Using the Index Use the first level signatures in the index tag. • Input: Need to expand to next level of the index. X: … … [DC][CB][BEST][RO][DO]… … P: [DC][AB][BEST][NR][OB] K = 2 Text Text B’ C’ R’ O’ Pattern Pattern 0 1 2 0 1 2 B’ 1 0 1 N’ 1 1 2 A’ 2 1 1 R’ 2 1 2 C’ O’ 2 1 2 1 3 3 D’ B’ 4 3 2 4 3 2

21. Using the Index-Example Use the second level signatures in the index tag. • Input: This candidate position is filtered out. X: … … [DC][CB][BEST][RO][DO]… … P: [DC][AB][BEST][NR][OB] Pattern is exhausted in this case. K = 2 Text Text B C C D R O D O Pattern Pattern 0 1 2 3 4 0 1 2 3 4 B 1 0 1 2 3 N 1 1 2 3 4 A 2 1 1 2 3 R 2 1 2 3 4 C 1 O 2 1 2 2 1 2 3 3 3 D B 4 3 2 2 1 4 3 2 2 3

22. Outline • Motivation • Preliminaries • Semantics • Multilevel filtering index • Two verification algorithms • Bounds based on CDF • Bounds based on local perturbation • Experiments

23. Verification Algorithms • The goal of verification is to conclude whether a candidate position selected by an index is a true match. • We present two algorithms, each of which gives an upper and a lower bound of the probability that .

24. Outline • Motivation • Preliminaries • Semantics • Multilevel filtering index • Two verification algorithms • Bounds based on CDF • Bounds based on local perturbation • Experiments

25. Bounds Based on Cumulative Distribution Functions (CDF) • The basic verification consists of two symmetric runs of a DP algorithm. We describe how we change such a DP algorithm to accommodate uncertain characters. • Our key idea is to compute (at most) k+1 pairs of values in each cell, i.e., where D denotes the edit distance.

26. A Basic Step Consider a basic step: how do we get D from its 3 neighbors? //probability of a match at cell D Pick one fixed neighbor cell to use. Use the union of upper bounds from the three neighbors argminiDi returns the index value i that minimizes Di; the minimization is defined as the Di (1 ≤ i ≤ 3) that has the greatest ( i.e. the one that has a small distance value with the highest prob. ).

27. Example : Bounds Based on CDF • Consider p = “CAT” and X is “C” followed by four characters, each of which has the same distribution G.1A.4T.5, denoting that it is G (A, T) with probability 0.1 (0.4, 0.5). K = 2. • Take the cell at the 3rd row and the 4th column as an example. How do we compute Fl[j] & Fu[j]? argminiDi is D3 Mixture of the upper bound in D1 and the union of those in all three neighbours.

28. Outline • Motivation • Preliminaries • Semantics • Multilevel filtering technique based on measuring signature distance • Two verification algorithms • Bounds based on CDF • Bounds based on local perturbation • Experiments

29. Bounds Based on Local Perturbation • Adjacent and remote possible worlds: Give a (k, τ)-matching query on a pattern p and on an uncertain text X, we say that a p.w. w of X, denoted x(w) is adjacent to pif . We say that it is remote to p if . Perturbation An initial adjacent\remote p.w. More adjacent\remote p.w. after perturbation.

30. How to Get Initial Adjacent/Remote Possible World. K=1 x Get a closest p.w. ( an adjacent p.w. with the smallest distance ) on the optimal path in DP table. C G.1A.4T.5 G.1A.4T.5 G.1A.4T.5 p C A T If D1 and D contain the same distance value, the corresponding variable in the test string is called a crucial variable of this p.w. Use a randomized algorithm to get a remote p.w. :

31. Perturbation – How? Let be the difference between k and the edit distance of this p.w and the pattern. Text string x For an closest p.w. : Pr(at most crucial variables ( of the total c curial variables ) change their values as in the optimal path ) is a lower bound of . Suppose there’re c crucial variables in this closest/remote p.w. For a remotep.w. : We could get a upper bound similarly.

32. Outline • Motivation • Preliminaries • Semantics • Multilevel filtering index • Two verification algorithms • Experiments

33. Setup of Experiment • We examine the behaviors of (k, τ)-pattern matching query using signature filtering and verification bounds with the following dataset. • The DNA dataset. • Raw datasets of sequencing runs of Escherichia coli 536 from the NCBI SRA (Sequence Read Archive) database. • Use Bowtie to align the short DNA sequences with the complete Escherichia coli genome reference. The mapping reports output by Bowtie show positions within the DNA that have more than one possible value. • The protein dataset. • Synthetic datasets. • We generate a few synthetic datasets based on the two real datasets above. • Vary the parameter values of data (such as the uncertainty ratio θ) or the size of the data.

34. Experiments Running time for various settings. Varying θ. Breakdown of I/O & CPU costs. Using larger synthetic data.

35. Experiments Varying |p| (protein). Varying threshold k. # of positions to be verified. Varying |p| (DNA).

36. Conclusions and Future Work • We study a real and unsolved problem of approximate substring matching over uncertain texts. • Proposed a novel semantics and demonstrate its advantages over an alternative one introduced by previous work. • Developed a q-gram based index to handle uncertain texts. • Proposed two efficient verification algorithms. • As future work, we plan to study the matching problem under correlated uncertainty to address a wider range of applications.

37. Thank You! • Questions?