Asynchronous pattern matching metrics
This presentation is the property of its rightful owner.
Sponsored Links
1 / 71

Asynchronous Pattern Matching - Metrics PowerPoint PPT Presentation


  • 74 Views
  • Uploaded on
  • Presentation posted in: General

Asynchronous Pattern Matching - Metrics. Amihood Amir. Motivation. Motivation. In the “old” days: Pattern and text are given in correct sequential order. It is possible that the content is erroneous. New paradigm: Content is exact, but the order of the pattern symbols may be scrambled.

Download Presentation

Asynchronous Pattern Matching - Metrics

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Asynchronous pattern matching metrics

Asynchronous Pattern Matching -Metrics

Amihood Amir


Motivation

Motivation


Motivation1

Motivation

In the “old” days: Pattern and text are given in correct sequential order. It is possible that the content is erroneous.

New paradigm: Content is exact, but the order of the pattern symbols may be scrambled.

Why? Transmitted asynchronously? The nature of the application?


Example swaps

Example: Swaps

Tehse knids of typing mistakes are very common

So when searching for pattern These we are seeking the symbols of the pattern but with an order changed by swaps.

Surprisingly, pattern matching with swaps is easier than pattern matching with mismatches (ACHLP:01)


Example reversals

Example: Reversals

AAAGGCCCTTTGAGCCC

AAAGAGTTTCCCGGCCC

Given a DNA substring, a piece of it can detach and reverse.

This process still computationally tough.

Question: What is the minimum number of reversals necessary to sort a permutation of 1,…,n


Global rearrangements

Global Rearrangements?

Berman & Hannenhalli(1996) called this Global Rearrangement as opposed to Local Rearrangement (edit distance). Showed it is NP-hard.

Our Thesis: This is a special case of errors in the address rather than content.


Example transpositions

Example: Transpositions

AAAGGCCCTTTGAGCCC

AATTTGAGGCCCAGCCC

Given a DNA substring, a piece of it can be transposed to another area.

Question: What is the minimum number of transpositions necessary to sort a permutation of 1,…,n ?


Complexity

Complexity?

Bafna & Pevzner(1998), Christie (1998), Hartman (2001): 1.5 Polynomial Approximation.

Not known whether efficiently computable.

This is another special case of errors in the address rather than content.


Example block interchanges

Example: Block Interchanges

AAAGGCCCTTTGAGCCC

AAGTTTAGGCCCAGCCC

Given a DNA substring, two non-empty subsequences can be interchanged.

Question: What is the minimum number of block interchanges necessary to sort a permutation of 1,…,n ?

Christie (1996): O(n )

2


Summary

Summary

Biology: sorting permutations

Reversals

(Berman & Hannenhalli, 1996)

Transpositions

(Bafna & Pevzner, 1998)

Pattern Matching:

Swaps

(Amir, Lewenstein & Porat, 2002)

NP-hard

?

O(n log m)

Block interchanges O(n2)

(Christie, 1996)

Note: A swap is a block interchange simplification

1. Block size

2. Only once

3. Adjacent


Edit operations map

Edit operations map

Reversal, Transposition, Block interchange:

1. arbitrary block size 2. not once 3. non adjacent

4. permutation5. optimization

Interchange:

1. block of size 1 2. not once 3. non adjacent

4. permutation5. optimization

Generalized-swap:

1. block of size 12. once 3. non adjacent

4. repetitions 5. optimization/decision

Swap:

1. block of size 1 2. once 3. adjacent

4. repetitions 5. optimization/decision


Asynchronous pattern matching metrics

S=abacbF=bbaca

S=abacbF=bbaac

S=abacbF=bcaba

Definitions

interchange

interchange

matches

S1=bbaca

S2=bbaac

S1=bbaca

S2=bcaba

generalized-swap

matches


Generalized swap matching

Generalized Swap Matching

INPUT:text T[0..n], pattern P[0..m]

OUTPUT: all i s.t. P generalized-swap matches T[i..i+m]

Reminder:Convolution

The convolution of the strings t[1..n] and p[1..m] is the string t*p such that:

Fact: The convolution of n-length text and m-length pattern can be done in O(n log m) time using FFT.


In pattern matching

In Pattern Matching

Convolutions:

b0 b1 b2

b0 b1 b2

b0 b1 b2

O(n log m)using FFT


Asynchronous pattern matching metrics

Problem:O(n log m) only in algebraically closed fields, e.g. C.

Solution: Reduce problem to (Boolean/integer/real) multiplication. S

This reduction costs!

Example: Hamming distance.

A B A B C

A B B B A

Counting mismatches is equivalent to Counting matches


Example

Example:

1 0 1

1 0 1

1 0 1

Count all “hits” of 1 in pattern and 1 in text.


Asynchronous pattern matching metrics

For

Define:

1 if a=b

0 o/w

Example:


Asynchronous pattern matching metrics

For

Do:

+

+

Result: The number of times a in pattern matches a in text + the number of times b in pattern matches b in text + the number of times c in pattern matches c in text.


Asynchronous pattern matching metrics

Generalized Swap Matching:a Randomized Algorithm…

Idea:assign natural numbers to alphabet symbols, and construct:

T’:replacing the number a by the pair a2,-a

P’:replacing the number b by the pair b, b2.

Convolution of T’ and P’ gives at every location 2i:

j=0..mh(T’[2i+j],P’[j])

where h(a,b)=ab(a-b).

 3-degree multivariate polynomial.


Generalized swap matching a randomized algorithm

Generalized Swap Matching:a Randomized Algorithm…

Since: h(a,a)=0h(a,b)+h(b,a)=ab(b-a)+ba(a-b)=0,

a generalized-swap match  0 polynomial.

Example:

Text:ABCBAABBC

Pattern:CCAABABBB

1 -1, 4 -2, 9 -3,4 -2,1 -1,1 -1,4 -2,4 -2,9 -3

3 9, 3 9, 1 1,1 1,2 4, 1 1,2 4, 2 4,2 4

3 -9,12-18,9 -3,4 -2,2 -4,1 -1,8 -8,8 -8,18 -12


Asynchronous pattern matching metrics

Generalized Swap Matching:a Randomized Algorithm…

Problem: It is possible that coincidentally the result will be 0 even if no swap match.

Example: for text ace and pattern bdf we get a multivariate degree 3 polynomial:

We have to make sure that the probability for such a possibility is quite small.


Generalized swap matching a randomized algorithm1

Generalized Swap Matching:a Randomized Algorithm…

What can we say about the 0’s of the polynomial?

By Schwartz-Zippel Lemma prob. of 0degree/|domain|.

Conclude:

Theorem: There exist an O(n log m) algorithm that reports all generalized-swap matches and reports false matches with prob.1/n.


Generalized swap matching de randomization

Generalized Swap Matching:De-randomization?

Can we detect 0’s thus de-randomize the algorithm?

Suggestion: Take h1,…hk having no common root.

It won’t work,

k would have to be too large !


Generalized swap matching de randomization1

Generalized Swap Matching: De-randomization?…

Theorem:(m/log m) polynomial functions are required to guarantee a 0 convolution value is a 0 polynomial.

Proof: By a linear reduction from word equality.

Given: m-bit words w1w2 at processors P1P2

Construct: T=w1,1,2,…,mP=1,2,…,m,w2.

Now, T generalized-swap matches P iff w1=w2.

P1 computes:

w1 * (1,2,…,m)

log m bit result

P2 computes:

(1,2,…,m) * w2

Communication Complexity:word equality requires exchanging (m) bits,

We get: klog m=(m), so k must be (m/log m).


Interchange distance problem

Interchange Distance Problem

INPUT:text T[0..n], pattern P[0..m]

OUTPUT: The minimum number of interchanges s.t. T[i..i+m] interchange matches P.

Reminder:permutation cycle

The cycles (143) 3-cycle, (2) 1-cycle represent 3241.

Fact: The representation of a permutation as a product of disjoint permutation cycles is unique.


Interchange distance problem1

Interchange Distance Problem…

Lemma: Sorting a k-length permutation cycle requires exactly k-1 interchanges.

Proof: By induction on k.

Cases: (1), (2 1), (3 1 2)

Theorem: The interchange distance of an m-length permutation  is m-c(), where c() is the number of permutation cycles in .

Result: An O(nm) algorithm to solve the interchange distance problem.

A connection between sorting by interchanges and generalized-swap matching?


Interchange generation distance problem

Interchange Generation Distance Problem

INPUT:text T[0..n], pattern P[0..m]

OUTPUT: The minimum number of interchange-generations s.t. T[i..i+m] interchange matches P.

Definition: Let S=S1,S2,…,Sk=F, Sl+1 derived from Slvia interchange Il. An interchange-generation is a subsequence of I1,…,Ik-1 s.t. the interchanges have no index in common.

Note:Interchanges in a generation may occur in parallel.


Interchange generation distance problem1

Interchange Generation Distance Problem…

Lemma: Let  be a cycle of length k>2. It is possible to sort  in 2 generations and k-1 interchanges.

Example:(1,2,3,4,5,6,7,8,0)

generation 1:

(1,8),(2,7),(3,6),(4,5)

(8,7,6,5,4,3,2,1,0)

generation 2:

(0,8),(1,7),(2,6),(3,5)

(0,1,2,3,4,5,6,7,8)


Sorting a general cycle in two rounds

Sorting a General Cycle in Two Rounds

Algorithm:

Exchange contents of locations 2 and n

3 and n-1

4 and n-2

.

.

.

Then bring every one to place simultaneously.


Example1

EXAMPLE:

Index 635 1 4 2

Sorted 0 1 2 3 4 5

Permutation 254 0 1 3

3 45 0 1 2

0 4 5 3 1 2

0 1 5 3 4 2

0 1 2 3 4 5


Why does it work

Why Does it Work?

We want to send 0 to its place

So in the last location should be the number where 0 is

now.

This number is in location 2.

Same reasoning true for 1, 2, …


Interchange generation distance problem2

Interchange Generation Distance Problem…

  • Theorem: Let maxl() be the length of the longest permutation cycle in an m-length permutation . The interchange generation distance of is exactly:

  • 0, if maxl()=1.

  • 1, if maxl()=2.

  • 2, if maxl()>2.

Note:There is a generalized-swap match iff sorting by interchanges is done in 1 generation.


L1 distance problem

L1-Distance Problem

INPUT:text T[0..n], pattern P[0..m]

OUTPUT: The L1-distance between T[i..i+m] and P for all i[0,n-m].

Where, L1-distance(T[i..i+m],P)=|j-Ti(j)|

Example:

Text:ABCBAABBC

Pattern:CCAABABBB

How do we pair the symbols?

Do we need to try all pairings of same letters?


L1 distance problem1

L1-Distance Problem

Definition:Let T1[0..n], be a string over ∑ and T2[0..n] a permutation of T1.

A pairing between T1 and T2 is a bijection

M:{0,…,n}→{0,…,n}

Where T1[i]=T2[M(i)] , for all i=0,…,n

The L1-distance betweenT1 and T2 under M is

The L1-distance between T1 and T2 is


L1 distance problem2

L1-Distance Problem

EXAMPLE:

T1= A B B A B C A C B

M1

T2= C A C B B B B A A

dM1L1(T1,T2)=


L1 distance problem3

L1-Distance Problem

EXAMPLE:

T1= A B B A B C A C B

M1

T2= C A C B B B B A A

dM1L1(T1,T2)=8


L1 distance problem4

L1-Distance Problem

EXAMPLE:

T1= A B B A B C A C B

M1

T2= C A C B B B B A A

dM1L1(T1,T2)=8+5=13


L1 distance problem5

L1-Distance Problem

EXAMPLE:

T1= A B B A B C A C B

M1

T2= C A C B B B B A A

dM1L1(T1,T2)=8+5+2=15


L1 distance problem6

L1-Distance Problem

EXAMPLE:

T1= A B B A B C A C B

M1

T2= C A C B B B B A A

dM1L1(T1,T2)=8+5+2+4=19


L1 distance problem7

L1-Distance Problem

EXAMPLE:

T1= A B B A B C A C B

M1

T2= C A C B B B B A A

dM1L1(T1,T2)=8+5+2+4+1=20


L1 distance problem8

L1-Distance Problem

EXAMPLE:

T1= A B B A B C A C B

M1

T2= C A C B B B B A A

dM1L1(T1,T2)=8+5+2+4+1+3=23


L1 distance problem9

L1-Distance Problem

EXAMPLE:

T1= A B B A B C A C B

M1

T2= C A C B B B B A A

dM1L1(T1,T2)=8+5+2+4+1+3+5=28


L1 distance problem10

L1-Distance Problem

EXAMPLE:

T1= A B B A B C A C B

M1

T2= C A C B B B B A A

dM1L1(T1,T2)=8+5+2+4+1+3+5+7=35


L1 distance problem11

L1-Distance Problem

EXAMPLE:

T1= A B B A B C A C B

M1

T2= C A C B B B B A A

dM1L1(T1,T2)=8+5+2+4+1+3+5+7+5=40


L1 distance problem12

L1-Distance Problem

EXAMPLE:

T1= A B B A B C A C B

M2

T2= C A C B B B B A A

dM2L1(T1,T2)=


L1 distance problem13

L1-Distance Problem

EXAMPLE:

T1= A B B A B C A C B

M2

T2= C A C B B B B A A

dM2L1(T1,T2)=1


L1 distance problem14

L1-Distance Problem

EXAMPLE:

T1= A B B A B C A C B

M2

T2= C A C B B B B A A

dM2L1(T1,T2)=1+2=3


L1 distance problem15

L1-Distance Problem

EXAMPLE:

T1= A B B A B C A C B

M2

T2= C A C B B B B A A

dM2L1(T1,T2)=1+2+2=5


L1 distance problem16

L1-Distance Problem

EXAMPLE:

T1= A B B A B C A C B

M2

T2= C A C B B B B A A

dM2L1(T1,T2)=1+2+2+4=9


L1 distance problem17

L1-Distance Problem

EXAMPLE:

T1= A B B A B C A C B

M2

T2= C A C B B B B A A

dM2L1(T1,T2)=1+2+2+4+1=10


L1 distance problem18

L1-Distance Problem

EXAMPLE:

T1= A B B A B C A C B

M2

T2= C A C B B B B A A

dM2L1(T1,T2)=1+2+2+4+1+5=15


L1 distance problem19

L1-Distance Problem

EXAMPLE:

T1= A B B A B C A C B

M2

T2= C A C B B B B A A

dM2L1(T1,T2)=1+2+2+4+1+5+2=17


L1 distance problem20

L1-Distance Problem

EXAMPLE:

T1= A B B A B C A C B

M2

T2= C A C B B B B A A

dM2L1(T1,T2)=1+2+2+4+1+5+2+5=22


L1 distance problem21

L1-Distance Problem

EXAMPLE:

T1= A B B A B C A C B

M2

T2= C A C B B B B A A

dM2L1(T1,T2)=1+2+2+4+1+5+2+5+2=24

Note thatdM2L1(T1,T2)=40

How do we choose the pairing function?


L1 distance problem22

L1-Distance Problem…

Fortunately, we know how an optimal pairing looks...

lemma: Let T1,T2∑m be two strings s.t. T2 is a permutation of T1. Let M be the pairing function that, for all a and k, moves the k-th a in T1 to the location of the k-th a in T2. Then,

dL1(T1,T2)=dML1(T1,T2)

Example: An optimal pairing

Text:ABCBAABBC

Pattern:CCAABABBB

Result: An O(nm) algorithm to solve the L1-distance problem.


L1 distance problem23

L1-Distance Problem…

Proof of pairing lemma:

Note that in M, for a fixed alphabet symbol a, there are no “crossovers”.

We will show that crossovers do not help.

Cases: M M’

1)

x y z

Cost in M=x+y+y+z

Cost in M’=x+y+z+y


L1 distance problem24

L1-Distance Problem…

Proof of pairing lemma (continued…):

2) M M’

x y z

Cost in M=x+z

Cost in M’=x+y+y+z


L1 distance problem25

L1-Distance Problem…

Proof of pairing lemma (continued…):

3) M M’

x y z

Cost in M=x+z

Cost in M’=x+y+z+y


L1 distance problem26

L1-Distance Problem…

For pattern with distinct entries we can do better...

Idea:Use computations from position i to position i+1.

Example:

Text:2312331

Pattern:123

Dist(1)=|1-3|+|2-1|+|3-2|=4

Direction relative to pattern:

LeftMatchRight

2,3 1


L1 distance problem27

L1-Distance Problem…

Now we move to the next location…

Text:2312331

Pattern:123

Dist(2)=?

Direction relative to pattern:

Left (+1 to Dist) MatchRight (-1 to Dist)

2,3 1,2


L1 distance problem28

L1-Distance Problem…

Next location distance computation…

Text:2312331

Pattern:123

Dist(2)=Dist(1)+|Left|-|Right|-left most+right most=

=4+1-1-1+|3-2|=4(=|1-2|+|2-3|+|3-1|)

Result: An O(n) algorithm to solve the L1-distance problem for pattern with distinct entries.


L2 distance problem

L2-Distance Problem

Definition:Let T1[0..n], be a string over ∑ and T2[0..n] a permutation of T1.

A pairing between T1 and T2 is a bijection

M:{0,…,n}→{0,…,n}

Where T1[i]=T2[M(i)] , for all i=0,…,n

The L2-distance betweenT1 and T2 under M is

The L2-distance between T1 and T2 is


L2 distance problem1

L2-Distance Problem

INPUT:text T[0..n], pattern P[0..m]

OUTPUT: The L2-distance between T[i..i+m] and P for all i[0,n-m].

Where, L2-distance(T(i),P)=|j-MT(i)(j)|2

Do we need to try all pairings of same letters?

No, the pairing lemma works here too.

Result: An O(nm) algorithm to solve the L2-distance problem.

We can do better…


L2 distance problem2

L2-Distance Problem…

Idea:Consider T and P of same length.

Let lista(P) and lista(T) for each letter a, be the list of locations in which a occurs in P and T.

The L2-distance between T and P is:

dL2(T,P)=aPj[0,…|list(a)|](lista(P)[j]-lista(T(i))[j])2

Schematically: Fix letter a:

T

P

list(T) = i1, i2, …, ik

list(P) = j1, j2, …, jk


L2 distance problem3

L2-Distance Problem…

easy

to

compute

easy

to

compute

convolution


L2 distance problem large text

L2-Distance Problem…large text

Idea:Consider lista(P) and lista(T(i)) for each letter a, the list of locations in which a occurs in P and T(i).

The L2-distance between T(i) and P is:

dL2(T(i),P)=aPj[0,…|list(a)|](lista(P)[j]-lista(T(i))[j])2

Now, we want to use lista(T) instead of lista(T(i)).

Now the text location indices are not fixed…


L2 distance problem large text1

L2-Distance Problem…large text

Schematically: Fix letter a:

T x

P

list(T) = i1, i2, …, ik,...,ir

list(P) = j1, j2, …, jk

For location x, the text indices need to be i2-x, i3-x, …

Convolution formula:


L2 distance problem large text2

L2-Distance Problem…large text

convolution

easy to

compute

easy to compute


L2 distance problem large alphabet

L2-Distance Problem…large alphabet

Problem: What happens when unbounded number of alphabet symbols? How many convolutions?

Let ∑={a1,…,ak} and let ni be the number of times ai occurs in T, i=1,…,k.

Clearly, n1+n2+…+nk=n.

Time: For each symbol ai:

Total Time:

Result: An O(n log m) algorithm to solve the L2-distance problem.


Open problems

Open Problems

1. Interchange distance faster than O(nm)?

2. Asynchronous communication – different errors in address bits.

3. Different error measures than interchange/block interchange/transposition/reversals for errors arising from address bit errors.

Note:The techniques employed in asynchronous pattern matching have so far proven new and different from traditional pattern matching.


The end

The End


  • Login