1 / 85

# Sequence Data Mining: Techniques and Applications - PowerPoint PPT Presentation

Sequence Data Mining: Techniques and Applications. Sunita Sarawagi IIT Bombay http://www.it.iitb.ac.in/~sunita. Mark Craven University of Wisconsin http://www.biostat.wisc.edu/~craven. What is a sequence?. Ordered set of elements: s = a 1 ,a 2 ,..a n Each element a i could be

Related searches for Sequence Data Mining: Techniques and Applications

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

## PowerPoint Slideshow about 'Sequence Data Mining: Techniques and Applications' - marjorie

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

### Sequence Data Mining: Techniques and Applications

Sunita Sarawagi

IIT Bombay

http://www.it.iitb.ac.in/~sunita

Mark Craven

University of Wisconsin

http://www.biostat.wisc.edu/~craven

• Ordered set of elements: s = a1,a2,..an

• Each element ai could be

• Numerical

• Categorical: domain a finite set of symbols S, |S|=m

• Multiple attributes

• The length n of a sequence is not fixed

• Order determined by time or position and could be regular or irregular

• Classical applications

• Speech: sequence of phonemes

• Language: sequence of words and delimiters

• Handwriting: sequence of strokes

• Bioinformatics:

• Genes: Sequence of 4 possible nucleotides, |S|=4

• Example: AACTGACCTGGGCCCAATCC

• Proteins: Sequence of 20 possible amino-acids, |S|=20

• Example:

• Telecommunications: alarms, data packets

• Retail data mining: Customer behavior, sessions in a e-store (Example, Amazon)

• Intrusion detection

• Intrusions could be detected at

• Network-level (denial-of-service attacks, port-scans, etc) [KDD Cup 99]

• Sequence of TCP-dumps

• Host-level (attacks on privileged programs like lpr, sendmail)

• Sequence of system calls

• |S| = set of all possible system calls ~100

open

lseek

lstat

mmap

execve

ioctl

ioctl

close

execve

close

• Traditional mining operations on sequences

• Classification

• Clustering

• Finding repeated patterns

• Primitives for handling sequence data

• Sequence-specific mining operations

• Partial sequence classification (Tagging)

• Segmenting a sequence

• Predicting next symbol of a sequence

• Applications in biological sequence mining

Given:

• a set of classes C and

• a number of example sequences in each class,

train a model so that for an unseen sequence we can say to which class it belongs

Example:

• Given a set of protein families, find family of a new protein

• Given a sequence of packets, label session as intrusion or normal

• Given several utterances of a set of words, classify a new utterance to the right word

• Conventional classification methods assume

• record data: fixed number of attributes

• single record per instance to be classified (no order)

• Sequences:

• variable length, order important.

• Adapting conventional classifiers to sequences

• Generative classifiers

• Boundary-based classifiers

• Distance based classifiers

• Kernel-based classifiers

• For each class i,

train a generative model Mi to

maximize likelihood over all

training sequences in the class i

• Estimate Pr(ci) as fraction of training instances in class i

• For a new sequence x,

• find Pr(x|ci)*Pr(ci) for each i using Mi

• choose i with the largest value of Pr(x|ci)*P(ci)

Pr(x|c1)*Pr(c1)

x

Pr(x|c2)*Pr(c2)

Pr(x|c3)*Pr(c3)

Need a generative model for sequence data

• Data: points in a fixed multi-dimensional space

• Output of training: Boundaries that define regions within which same class predicted

• Application: Tests on boundaries to find region

Decision trees

Neural networks

Linear discriminants

Need to embed sequence data in a fixed coordinate space

• Define function K(xi, x) that intuitively defines similarity between two sequences and satisfies two properties

• K is symmetric i.e., K(xi, x) = K(x, xi)

• K is positive definite

• Training: learn for each class c,

• wicfor each train sequence xi

• bc

• Application: predict class of x

• For each class c, find f(x,c) = S wicK(xi, x)+bc

• Predicted class is c with highest value f(x,c)

• Well-known kernel classifiers

• Nearest neighbor classifier

• Support Vector Machines

Need kernel/similarity function

• Given a set of sequences, create groups such that similar sequences in the same group

• Three kinds of clustering algorithms

• Distance-based:

• K-means

• Various hierarchical algorithms

• Model-based algorithms

• Expectation Maximization algorithm

• Density-based algorithms

• Clique

Need similarity function

Need generative models

Need dimensional embedding

• Primitives for handling sequence data

• Embed sequence in a fixed dimensional space

• All conventional record mining techniques will apply

• Generative models for sequence

• Sequence classification: generative methods

• Clustering sequences: model-based approach

• Distance between two sequences

• Sequence classification: SVM and NN

• Clustering sequences: distance-based approach

• Sequence-specific mining operations

• Applications

• Ignore order, each symbol maps to a dimension

• extensively used in text classification and clustering

• Extract aggregate features

• Real-valued elements: Fourier coefficients, Wavelet coefficients, Auto-regressive coefficients

• Categorical data: number of symbol changes

• Sliding window techniques (k: window size)

• Define a coordinate for each possible k-gram a (mk coordinates)

• a-th coordinate is number of times a in sequence

• (k,b) mismatch score: a-th coordinate is number of k-grams in sequence with b mismatches with a

• Define a coordinate for each of the k-positions

• Multiple rows per sequence

Sliding window examples

open

lseek

ioctl

mmap

execve

ioctl

ioctl

open

execve

close

mmap

One symbol per column

One row per trace

Multiple rows per trace

mis-match scores: b=1

• Short sequences of system calls made during normal execution of system calls are very consistent, yet different from the sequences of its abnormal executions

• Two approaches

• STIDE (Warrender 1999)

• Create dictionary of unique k-windows in normal traces, count what fraction occur in new traces and threshold.

• RIPPER –based (Lee 1998)

• next...

• When both normal and abnormal data available

• class label = normal/abnormal:

• When only normal trace,

• class-label=k-th system call

Learn rules to predict class-label [RIPPER]

• Both-traces:

• if the 2nd system call is vtimes and the 7th is vtrace, then the sequence is “normal”

• if the 6th system call is lseek and the 7th is sigvec, then the sequence is “normal”

• if none of the above, then the sequence is “abnormal”

• Only-normal:

• if the 3rd system call is lstat and the 4th is write, then the 7th is stat

• if the 1st system call is sigblock and the 4th is bind, then the 7th is setsockopt

• if none of the above, then the 7th is open

• The output rule sets contain ~250 rules, each with 2 or 3 attribute tests

• Score each trace by counting fraction of mismatches and thresholding

Summary: Only normal traces sufficient to detect intrusions

Percent of mismatching traces

• Different programs need different thresholds

• Simple methods (e.g. STIDE) work as well

• Results sensitive to window size

• Is it possible to do better with sequence specific methods?

[from Warrender 99]

• Primitives for handling sequence data

• Embed sequence in a fixed dimensional space

• All conventional record mining techniques will apply

• Distance between two sequences

• Sequence classification: SVM and NN

• Clustering sequences: distance-based approach

• Generative models for sequences

• Sequence classification: whole and partial

• Clustering sequences: model-based approach

• Sequence-specific mining operations

• Applications in biological sequence mining

• Independent model

• One-level dependence (Markov chains)

• Fixed memory (Order-l Markov chains)

• Variable memory models

• More complicated models

• Hidden Markov Models

• Model structure

• A parameter for each symbol in S

• Probability of a sequence s being generated from the model

• example: Pr(AACA)

= Pr(A) Pr(A) Pr(C) Pr(A) = Pr(A)3 Pr(C)

= 0.13 ´0.9

• Training transition probabilities

• Data T : set of sequences

• Count(s ε T): total number of times substring s appears in training data T

Pr(s) = Count(sε T) / length(T)

Pr(A) = 0.1

Pr(C) = 0.9

• Model structure

• A state for each symbol in S

• Edges between states with probabilities

• Probability of a sequence s being generated from the model

• Example: Pr(AACA)

= Pr(A) Pr(A|A) Pr(C|A) Pr(A|C)

= 0.5 ´ 0.1 ´ 0.9 ´ 0.4

• Training transition probability between states

Pr(s|b) = Count(bsε T) / Count(bε T)

start

0.5

0.5

0.4

A

C

0.9

0.6

0.1

0.5

Higher order Markov Chains

l = memory of sequence

• Model

• A state for each possible suffix of length l |S|lstates

• Edges between states with probabilities

• Pr(AACA)

= Pr(AA)Pr(C|AA) Pr(A|AC)

= 0.5 ´ 0.9 ´ 0.7

• Training model

Pr(s|s) = count(ssε T) / count(s ε T)

C 0.9

AA

AC

C 0.6

A 0.1

C 0.3

A 0.4

A 0.7

CA

CC

0.8

C 0.2

l = 2

start

• Probabilistic Suffix Automata (PSA)

• Model

• States: substrings of size no greater than lwhere no string is suffix of another

• Calculating Pr(AACA):

• Pr(A)Pr(A|A)Pr(C|A)Pr(A|AC)

• 0.5 ´ 0.3 ´ 0.7 ´ 0.1

• Training: not straight-forward

• Eased by Prediction Suffix Trees

• PSTs can be converted to PSA after training

0.2

0.5

C 0.9

C 0.6

AC

CC

C 0.7

A 0.6

A 0.1

A

A 0.3

• Suffix trees with emission probabilities of observation attached with each tree node

• Linear time algorithms exist for constructing such PSTs from training data [Apostolico 2000]

C 0.6

0.28, 0.72

e

C 0.9

AC

CC

0.3, 0.7

A

C

0.25, 0.75

C 0.7

A 0.1

A

AC

CC

0.4, 0.6

0.1, 0.9

Pr(AACA)=0.28 ´ 0.3 ´ 0.7 ´ 0.1

A 0.3

0.9

0.5

0.1

0.8

0.2

Hidden Markov Models

A

C

0.6

0.4

• Doubly stochastic models

• Efficient dynamic programming algorithms exist for

• Finding Pr(S)

• The highest probability path P that maximizes Pr(S|P) (Viterbi)

• Training the model

• (Baum-Welch algorithm)

A

C

0.9

0.1

S1

S2

S4

S3

A

C

0.3

0.7

A

C

0.5

0.5

• Models trained to maximize likelihood of data might perform badly when

• Model not representative of data

• Training data insufficient

• Alternatives to Maximum-likelihood/EM

• Objective functions:

• Minimum classification error

• Maximum posterior probability of actual label Pr(c|x)

• Maximum mutual information with class

• Harder to train above functions, number of alternatives to EM proposed

• Generalized probabilistic descent [Katagiri 98]

• Deterministic annealing [Rao 01]

• Training:

• Initial number of states = 40 (roughly equals number of distinct system calls)

• Train on normal traces

• Testing:

• Need to handle variable length and online data

• For each call, find the total probability of outputting given all calls before it.

• If probability below a threshold call it abnormal.

• Trace is abnormal if fraction of abnormal calls are high

• HMMs

• Take longer time to train

• Less sensitive to thresholds, no window parameter

• Best overall performance

• VMM and Sparse Markov Transducers also shown to perform significantly better than fixed window methods [Eskin 01]

[from Warrender 99]

[from Warrender 99]

• Primitives for handling sequence data

• Sequence-specific mining operations

• Partial sequence classification (Tagging)

• Segmenting a sequence

• Predicting next symbol of a sequence

• Applications in biological sequence mining

• The tagging problem:

• Given:

• A set of tags L

• Training examples of sequences showing the breakup of the sequence into the set of tags

• Learn to breakup a sequence into tags

• (classification of parts of sequences)

• Examples:

• Text segmentation

• Break sequence of words forming an address string into subparts like Road, City name etc

• Continuous speech recognition

• Identify words in continuous speech

Journal

Year

Author

Volume

Page

Text sequence segmentation

House number

Zip

State

Building

City

4089 Whispering Pines Nobel Drive San Diego CA 92122

P.P.Wangikar, T.P. Graycar, D.A. Estell, D.S. Clark, J.S. Dordick (1993) Protein and Solvent Engineering of Subtilising BPN' in Nearly Anhydrous Organic Media J.Amer. Chem. Soc. 115, 12231-12237.

• Learn to identify start and end boundaries of each label

• Method: for each label, build two classifiers for accepting its two boundaries.

• Any conventional classifier would do:

• Rule-based most common.

• K-windows approach:

• For each label, train a classifier on k-windows

• During testing

• classify each k-window

• Adapt state-based generative models like HMM

S1

S2

Prefix

Suffix

S4

S3

Building name

S1

S2

Prefix

Suffix

S4

S3

State-based models for sequence tagging

• Two approaches:

• Separate model per tag with special prefix and suffix states to capture the start and end of a tag

Nested model

Each element another HMM

• Example: IE

• Naïve Model: One state per element

… Mahatma Gandhi Road Near Parkland ...

… [Mahatma Gandhi Road Near : Landmark] Parkland ...

• Disadvantages of generative models (HMMs)

• Maximizing joint probability of sequence and labels may not maximize accuracy

• Conditional independence of features a restrictive assumption

• Alternative: Conditional Random Fields

• Maximize conditional probability of labels given sequence

• Arbitrary overlapping features allowed

• Primitives for handling sequence data

• Sequence-specific mining operations

• Partial sequence classification (Tagging)

• Segmenting a sequence

• Predicting next symbol of a sequence

• Applications in biological sequence mining

Basic premise: Sequence assumed to be generated from multiple models

Goal: discover boundaries of model change

Applications:

Bioinformatics

Better models by breaking up heterogeneous sequences

Return

A

Pick

B

A has a set of coins with different biases

A repeatedly

Picks arbitrary coin

Tosses it arbitrary number of times

B observes H/T

Guesses transition points and biases

Simpler problem: Segmenting a 0/1 sequence

Return

Pick

A

Toss

B

• Can assume n different coins with bias 0 or 1

• Data fits perfectly (with probability one)

• Many coins needed

• Or assume one coin

• May fit data poorly

• “Best segmentation” is a compromise between model length and data fit

5/7

1/3

1/4

For each segment i:

L(Mi): cost of model parameters: log(Pr(head))

+ segment boundary: log (sequence length)

L(Di|Mi): cost of describing data in segment Si given model Mi: log(Hh T(1-h)) H: #heads, T: #tails

Goal: find segmentation that leads to smallest total cost

åsegment i L(Mi) + L(Di|Mi)

How to find optimal segments

Sequence of 17 tosses:

Derived graph with 18 nodes and all possible edges

Edge cost = model cost

+ data cost

• In previous method each segment is assumed to be independent of each other, does not allow model reuse of the form:

• The (k,h) segmentation problem:

• Assume a fixed number h of models is to be used for segmenting into k parts an n-element sequence (k > h)

• (k,k) segmentation solvable by dynamic programming

• General (k,h) problem NP hard

S3

S4

S1

Approximations: for (k,h) segmentation

• Solve (k,k) to get segments

• Solve (n,h) to get H models

• Example: 2-medians

• Assign each segment to best of H

A second variant (Replace step 2 above with)

• Find summary statistics for each k segment, cluster them into H clusters

M1

M2

From:

Gionis & Mannila2003

• Traditional mining operations on sequences

• Primitives for handling sequence data

• Sequence-specific mining operations

• Applications in biological sequence mining

• Classifying proteins according to family

• Finding genes in DNA sequences

BREAK

Given: amino-acid sequence of a protein

Do: predict the family to which it belongs

GDLSTPDAVMGNPKVKAHGKKVLGAFSDGLAHLDNLKGTFATLSELHCDKLHVDPENFRLLGNVCVLAHHFGKEFTPPVQAAYAKVVAGVANALAHKYH

http://www.ornl.gov/hgmis

• A protein family is…

Figure from the DOE Human Genome Program

http://www.ornl.gov/hgmis

• The amino-acid sequence of a protein determines its structure

• The structure of a protein determines its function

• Proteins play many key roles in cells

• structural support

• storage of amino acids

• transport of other substances

• coordination of an organism’s activities

• response of cell to chemical stimuli

• movement

• protection against disease

• selective acceleration of chemical reactions

• The SCOP and CATH databases provide hierarchical taxonomies of protein families

The sequences in a family may vary in length

Some positions are more conserved than others

Figure from www-cryst.bioc.cam.ac.uk/~max/res_globin.html

R 0.12

D 0.04

N 0.29

C 0.01

E 0.03

Q 0.02

G 0.01

d

d

d

3

2

1

i

i

i

i

1

2

3

0

start

end

Insert and match states have

emission distributions over

sequence characters

m

m

2

m

3

1

Profile HMMs

• Profile HMMs are commonly used to model families of sequences

Delete states are silent; they

Account for characters missing

in some sequences

Insert states account

for extra characters

in some sequences

Match states represent

key conserved positions

β-galactosidase

β-glucanase

β-amylase

α-amylase

Profile HMMs

• To classify sequences according to family, we can train a profile HMM to model the proteins of each family of interest

• Given a sequence x, use Bayes’ rule to make classification

• Consider computing in a profile HMM

• The probability that the path is taken and the sequence is generated:

transition

probabilities

emission

probabilities

end

How Likely Is A Given Sequence?

0.4

0.2

A 0.4

C 0.1

G 0.2

T 0.3

A 0.2

C 0.3

G 0.3

T 0.2

0.8

0.6

0.5

1

3

0

5

A 0.4

C 0.1

G 0.1

T 0.4

A 0.1

C 0.4

G 0.4

T 0.1

0.5

0.9

0.2

2

4

0.1

0.8

• the probability over all paths is:

• but the number of paths can be exponential in the length of the sequence...

• the Forward algorithm enables us to compute this efficiently using dynamic programming

• define to be the probability of being in state k having observed the first i characters of x

• we want to compute , the probability of being in the end state having observed all of x

• can define this recursively

probability that we’re in start state and

have observed 0 characters from the sequence

• recursion for emitting states (i =1…L):

• recursion for silent states:

probability that we’re in the end state and

have observed the entire sequence

• termination:

The Forward Algorithm

• The parameters in profile HMMs are typically trained using the Baum-Welch method (an EM algorithm)

• Heuristic methods are used to determine the length of the model

• Initialize using an alignment of the training sequences (as in globin example)

• Iteratively make local adjustments to length if delete or insert states are used “too much” for training sequences

The Fisher kernel method for protein classification

• Standard HMMs are generative models

• Training involves estimating

• Predictions based on are made using Bayes’ rule

• Sometimes we can get more accurate predictions using discriminative methods which try to optimize directly

• One example: the Fisher kernel method [Jaakola et al. ‘00]

The Fisher kernel method for protein classification

• Consider learning to discriminate proteins in class from proteins other families

• Train an HMM for

• Use HMM to map each protein sequence x into a fixed-length vector

• Train an support vector machine (SVM) whose kernel function is the Euclidean distance between vectors

• The resulting discriminative model is given by

• The Fisher kernel method for protein classification

• How can we get an informative fixed-length vector? Compute the Fisher score

i.e. each component of is a derivative of the log-likelihood of x with respect to a particular parameter in the HMM

Figure from Jaakola et al., ISMB 1999

• classifying 2447proteins into 33 families

• x-axis represents the median fraction of negative sequences that score as high as a positive sequence for a given family’s model

profile HMMs w/

Fisher kernel SVM

profile HMMs

BLAST-based

methods

Given: an uncharacterized DNA sequence

Do: locate the genes in the sequence, including the coordinates of individual exons and introns

image from the UCSC Genome Browser

http://genome.ucsc.edu/

http://www.ornl.gov/hgmis

region

intergenic

region

exon

intron

exon

intron

exon

ATG GAA ACC CGA TCG GGC … AC

G TAA AGT CTA …

The structure of genes

• Genes consist of alternating sequences of exons and introns

• Introns are spliced out before the gene is translated into protein

• Exons consist of three-letter words, called codons

• Each codon encodes a single amino acid (character in a protein sequence)

the different ways an intron can interrupt

a coding sequence (after 1st base in codon,

after 2nd base or after 3rd base)

Complementary submodel

(not shown) detects genes on

opposite DNA strand

The GENSCAN HMM for gene finding[Burge & Karlin ‘97]

Each shape represents a functional unit

of a gene or genomic region

• For each sequence type, GENSCAN models

• the length distribution

• the sequence composition

• Length distribution models vary depending on sequence type

• Nonparametric (using histograms)

• Parametric (using geometric distributions)

• Fixed-length

• Sequence composition models vary depending on type

• 5th-order, inhomogeneous

• 5th -order homogenous

• Independent and 1st-order inhomogeneous

• Tree-structured variable memory

• For exons, GENSCAN uses

• Histograms to represent exon lengths

• 5th-order, inhomogeneous Markov models to represent exon sequences

• 5th-order, inhomogeneous models can represent statistics about pairs of neighboring codons

A 5th-order Markov model for DNA

AAAAA

CTACA

Pr(A | GCTAC)

CTACC

start

CTACG

CTACT

Pr(GCTAC)

GCTAC

TTTTT

G C T A C G

Gis in 3rd codon position

Gis in 1st position

C T A C G G

Ais in 2nd position

T A C G G A

Markov models for exons

• for each “word” we evaluate, we’ll want to consider its position with respect to the assumed codon framing

• thus we’ll want to use an inhomogenous model to represent genes

AAAAA

AAAAA

start

TTTTT

TTTTT

TTTTT

A 5th-order inhomogeneous model

Transitions go to

states in position 1

CTACA

CTACA

CTACC

CTACC

CTACG

CTACG

TACAA

CTACT

CTACT

TACAC

TACAG

TACAT

GCTAC

GCTAC

position 1

position 2

position 3

given: an uncharacterized DNA sequence

do: find the most probable path through the model for the sequence

• This path will specify the coordinates of the predicted genes (including intron and exon boundaries)

• The Viterbi algorithm is used to compute this path

Finding the most probable path: the Viterbi algorithm

• define to be the probability of the most probable path accounting for the first i characters of x and ending in state k

• we want to compute , the probability of the most probable path accounting for all of the sequence and ending in the end state

• can define recursively

• can use dynamic programming to find efficiently

Finding the most probable path: the Viterbi algorithm

• initialization:

• recursion for emitting states (i =1…L):

keep track of most

probable path

• recursion for silent states:

• to recover the most probable path, follow pointers back starting at

• termination:

The Viterbi path represents

a parse of a given sequence,

predicting exons, introns, etc

ACCGTTACGTGTCATTCTACGTGATCATCGGATCCTAGAATCATCGATCCGTGCGATCGATCGGATTAGCTAGCTTAGCTAGGAGAGCATCGATCGGATCGAGGAGGAGCCTATATAAATCAA

Some lessons from these biological applications

• HMMs provide state-of-the-art performance in protein classification and gene finding

• HMMs can naturally classify and parse sequences of variable length

• Much domain knowledge can be incorporated into the structure of the models

• Types, lengths and ordering of sequence features

• Appropriate amount of memory to represent various sequence features

• Models can vary representation across sequence features

• Discriminative methods often provide superior predictive accuracy to generative methods

• S. F. Altschul, T. L. Madden, A. A. Schaffer, J. Zhang, Z. Zhang, W. Miller, and D. J. Lipman. Gapped BLAST and PSI-BLAST: A new generation of protein database search programs. Nucleic Acids Research, 25:3389–3402, 1997.

• Apostolico, A., and Bejerano, G. 2000. Optimal amnesic probabilistic automata or how to learn and classify proteins in linear time and space. In Proceedings of RECOMB2000. http://citeseer.nj.nec.com/apostolico00optimal.html

• Vinayak R. Borkar, Kaustubh Deshmukh, and Sunita Sarawagi. Automatic text segmentation for extracting structured records. SIGMOD 2001.

• C. Burge and S. Karlin, Prediction of Complete Gene Structures in Human Genomic DNA. Journal of Molecular Biology, 268:78-94, 1997.

• Mary Elaine Calif and R. J. Mooney. Relational learning of pattern-match rules for information extraction. AAAI 1999.

• S. Chakrabarti, S. Sarawagi and B.Dom, Mining surprising patterns using temporal description length,VLDB, 1998

• M. Collins, “Discriminitive training method for Hidden Markov Models: Theory and experiments with perceptron algorithms, EMNLP 2002

• R. Durbin, S. Eddy, A. Krogh, and G. Mitchison, Biological sequence analysis: probabilistic models of proteins and nucleic acids, Cambridge University Press, 1998.

• Eleazar Eskin, Wenke Lee and Salvatore J. Stolfo. ``Modeling System Calls for Intrusion Detection with Dynamic Window Sizes.'' Proceedings of DISCEX II. June 2001.

• IDS http://www.cs.columbia.edu/ids/publications/

• D Freitag and A McCallum, Information Extraction with HMM Structures Learned by Stochastic Optimization, AAAI 2000

• Gionis and H. Mannila: Finding recurrent sources in sequences. ACM ReCOMB 2003

• Michael T. Goodrich, Efficient Piecewise-Approximation Using the Uniform Metric Symposium on Computational Geometry , (1994)

• D. Haussler. Convolution kernels on discrete structure. Technical report, UC Santa Cruz, 1999.

• K. Karplus, C. Barrett, and R. Hughey, Hidden Markov models for detecting remote protein homologies. Bioinformatics 14(10): 846-856, 1998.

• L. Lo Conte, S. Brenner, T. Hubbard, C. Chothia, and A. Murzin. SCOP database in 2002: refinements accommodate structural genomics. Nucleic Acids Research, 30:264-267, 2002

• Wenke Lee and Sal Stolfo. ``Data Mining Approaches for Intrusion Detection'' In Proceedings of the Seventh USENIX Security Symposium (SECURITY '98), San Antonio, TX, January 1998

• A. McCallum and D. Freitag and F. Pereira, Maximum entropy Markov models for information extraction and segmentation, ICML-2000

• Rabiner, Lawrence R., 1990. A tutorial on hidden Markov models and selected applications in speech recognition. In Alex Weibel and Kay-Fu Lee (eds.), Readings in Speech Recognition. Los Altos, CA: Morgan Kaufmann, pages 267--296.

• D. Ron, Y. Singer and N. Tishby. The power of amnesia: learning probabilistic automata with variable memory length. Machine Learning, 25:117-- 149, 1996

• Warrender, Christina, Stephanie Forrest, and Barak Pearlmutter. Detecting Intrusions Using System Calls: Alternative Data Models. To appear, 1999 IEEE Symposium on Security and Privacy. 1999