accelerating read mapping with fasthash n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Accelerating Read Mapping with FastHASH PowerPoint Presentation
Download Presentation
Accelerating Read Mapping with FastHASH

Loading in 2 Seconds...

play fullscreen
1 / 34

Accelerating Read Mapping with FastHASH - PowerPoint PPT Presentation


  • 63 Views
  • Uploaded on

Accelerating Read Mapping with FastHASH. Hongyi Xin † Donghyuk Lee † Farhad Hormozdiari ‡ Samihan Yedkar † Can Alkan § Onur Mutlu † † Carnegie Mellon University § University of Washington ‡ University of California Los Angeles. Outline. Read Mapping and i ts Challenges

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

Accelerating Read Mapping with FastHASH


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
accelerating read mapping with fasthash

Accelerating Read Mapping with FastHASH

HongyiXin†Donghyuk Lee†FarhadHormozdiari ‡ SamihanYedkar†Can Alkan§OnurMutlu†

† Carnegie Mellon University § University of Washington

‡ University of California Los Angeles

outline
Outline
  • Read Mapping and its Challenges
  • Hash Table-Based Mappers
  • Problem and Goal
  • Key Observations
  • Mechanisms
  • Results
  • Conclusion
outline1
Outline
  • Read Mapping and its Challenges
  • Hash Table-Based Mappers
  • Problem and Goal
  • Key Observations
  • Mechanisms
  • Results
  • Conclusion
read mapping
Read Mapping
  • A post-processing procedure after DNA sequencing
  • Map many short DNA fragments (reads) to a known reference genome with some minor differences allowed

Reference genome

Mapping short reads to reference genome is challenging (billions of 50-300 base pair reads)

Reads

DNA, physically

DNA, logically

challenges
Challenges
  • Need to find many mappings of each read
    • A short read may map to many locations, especially with Next Generation DNA Sequencing
    • How can we find all mappings efficiently?
  • Need to tolerate small variances/errors in each read
    • Each individual is different: Subject’s DNA may slightly differ from the reference (Mismatches, insertions, deletions)
    • How can we efficiently map each read with up to e errors present?
  • Need to map each read very fast (i.e., performance is important)
    • Human DNA is 3.2 billion base pairs long  Millions to billions of reads (State-of-the-art mappers take weeks to map a human’s DNA)
    • How can we design a much higher performance read mapper?
outline2
Outline
  • Read Mapping and its Challenges
  • Hash Table-Based Mappers
    • Preprocess the reference into a Hash Table
    • Use Hash Table to map reads
  • Problem and Goal
  • Key Observations
  • Mechanisms
  • Results
hash table based mappers alkan ng 09
Hash Table-Based Mappers [Alkan+ NG’09]

Location list—where the k-mer

occurs in reference gnome

k-mer or 12-mer

AAAAAAAAAAAA

Reference genome

AAAAAAAAAAAC

NULL

AAAAAAAAAAAT

......

CCCCCCCCCCCC

......

......

......

TTTTTTTTTTTT

Once for a reference

outline3
Outline
  • Read Mapping and its Challenges
  • Hash Table-Based Mappers
    • Preprocess the reference into a Hash Table
    • Use Hash Table to map reads
  • Problem and Goal
  • Key Observations
  • Mechanisms
  • Results
hash table based mappers alkan ng 091
Hash Table-Based Mappers [Alkan+ NG’09]

AAAAAAAAAAAACCCCCCCCCCCCTTTTTTTTTTT

read

k-mers

TTTTTTTTTTTT

CCCCCCCCCCCC

AAAAAAAAAAAA

Reference Genome

Hash Table (HT)

12

324

***

Invalid mapping

Valid mapping

AAAAAAAAAAAA

..****************************************..

…AAAAAAAAAAAACCCCCCCCCCCCTTTTTTTTTTTT…

.. AAAAAAAAAAAAAACGCTTCCACCTTAATCTGGTTG..

CCCCCCCCCCCC

AAAAAAAAAAAACCCCCCCCCCCCTTTTTTTTTTTT

TTTTTTTTTTTT

read

Verification/Local Alignment

advantages of hash table based mappers
Advantages of Hash Table Based Mappers
  • + Guaranteed to find all mappings
  • + Tolerate up to eerrors
outline4
Outline
  • Read Mapping and its Challenges
  • Hash Table-Based Mappers
  • Problem and Goal
  • Key Observations
  • Mechanisms
  • Results
  • Conclusion
problem and goal
Problem and Goal
  • Poor performance of existing read mappers: Very slow
    • Verification/alignment takes too long to execute
    • Verification requires a memory access for reference genome + many base-pair wise comparisons between the reference and the read
  • Goal: Speed up the mapper by reducing the cost of verification

95%

reducing the cost of verification
Reducing the Cost of Verification
  • We observe that most verification calculations are unnecessary
    • 1 out of 1000 potential locations passes the verification process
  • We also observe that we can get rid of unnecessary verification calculations by
    • Detecting and rejecting earlyinvalid mappings
    • Reducingthe number of potential mappings
outline5
Outline
  • Read Mapping and its Challenges
  • Hash Table-Based Mappers
  • Problem and Goal
  • Key Observations
  • Mechanisms
  • Results
  • Conclusion
key observations
Key Observations
  • Observation 1
    • Adjacent k-mers in the read should also be adjacent in the reference genome
    • Hence, mapper can quickly reject mappings that do not satisfy this property
  • Observation 2
    • Some k-mers are cheaper to verify than othersbecause they have shorter location lists (they occur less frequently in the reference genome)
      • Mapper needs to examine only e+1 k-mers’ locations to tolerate eerrors
    • Hence, mapper can choose the cheapest e+1k-mersand verify their locations
outline6
Outline
  • Read Mapping and its Challenges
  • Hash Table-Based Mappers
  • Problem and Goal
  • Key Observations
  • Mechanisms
  • Results
  • Conclusion
fasthash mechanisms
FastHASH Mechanisms
  • Adjacency Filtering (AF): Rejects obviously invalid mapping locations at early stage to avoid unnecessary verifications
  • Cheap K-mer Selection (CKS): Reduces the absolute number of potential mapping locations
adjacency filtering af
Adjacency Filtering (AF)
  • Goal:detect invalid mappings at early stage
  • Key Insight:For a valid mapping, adjacent k-mers in the read are also adjacent in the reference genome
  • Key Idea: search for adjacent locations in the k-mers’ location lists
    • If more than ek-mers fail—there must be more than e errors—invalid mapping

read

AAAAAAAAAAAACCCCCCCCCCCCTTTTTTTTTTT

Reference genome

Invalid mapping

Valid mapping

adjacency filtering af1
Adjacency Filtering (AF)

read

AAAAAAAAAAAACCCCCCCCCCCCTTTTTTTTTTT

+12

+24

k-mers

TTTTTTTTTTTT

CCCCCCCCCCCC

AAAAAAAAAAAA

Reference Genome

Hash Table (HT)

12

940

***

557

324

569?

952?

336?

36?

24?

AAAAAAAAAAAA

…AAAAAAAAAAAACCCCCCCCCCCCTTTTTTTTTTTT…

CCCCCCCCCCCC

AAAAAAAAAAAACCCCCCCCCCCCTTTTTTTTTTTT

TTTTTTTTTTTT

fasthash mechanisms1
FastHASH Mechanisms
  • Adjacency Filtering (AF): Rejects obviously invalid mapping locations at early stage to avoid unnecessary verifications
  • Cheap K-mer Selection (CKS): Reduces the absolute number of potential mapping locations
cheap k mer selection cks
Cheap K-mer Selection (CKS)
  • Goal:Reduce the number of potential mappings
  • Key insight:
    • K-mers have different cost to examine: Some k-mers are cheaperas they have fewer locations than others (occur less frequently in reference genome)
  • Key idea:
    • Sort the k-mers based on their number of locations
    • Select the k-mers with fewest locations to verify
cheap k mer selection
Cheap K-mer Selection

read

  • e=2(examine 3 k-mers)

AAGCTCAATTTCCCTCCTTAATTTTCCTCTTAAGAA GGGTATGGCTAG AAGGTTGAGAGCCTTAGGCTTACC

Locations

Number of Locations

Expensive 3 k-mers

Cheapest 3 k-mers

Previous work needs to verify:

3004 locations

FastHASH verifies only:

8 locations

outline7
Outline
  • Read Mapping and its Challenges
  • Hash Table-Based Mappers
  • Problem and Goal
  • Key Observations
  • Mechanisms
  • Results
  • Conclusion
methodology
Methodology
  • Implemented FastHASH on top of state-of-the-art mapper: mrFAST
    • New version mrFAST-2.5.0.0 over mrFAST-2.1.0.6
  • Tested with real read sets generated from Illumina platform
    • 1M reads of a human (160 base pairs)
    • 500K reads of a chimpanzee (101 base pairs)
    • 500K reads of a orangutan (70 base pairs)
  • Tested with simulated reads generated from reference genome
    • 1M simulated reads of human (180 base pairs)
  • Evaluation system
    • Intel Core i7 Sandy Bridge machine
    • 16 GB of main memory
fasthash speedup
FastHASH Speedup

human

19x

chimpanzee

orangutan

simulated

With FastHASH, new mrFAST obtains up to 19x speedup over previous version, without losing valid mappings

analysis
Analysis
  • Reduction of potential mappings with FastHASH

99%

99%

99%

99%

99%

FastHASHfilters out over 99% of the potential mappings without sacrificing any valid mappings

other key results in the paper
Other Key Results (In the paper)
  • FastHASH finds all possible valid mappings
  • Correctly mapped all simulated reads (with fewer than e artificially added errors)
outline8
Outline
  • Read Mapping and its Challenges
  • Hash Table-Based Mappers
  • Problem and Goal
  • Key Observations
  • Mechanisms
  • Results
  • Conclusion
conclusion
Conclusion
  • Problem: Existing read mappers perform poorly in mapping billions of short reads to the reference genome, in the presence of errors
  • Observation: Most of the verification calculations are unnecessary
  • Key Idea: To reduce the cost of unnecessary verification
    • Reject invalid mappings early (Adjacency Filtering)
    • Reduce the number of possible mappings to examine (Cheap K-mer Selection)
  • Key Result: FastHASH obtains up to 19x speedup over the state-of-the-art mapper without losing valid mappings
acknowledgements
Acknowledgements
  • Carnegie Mellon University (Hongyi Xin, Donghyuk Lee, SamihanYedkar and OnurMutlu, co-authors)
  • Bilkent University (Can Alkan, co-author)
  • University of Washington (Evan Eichlerand Can Alkan)
  • UCLA (FarhadHormozdiari, co-author)
  • NIH (National Institutes of Health) for financial support
thank you
Thank you! 
  • Questions?
  • Download link to FastHASH
  • You can find the slides on SAFARI group website:
    • http://www.ece.cmu.edu/~safari
accelerating read mapping with fasthash1

Accelerating Read Mapping with FastHASH

HongyiXin†Donghyuk Lee†FarhadHormozdiari ‡ SamihanYedkar†Can Alkan§OnurMutlu†

† Carnegie Mellon University § University of Washington

‡ University of California Los Angeles

mapper comparison number of valid mappings
Mapper Comparison: Number of Valid Mappings
  • Bowtie does not support error threshold larger than 3

FastHASH is able to find many more valid mappings than Bowtie and BWA

mapper comparison execution time
Mapper Comparison: Execution Time
  • Bowtie does not support error threshold larger than 3

FastHASH is slower for e <= 3, but is much more comprehensive (can find many more valid mappings)