google bigtable l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Google Bigtable PowerPoint Presentation
Download Presentation
Google Bigtable

Loading in 2 Seconds...

play fullscreen
1 / 19

Google Bigtable - PowerPoint PPT Presentation


  • 368 Views
  • Uploaded on

Google Bigtable. Fay Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C. Hsieh, Deborah A. Wallach, Mike Burrows, Tushar Chandra, Andrew Fikes, Robert E. Gruber Google, Inc. UWCS OS Seminar Discussion Erik Paulson 2 October 2006.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Google Bigtable' - valerie


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
google bigtable

Google Bigtable

Fay Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C. Hsieh, Deborah A. Wallach, Mike Burrows, Tushar Chandra, Andrew Fikes, Robert E. Gruber

Google, Inc.

UWCS OS Seminar Discussion

Erik Paulson

2 October 2006

See also the (other)UW presentation by Jeff Dean in September of 2005

(See the link on the seminar page, or just google for “google bigtable”)

before we begin
Before we begin…
  • Intersection of databases and distributed systems
  • Will try to explain (or at least warn) when we hit a patch of database
  • Remember this is a discussion!
google scale
Google Scale
  • Lots of data
    • Copies of the web, satellite data, user data, email and USENET, Subversion backing store
  • Many incoming requests
  • No commercial system big enough
    • Couldn’t afford it if there was one
    • Might not have made appropriate design choices
  • Firm believers in the End-to-End argument
  • 450,000 machines (NYTimes estimate, June 14th 2006
building blocks
Building Blocks
  • Scheduler (Google WorkQueue)
  • Google Filesystem
  • Chubby Lock service
  • Two other pieces helpful but not required
    • Sawzall
    • MapReduce (despite what the Internet says)
  • BigTable: build a more application-friendly storage service using these parts
google file system
Google File System
  • Large-scale distributed “filesystem”
  • Master: responsible for metadata
  • Chunk servers: responsible for reading and writing large chunks of data
  • Chunks replicated on 3 machines, master responsible for ensuring replicas exist
  • OSDI ’04 Paper
chubby
Chubby
  • {lock/file/name} service
  • Coarse-grained locks, can store small amount of data in a lock
  • 5 replicas, need a majority vote to be active
  • Also an OSDI ’06 Paper
data model a big map
Data model: a big map
  • <Row, Column, Timestamp> triple for key - lookup, insert, and delete API
  • Arbitrary “columns” on a row-by-row basis
    • Column family:qualifier. Family is heavyweight, qualifier lightweight
    • Column-oriented physical store- rows are sparse!
  • Does not support a relational model
    • No table-wide integrity constraints
    • No multirow transactions
sstable
SSTable
  • Immutable, sorted file of key-value pairs
  • Chunks of data plus an index
    • Index is of block ranges, not values

SSTable

64K block

64K block

64K block

Index

tablet
Tablet
  • Contains some range of rows of the table
  • Built out of multiple SSTables

Start:aardvark

End:apple

Tablet

SSTable

SSTable

64K block

64K block

64K block

64K block

64K block

64K block

Index

Index

table
Table
  • Multiple tablets make up the table
  • SSTables can be shared
  • Tablets do not overlap, SSTables can overlap

Tablet

Tablet

apple

boat

aardvark

apple_two_E

SSTable

SSTable

SSTable

SSTable

servers
Servers
  • Tablet servers manage tablets, multiple tablets per server. Each tablet is 100-200 megs
    • Each tablet lives at only one server
    • Tablet server splits tablets that get too big
  • Master responsible for load balancing and fault tolerance
    • Use Chubby to monitor health of tablet servers, restart failed servers
    • GFS replicates data. Prefer to start tablet server on same machine that the data is already at
editing a table
Editing a table
  • Mutations are logged, then applied to an in-memory version
  • Logfile stored in GFS

Tablet

Insert

Memtable

Insert

Delete

boat

apple_two_E

Insert

Delete

Insert

SSTable

SSTable

compactions
Compactions
  • Minor compaction – convert the memtable into an SSTable
    • Reduce memory usage
    • Reduce log traffic on restart
  • Merging compaction
    • Reduce number of SSTables
    • Good place to apply policy “keep only N versions”
  • Major compaction
    • Merging compaction that results in only one SSTable
    • No deletion records, only live data
locality groups
Locality Groups
  • Group column families together into an SSTable
    • Avoid mingling data, ie page contents and page metadata
    • Can keep some groups all in memory
  • Can compress locality groups
  • Bloom Filters on locality groups – avoid searching SSTable
lessons learned
Lessons learned
  • Interesting point- only implement some of the requirements, since the last is probably not needed
  • Many types of failure possible
  • Big systems need proper systems-level monitoring
  • Value simple designs