Coblitz a scalable large file transfer service cos 461
This presentation is the property of its rightful owner.
Sponsored Links
1 / 31

CoBlitz: A Scalable Large-file Transfer Service (COS 461) PowerPoint PPT Presentation


  • 87 Views
  • Uploaded on
  • Presentation posted in: General

CoBlitz: A Scalable Large-file Transfer Service (COS 461). KyoungSoo Park Princeton University. Large-file Distribution. Increasing demand for large files Movies or software release On-line movie / downloads Linux distribution Files are 100MB ~ tens of GB One-to-many downloads

Download Presentation

CoBlitz: A Scalable Large-file Transfer Service (COS 461)

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Coblitz a scalable large file transfer service cos 461

CoBlitz: A Scalable Large-file Transfer Service(COS 461)

KyoungSoo Park

Princeton University


Large file distribution

Large-file Distribution

  • Increasing demand for large files

  • Movies or software release

    • On-line movie/ downloads

    • Linux distribution

  • Files are 100MB ~ tens of GB

  • One-to-many downloads

    How to serve large files to many clients?

    • Content Distribution Network(CDN)?

    • Peer-to-peer system?


What cdns are optimized for

What CDNs Are Optimized For

Most Web files are small (1KB ~ 100KB)


Why not web cdns

Why Not Web CDNs?

  • Whole file caching in participating proxy

    • Optimized for 10KB objects

    • 2GB = 200,000 x 10KB

  • Memory pressure

    • Working sets do not fit in memory

    • Disk access is 1000 times slower

  • Waste of resources

    • More servers needed

    • Provisioning is a must


Peer to peer

Peer-to-Peer?

  • BitTorrent takes up ~30% Internet BW

1. Download a “torrent” file

2. Contact the tracker

3. Enter the “swarm” network

4. Chunk exchange policy

- Rarest chunk first or random

- Tit-for-tat: incentive to upload

- Optimistic unchoking

5. Validate the checksums

up

down

peers

torrent

tracker

Benefit: extremely good use of resources!


Peer to peer1

Peer-to-Peer?

  • Custom software

    • Deployment is a must

    • Configurations needed

  • Companies may want managed service

    • Handles flash crowds

    • Handles long-lived objects

  • Performance problem

    • Hard to guarantee the service quality

    • Others are discussed later


What we d l ike is

What We’d Like Is

Large-file service with

No custom client

No custom server

No prepositioning

No rehosting

No manual provisoning


Coblitz scalable l arge file cdn

CoBlitz: Scalable Large-file CDN

  • Reducing the problem to small-file CDN

    • Split large-files into chunks

    • Distribute chunks at proxies

    • Aggregate memory/cache

    • HTTP needs no deployment

  • Benefits

    • Faster than BitTorrent by 55-86% (~500%)

    • One copy from origin serves 43-55 nodes

    • Incremental build on existing CDNs


How i t w orks

Origin

Server

HTTP

RANGE QUERY

coblitz.codeen.org

chunk 1

chunk 2

chunk 1

chunk 2

chunk 1

chunk1

chunk 3

chunk 3

chunk 5

chunk 5

chunk 5

chunk 4

chunk 4

chunk 5

How It Works

Only reverse proxy(CDN) caches the chunks!

CDN = Redirector +

Reverse Proxy

DNS

chunk1

chunk2

CDN

CDN

chunk3

Client

Agent

CDN

CDN

Agent

Client

CDN

CDN

chunk4

chunk5


Smart agent

waiting

done

done

Smart Agent

  • Preserves HTTP semantics

  • Parallel chunk requests

CDN

sliding window of “chunks”

done

CDN

HTTP

Client

CDN

CDN

waiting

waiting

done

CDN

done

no action

waiting

no action

waiting

no action

waiting

Agent


Chunk indexing consistent hashing

Chunk Indexing: Consistent Hashing

Problem: How to find the node responsible for a specific chunk?

Static hashing

f(x) = some_f(x) % n

But n is dynamic for servers

- node can go down

- new node can join

… N-1 0 …

X1

X3

CDN node (proxy)

Consistent Hashing

F(x) = some_F(x) % N

(N is a large but fixed number)

Find a live node k, where

|F(k) – F(URL) | is minimum

Xk : Chunk request

X2


Operation challenges

Operation & Challenges

  • Provides public service over 2.5 years

    • http://coblitz.codeen.org:3125/URL

  • Challenges

    • Scalability & robustness

    • Peering set difference

    • Load to the origin server


Unilateral peering

Unilateral Peering

  • Independent proximity-aware peering

    • Pick “n” close nodes around me

    • Cf. BitTorrent picks “n” nodes randomly

  • Motivation

    • Partial network connectivity

      • Internet2, CANARIE nodes

      • Routing disruption

    • Isolated nodes

  • Benefits

    • No synchronized maintenance problem

    • Improve both scalability & robustness


Peering set difference

Both can reach

Only can reach

Only can reach

Peering Set Difference

  • No perfect clustering by design

  • Assumption

    • Close nodes shares common peers


Peering set difference1

Peering Set Difference

  • Highly variable App-level RTTs

    • 10 x times variance than ICMP

  • High rate of change in peer set

  • Close nodes share less than 50%

    • Low cache hit

    • Low memory utility

    • Excessive load to the origin


Peering set difference2

Peering Set Difference

  • How to fix?

    • Avg RTT  min RTT

    • Increase # of samples

    • Increase # of peers

    • Hysteresis

  • Close nodes share more than 90%


Reducing origin load

Reducing Origin Load

  • Still have peering set difference

    • Critical in traffic to origin

  • Proximity-based routing

    • Converge exponentially fast

    • 3-15% do one more hop

    • Implicit overlay tree

  • Result

    • Origin load reduction by 5x

Origin server

Rerun hashing


Scale experiments

Scale Experiments

  • Use all live PlanetLab nodes as clients

    • 380~400 live nodes at any time

    • Simultaneous fetch of 50MB file

  • Test scenarios

    • Direct

    • BitTorrent Total/Core

    • CoBlitz uncached/cached/staggered

    • Out-of-order numbers in paper


Throughput distribution

55-86%

Throughput Distribution

1

0.9

BT-Core

0.8

Out-of-order staggered

0.7

0.6

Fraction of Nodes <= X (CDF)

0.5

Direct

0.4

BT

-

total

0.3

BT

-

core

In

-

order uncached

0.2

In

-

order staggered

0.1

In

-

order cached

0

0

2000

4000

6000

8000

10000

Throughput(Kbps)


Downloading times

95% percentile: 1000+ secs faster

Downloading Times


Why is bittorrent slow

Why Is BitTorrent Slow?

  • In the experiments

    • No locality – randomly choose peers

    • Chunk indexing – extra communication

      • Trackerless BitTorrent – Kademlia DHT

  • In practice

    • Upload capacity of typical peers is low

      • 10 to a few 100 Kbps for cable/DSL users

    • Tit for tat may not be fair

      • A few high-capacity uploaders help the most

      • BitTyrant[NSDI’07]


Synchronized workload congestion

Synchronized Workload Congestion

Origin Server


Addressing congestion

Addressing Congestion

  • Proximity-based multi-hop routing

    • Overlay tree for each chunk

  • Dynamic chunk-window resizing

    • Increase by 1/log(x), (where x is win size) if chunk finishes < average

    • Decrease by 1 if retry kills the first chunk


Number of failures

Number of Failures


Performance a fter flash crowds

BitTorrent: 20% > 5Mbps

CoBlitz:70+% > 5Mbps

Performance After Flash Crowds


Data reuse

Data Reuse

7 fetches for 400 nodes, 98% cache hit


Real world usage

Real-world Usage

  • 1-2 Terabytes/day

  • Fedora Core official mirror

    • US-East/West, England, Germany, Korea, Japan

  • CiteSeer repository (50,000+ links)

  • University Channel (podcast/video)

  • Public lecture distribution by PU OIT

  • Popular game patch distribution

  • PlanetLab researchers

    • Stork(U of Arizona) + ~10 others


Fedora core 6 release

Fedora Core 6 Release

  • October 24th, 2006

  • Peak Throughput 1.44Gbps

Release point 10am

1 G

Origin Server

30-40Mbps


On fedora core mirror list

On Fedora Core Mirror List

  • Many people complained about I/O

    • Performing peak 500Mbps out of 2Gbps

      • 2 Sun x4200 w/Dual Operons, 2G mem

      • 2.5 TB Sata-based SAN

      • All ISOs in disk cache or in-memoy FS

  • CoBlitz uses 100MB mem per node

    • Many PL node disks are IDEs

    • Most nodes are BW capped at 10Mpbs


Conclusion

Conclusion

  • Scalable large-file transfer service

  • Evolution under real traffic

    • Up and running 24/7 for over 2.5 years

    • Unilateral peering, multi-hop routing, window size adjustment

  • Better performance than P2P

    • Better throughput, download time

    • Far less origin traffic


Thank you

Thank you!

More information:

http://codeen.cs.princeton.edu/coblitz/

How to use:

http://coblitz.codeen.org:3125/URL*

*Some content restrictions apply

See Web site for details

Contact me if you want full access!


  • Login