the hadoop fair scheduler n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
The Hadoop Fair Scheduler PowerPoint Presentation
Download Presentation
The Hadoop Fair Scheduler

Loading in 2 Seconds...

play fullscreen
1 / 31

The Hadoop Fair Scheduler - PowerPoint PPT Presentation


  • 653 Views
  • Uploaded on

The Hadoop Fair Scheduler. Matei Zaharia Cloudera / Facebook / UC Berkeley. UC Berkeley. Outline. Motivation / Hadoop usage at Facebook Fair scheduler basics Configuring the fair scheduler Future plans. Motivation. Provide short response times to small jobs in a shared Hadoop cluster

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

The Hadoop Fair Scheduler


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
the hadoop fair scheduler

The Hadoop Fair Scheduler

Matei Zaharia

Cloudera / Facebook / UC Berkeley

UC Berkeley

outline
Outline
  • Motivation / Hadoop usage at Facebook
  • Fair scheduler basics
  • Configuring the fair scheduler
  • Future plans
motivation
Motivation
  • Provide short response times to small jobs in a shared Hadoop cluster
  • Improve utilization over private clusters / HOD
hadoop usage at facebook
Hadoop Usage at Facebook
  • Data warehouse running Hive
  • 600 machines, 4800 cores, 2.4 PB disk
  • 3200 jobs per day
  • 50+ engineers have used Hadoop
facebook data pipeline
Facebook Data Pipeline

Web Servers

Scribe Servers

Network Storage

Hive Queries

Analysts

Summaries

Hadoop Cluster

MySQL

Oracle RAC

facebook job types
Facebook Job Types
  • Production jobs: load data, compute statistics, detect spam, etc
  • Long experiments: machine learning, etc
  • Small ad-hoc queries: Hive jobs, sampling

GOAL: Provide fast response times for small jobs and

guaranteed service levels for production jobs

outline1
Outline
  • Motivation / Hadoop usage at Facebook
  • Fair scheduler basics
  • Configuring the fair scheduler
  • Future plans
fair scheduler basics
Fair Scheduler Basics
  • Group jobs into “pools”
  • Assign each pool a guaranteed minimum share (split up among its jobs)
  • Split excess capacity evenly between jobs
pools
Pools
  • Determined from a configurable job property
    • Default (before 0.20): mapred.queue.name
    • At Facebook: user.name (one pool per user)
  • Unmarked jobs go into a “default pool”
  • Pools have properties:
    • Minimum map slots
    • Minimum reduce slots
    • Limit on # of running jobs
scheduling algorithm
Scheduling Algorithm
  • Divide each pool’s min share among its jobs
  • Divide excess capacity among all jobs*
  • When a slot needs to be assigned:
    • If there is any job below its min share, schedule it
    • Else schedule the job that we’ve been most unfair to (based on “deficit”)

* Fair schedulers from Hadoop 0.20 on will share equally between pools, not jobs;

patch available at https://issues.apache.org/jira/browse/HADOOP-4789

scheduler dashboard2
Scheduler Dashboard

Reassign pool

scheduler dashboard3
Scheduler Dashboard

Change priority

scheduler dashboard4
Scheduler Dashboard

FIFO mode (for testing)

additional features
Additional Features
  • Job weights for unequal sharing:
    • Based on priority (each level is 2x more)
    • Based on size (mapred.fairscheduler.sizebasedweight)
  • Limits on # of running jobs:
    • Per user
    • Per pool
outline2
Outline
  • Motivation / Hadoop usage at Facebook
  • Fair scheduler basics
  • Configuring the fair scheduler
  • Future plans
installing the scheduler
Installing the Scheduler
  • Compile it:
    • ant package
  • Place it on the classpath:
    • cp build/contrib/fairscheduler/*.jar lib
    • Alternatively, add the JAR to HADOOP_CLASSPATH in conf/hadoop-env.sh
configuration files
Configuration Files
  • Hadoop config (conf/hadoop-site.xml)
    • Contains scheduler options, pointer to pools file
  • Pools file (pools.xml)
    • Contains min share allocations and limits on pools
    • Reloaded every 15 seconds to allow reconfiguring pools at runtime
minimal hadoop site xml
Minimal hadoop-site.xml

<property>

<name>mapred.jobtracker.taskScheduler</name>

<value>org.apache.hadoop.mapred.FairScheduler</value>

</property>

<property>

<name>mapred.fairscheduler.allocation.file</name>

<value>/path/to/pools.xml</value>

</property>

minimal pools xml
Minimal pools.xml

<?xml version="1.0"?>

<allocations>

</allocations>

configuring a pool
Configuring a Pool

<?xml version="1.0"?>

<allocations>

<pool name="ads">

<minMaps>10</minMaps>

<minReduces>5</minReduces>

</pool>

</allocations>

  • Any pools not configured in pools.xml will have minMaps=0 and minReduces=0
setting running job limits
Setting Running Job Limits

<?xml version="1.0"?>

<allocations>

<pool name="ads">

<minMaps>10</minMaps>

<minReduces>5</minReduces>

<maxRunningJobs>3</maxRunningJobs>

</pool>

<user name="matei">

<maxRunningJobs>1</maxRunningJobs>

</user>

</allocations>

default jobs limit for users
Default Jobs Limit for Users

<?xml version="1.0"?>

<allocations>

<pool name="ads">

<minMaps>10</minMaps>

<minReduces>5</minReduces>

<maxRunningJobs>3</maxRunningJobs>

</pool>

<user name="matei">

<maxRunningJobs>1</maxRunningJobs>

</user>

<userMaxJobsDefault>10</userMaxJobsDefault>

</allocations>

other hadoop site xml properties
Other hadoop-site.xml Properties

mapred.fairscheduler.assignmultiple:

  • Assign a map and reduce on each heartbeat; improves ramp-up speed and throughput; recommendation: set to true
other hadoop site xml properties1
Other hadoop-site.xml Properties

mapred.fairscheduler.poolnameproperty:

  • Which jobconf property to use to determine what pool a job is in
    • Default: mapred.queue.name (queue name)
    • Another useful option: user.name
    • Can also make up your own, e.g. “project”
other hadoop site xml properties2
Other hadoop-site.xml Properties

mapred.fairscheduler.weightadjuster:

  • Allows modifying job weights through a plugin class; one useful example is provided – a new job booster to let short jobs finish faster:

Please see README for details

<property>

<name>mapred.fairscheduler.weightadjuster</name>

<value>org.apache.hadoop.mapred.NewJobWeightBooster</value>

</property>

outline3
Outline
  • Motivation / Hadoop usage at Facebook
  • Fair scheduler basics
  • Configuring the fair scheduler
  • Future plans
future plans
Future Plans
  • Share equally between pools, not jobs (Hadoop 0.20 release, HADOOP-4789)
  • Preemption if a job is starved of its min or fair share for some timeout (HADOOP-4665)
  • Locality wait optimization (HADOOP-4667)
future plans1
Future Plans
  • Simpler scheduling model (HADOOP-4803)
  • FIFO pools (HADOOP-4803, HADOOP-5186)
  • Delayed job initialization (HADOOP-5186)
  • Scalability and operational improvements
thanks
Thanks!
  • The Fair Scheduler is available in Hadoop 0.19; docs in src/contrib/fairscheduler/README
  • Hadoop 0.17 and 0.18 versions at http://issues.apache.org/jira/browse/HADOOP-3746

matei@cloudera.com