1 / 39

Hadoop , Hadoop , Hadoop !!!

Hadoop , Hadoop , Hadoop !!!. Jerome Mitchell Indiana University. Outline. BIG DATA Hadoop MapReduce The Hadoop Distributed File System (HDFS) Workflow Conclusions References HandsOn. LOTS OF DATA EVERYWHERE. Why Should You Care? Even Grocery Stores Care. What is Hadoop ?.

mave
Download Presentation

Hadoop , Hadoop , Hadoop !!!

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Hadoop, Hadoop, Hadoop!!! Jerome Mitchell Indiana University

  2. Outline • BIG DATA • HadoopMapReduce • The Hadoop Distributed File System (HDFS) • Workflow • Conclusions • References • HandsOn

  3. LOTS OF DATA EVERYWHERE

  4. Why Should You Care? Even Grocery Stores Care

  5. What is Hadoop? • At Google MapReduce operation are run on a special file system called Google File System (GFS) that is highly optimized for this purpose. • GFS is not open source. • Doug Cutting and Yahoo! reverse engineered the GFS and called it Hadoop Distributed File System (HDFS). • The software framework that supports HDFS, MapReduce and other related entities is called the project Hadoop or simply Hadoop. • This is open source and distributed by Apache.

  6. What exactly is Hadoop • A growing collection of subprojects

  7. Motivation for MapReduce • Large-Scale Data Processing • Want to use 1000s of CPUs • But, don’t want the hassle of managing things • MapReduce Architecture provides • Automatic Parallelization and Distribution • Fault Tolerance • I/O Scheduling • Monitoring and Status Updates

  8. MapReduce Model • Input & Output: a set of key/value pairs • Two primitive operations • map: (k1,v1)  list(k2,v2) • reduce: (k2,list(v2))  list(k3,v3) • Each map operation processes one input key/value pair and produces a set of key/value pairs • Each reduce operation • Merges all intermediate values (produced by map ops) for a particular key • Produce final key/value pairs • Operations are organized into tasks • Map tasks: apply map operation to a set of key/value pairs • Reduce tasks: apply reduce operation to intermediate key/value pairs • Each MapReduce job comprises a set of map and reduce (optional) tasks.

  9. HDFS Architecture

  10. The WorkFlow • Load data into the Cluster (HDFS writes) • Analyze the data (MapReduce) • Store results in the Cluster (HDFS) • Read the results from the Cluster (HDFS reads)

  11. Hadoop Server Roles Client Distributed Data Storage Distributed Data Processing HDFS MapReduce Secondary Name Node Name Node Master Job Tracker Data Nodes & Task Tracker Data Nodes & Task Tracker Data Nodes & Task Tracker Slaves Data Nodes & Task Tracker Data Nodes & Task Tracker Data Nodes & Task Tracker

  12. Hadoop Cluster Switch Switch Switch Switch Switch Switch DN + TT DN + TT DN + TT DN + TT DN + TT DN + TT DN + TT DN + TT DN + TT DN + TT DN + TT DN + TT DN + TT DN + TT DN + TT DN + TT DN + TT Rack 1 Rack 2 Rack 3 Rack N

  13. Hadoop Rack Awareness – Why? Name Node metatadata Switch Rack Aware Results.txt = BLK A:DN1, DN5, DN6 BLK B: DN7, DN1, DN2 BLK C DN5, DN8, DN9 Rack 1:Data Node 1 Data Node 2Data Node 3 Rack 1: Data Node 5 Data Node 6 Data Node 7 Switch Switch Switch Data Node 1 Data Node 5 Data Node 5 Data Node 2 Data Node 10 Data Node 6 Data Node 3 Data Node 11 Data Node 7 Data Node 5 Data Node 12 Data Node 8 Rack 5 Rack 9 Rack 1

  14. Sample Scenario How many times did our customers type the word refund into emails sent to Indiana University? Huge file containing all emails sent to Indiana University… File.txt

  15. Writing Files to HDFS File.txt NameNode BLK A BLK B BLK C Client Data Node 6 Data Node 1 Data Node 5 Data Node N BLK A BLK C BLK B

  16. Data Node Reading Files From HDFS Name Node Rack Aware metatadata Switch Rack 1:Data Node 1 Data Node 2Data Node 3 Rack 1: Data Node 5 Results.txt = BLK A:DN1, DN5, DN6 BLK B: DN7, DN1, DN2 BLK C DN5, DN8, DN9 Switch Switch Switch Data Node 1 Data Node 1 Data Node 1 Data Node 2 Data Node 2 Data Node 2 Data Node Data Node Data Node Data Node Data Node Data Node Rack 5 Rack 9 Rack 1

  17. MapReduce: Three Phases 1. Map 2. Sort 3. Reduce

  18. Data Processing: Map Job Tracker Name Node Map Task Map Task Map Task Data Node 1 Data Node 5 Data Node 9 BLK C BLK B BLK A File.txt

  19. map map k k k v v v k k k v v v MapReduce: The Map Step Input key-value pairs Intermediate key-value pairs … … k v

  20. Data Processing: Reduce HDFS Results.txt Reduce Task Map Task Map Task Map Task BLK C BLK B BLK A

  21. Intermediate key-value pairs Key-value groups reduce reduce k k v v k v v v k k k v v v k v v group k v … … k v k v MapReduce: The Reduce Step Output key-value pairs …

  22. Clients Reading Files from HDFS NameNode Client metatadata Results.txt = BLK A:DN1, DN5, DN6 BLK B: DN7, DN1, DN2 BLK C DN5, DN8, DN9 Switch Switch Switch Data Node 1 Data Node 1 Data Node 1 Data Node 2 Data Node 2 Data Node 2 Data Node Data Node Data Node Data Node Data Node Data Node Rack 5 Rack 9 Rack 1

  23. Conclusions • We introduced MapReduce programming model for processing large scale data • We discussed the supporting Hadoop Distributed File System • The concepts were illustrated using a simple example • We reviewed some important parts of the source code for the example.

  24. References • Apache Hadoop Tutorial: http://hadoop.apache.org http://hadoop.apache.org/core/docs/current/mapred_tutorial.html • Dean, J. and Ghemawat, S. 2008. MapReduce: simplified data processing on large clusters.Communication of ACM 51, 1 (Jan. 2008), 107-113. • Cloudera Videos by Aaron Kimball: http://www.cloudera.com/hadoop-training-basic 4. http://www.cse.buffalo.edu/faculty/bina/mapreduce.html

  25. Finally, A Hands-on Assignment!

  26. The MapReduce Framework (pioneered by Google)

  27. Automatic Parallel Execution in MapReduce (Google) Handles failures automatically, e.g., restarts tasks if a node fails; runs multiples copies of the same task to avoid a slow task slowing down the whole job

  28. The Map (Example) When in the course of human events it … inputs tasks (M=3) partitions (intermediate files) (R=2) (when,1), (course,1) (human,1) (events,1) (best,1) … map Over the past five years, the authors and many… It was the best of times and the worst of times… (in,1) (the,1) (of,1) (it,1) (it,1) (was,1) (the,1) (of,1) … map (over,1), (past,1) (five,1) (years,1) (authors,1) (many,1) … (the,1), (the,1) (and,1) … This paper evaluates the suitability of the … (this,1) (paper,1) (evaluates,1) (suitability,1) … map (the,1) (of,1) (the,1) …

  29. The Reduce (Example) partition (intermediate files) (R=2) reduce task sort (in,1) (the,1) (of,1) (it,1) (it,1) (was,1) (the,1) (of,1) … (the,1), (the,1) (and,1) … (and, (1)) (in,(1)) (it, (1,1)) (the, (1,1,1,1,1,1)) (of, (1,1,1)) (was,(1)) (the,1) (of,1) (the,1) … reduce (and,1) (in,1) (it, 2) (of, 3) (the,6) (was,1) Note: only one of the two reduce tasks shown

  30. Map function Reduce function Run this program as a MapReduce job Lifecycle of a MapReduce Job

  31. 190+ parameters in Hadoop Set manually or defaults are used Job Configuration Parameters

  32. Formatting the NameNode Before we start, we have to format Hadoop's distributed filesystem (HDFS) for the namenode. You need to do this the first time you set up Hadoop. Do not format a running Hadoopnamenode, this will cause all your data in the HDFS filesytem to be erased. To format the filesystem, run the command (from the master): -------------------------------------------- bin/hadoopnamenode -format --------------------------------------------- Starting Hadoop: Starting hadoop is done in two steps: First, the HDFS daemons are started: the namenode daemon is started on master, and datanode daemons are started on all slaves (here: master and slave). Second, the MapReduce daemons are started: the jobtracker is started on master, and tasktracker daemons are started on all slaves (here: master and slave).

  33. HDFS daemons: Run the command <HADOOP_HOME>/bin/start-dfs.sh on the machine you want the namenode to run on. This will bring up HDFS with the namenode running on the machine you ran the previous command on, and datanodes on the machines listed in the conf/slaves file. In our case, we will run bin/start-dfs.sh on master: ------------------------- bin/start-dfs.sh ------------------------ On slave, you can examine the success or failure of this command by inspecting the log file <HADOOP_HOME>/logs/hadoop-hadoop-datanode-slave.log. Now, the following Java processes should run on master: root@ubuntu:$HADOOP_HOME/bin: jps 14799 NameNode 15314 Jps 14880 DataNode 14977 SecondaryNameNode ------------------------------------

  34. MapReduce Daemons: In our case, we will run bin/start-mapred.sh on master: ------------------------------------- bin/start-mapred.sh ------------------------------------- On slave, you can examine the success or failure of this command by inspecting the log file <HADOOP_HOME>/logs/hadoop-hadoop-tasktracker-slave.log. At this point, the following Java processes should run on master: ---------------------------------------------------- root@ubuntu:$HADOOP_HOME/bin$ jps 16017 Jps 14799 NameNode 15686 TaskTracker 14880 DataNode 15596 JobTracker 14977 SecondaryNameNode ----------------------------------------------------

  35. We will execute your first HadoopMapReducejob. We will use the WordCount example, which will read a text file and count the frequency of words. The input is a text file and the output is a text file with each line of which contains a word and the count of how often it occurred, separated by a tab. Download example input data: Copy local data file to HDFS Before we run the actual MapReduce job, we first have to copy the files from our local file system to Hadoop's HDFS ----------------------------- root@ubuntu:$HADOOP_HOME/bin$ hadoopdfs -copyFromLocal /tmp/source destination

  36. Build WordCount Execute build.sh Run the MapReduce job Now, we actually run the WordCount example job. This command will read all the files in the HDFS “destination” directory , process it, and store the result in the HDFS directory “output”. ----------------------------------------- root@ubuntu:$HADOOP_HOME/bin bin/hadoophadoop-example wordcount destination output ----------------------------------------- You can check if the result is successfully stored in HDFS directory “output”. Retrieve the job result from HDFS To inspect the file, you can copy it from HDFS to the local file system. ------------------------------------- root@ubuntu:/usr/local/hadoop$ mkdir /tmp/output root@ubuntu:/usr/local/hadoop$ bin/hadoopdfs –copyToLocal output/part-00000 /tmp/output ---------------------------------------- Alternatively, you can read the file directly from HDFS without copying it to the local file system by using the command : --------------------------------------------- root@ubuntu:/usr/local/hadoop$ bin/hadoopdfs –cat output/part-00000

  37. Even though the Hadoop framework is written in Java, programs for Hadoop need not to be coded in Java but can also be developed in other languages like Python or C++. Creating a launching program for your application The launching program configures: – The Mapper and Reducer to use – The output key and value types (input types are inferred from the InputFormat) – The locations for your input and output The launching program then submits the job and typically waits for it to complete A Map/Reduce may specify how it’s input is to be read by specifying an InputFormat to be used A Map/Reduce may specify how it’s output is to be written by specifying an OutputFormat to be used

More Related