1 / 54

Hadoop&HDFS

Hadoop&HDFS. OUTLINE. Introduction Architecture Hadoop Distribution File System Architecture of HDFS NameNode DataNode HDFS Client Replica Management. OUTLINE. Introduction Architecture Hadoop Distribution File System Architecture of HDFS NameNode DataNode HDFS Client

adolfo
Download Presentation

Hadoop&HDFS

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Hadoop&HDFS

  2. OUTLINE • Introduction • Architecture • Hadoop Distribution File System • Architecture of HDFS • NameNode • DataNode • HDFS Client • Replica Management

  3. OUTLINE • Introduction • Architecture • Hadoop Distribution File System • Architecture of HDFS • NameNode • DataNode • HDFS Client • Replica Management

  4. What is Hadoop?

  5. Hadoop 起源(2002~2004) • 發起人－Doug Cutting • Lucene • 用Java 設計的高效能文件索引引擎API • 索引文件中的每一字，讓搜尋的效率比傳統逐字比較還要高的多 • Nutch • 開放原始碼的網站搜尋引擎 • 利用Lucene 函式庫開發

  6. Hadoop 轉折點 • Nutch遇到處理大量網站資料的瓶頸 • Google發表三大關鍵技術 • SOSP 2003 : “The Google File System” • OSDI 2004 : “MapReduce : Simplifed Data Processing on Large Cluster” • OSDI 2006 : “Bigtable: A Distributed Storage System for Structured Data”

  7. Hadoop 起源 (2004~Now) • 參考 Google 提出的技術並先後於Nutch上實作 • 分散式檔案系統Nutch Distributed File System (NDFS) • MapReduce • 在2006年時，Nutch 把分散式計算 (Distributed Computing) 的部分獨立出來，稱之為Hadoop • NDFS改名為 Hadoop Distributed File System (HDFS)

  8. Hadoop 的特色 • 在數據資料沒有相依性的情況下，可以有效率的平行處理這些資料。 • 可以透過自動維護資料副本的功能，提供容錯機制，讓錯誤發生時可自動回復。 • 可以提供可靠的資料儲存及分析處理的能力。

  9. Linux Linux Linux Linux

  10. Hadoop 的架構(1/3) • Hadoop 專案包含一些相關子專案 Pig Chukwa Hive MapReduce Avro ZooKeeper HBase HDFS Hadoop Core

  11. Hadoop 的架構(2/3) • Hadoop Core： • 核心部分包含一些分散式檔案系統及一般輸出入的重要元件跟介面。 • Avro： • 一個有效率，跨越各種語言的RPC的資料序列化系統。 • MapReduce： • 一個分散式資料處理模式及執行環境。 • HDFS： • 一個分散式檔案系統。 • Pig： • 處理大量資料集的資料流語言與執行環境。

  12. Hadoop 的架構(3/3) • HBase： • 一個以列 (row) 為導向的分散式資料庫系統。 • ZooKeeper： • 一個分散式協同服務，可以提供分散式應用程式的原始指令。 • Hive： • 一個分散式資料倉儲系統，管理HDFS上所儲存的資料，並提供SQL為基礎的查詢語言。 • Chukwa： • 一個分散式資料收集及分析系統。

  13. Google References The Google File System [2003] MapReduce [2004] Bigtable [2006]

  14. Hadoop 與 Google 架構的不同

  15. OUTLINE • Introduction • Architecture • Hadoop Distribution File System • Architecture of HDFS • NameNode • DataNode • HDFS Client • Replica Management

  16. OUTLINE • Introduction • Architecture • Hadoop Distribution File System • Architecture of HDFS • NameNode • DataNode • HDFS Client • Replica Management

  17. Architecture of HDFS HDFS Client NN DN DN DN DN DN Cluster NN: NameNode DN: DataNode

  18. File Storing DN DN DN Tempo Block 64MB Block 64MB Block 64MB Block 64MB File 100MB Temp Block 36MB Block 36MB Block 36MB Block 36MB DN DN DN DN: DataNode

  19. OUTLINE • Introduction • Architecture • Hadoop Distribution File System • Architecture of HDFS • NameNode • DataNode • HDFS Client • Replica Management

  20. Responsibilities of NameNode • Maintaining the namespace tree and the mapping of file blocks to DataNodes • Replica management

  21. Namespace • Files and directories are represented by inodes. • The inode data and the list of blocks belonging to each file comprise to metadata of the name system called image. • The persistent record of the image called checkpoint. • The modification log of the image called journal.

  22. Namespace Storing • NameNode keeps the image in RAM. • Checkpoint and journal are stored in the local host’s native files system.

  23. Checkpoint & Journal Journal Checkpoint

  24. NameNode’s Version

  25. Protecting the Critical Information • If ether the checkpoint or the journal is missing, or be corrupt, the namespace will be lost party or entirely. • Storing checkpoint and journal in multiple store directories and NFS server • Creating periodic checkpoints by either CheckpointNode or BackupNode, and storing checkpoint in it.

  26. CheckpointNode Options • Downloading checkpoint and journal from NameNode • Combining the checkpoint and the journal to create a new checkpoint and an empty journal • Returning the new checkpoint back to the NameNode

  27. BackupNode • BackupNode like a Checkpoint, but in addition maintains an image in memory.

  28. OUTLINE • Introduction • Architecture • Hadoop Distribution File System • Architecture of HDFS • NameNode • DataNode • HDFS Client • Replica Management

  29. Responsibilities of Each DataNode • Storing blocks and theirs metadata • Sending block report and heartbeats to the NameNode

  30. Blocks &Metadata

  31. DataNode’s Version

  32. Verification Log

  33. Block Report • Once an hour • Contains block id, generation stamp and the size of each block • Is important information for Replica Management

  34. Heartbeats • Once every three seconds • To confirm the block replicas are available • Contains total storage capacity, fraction of storage in use and number of data transfers currently in progress • NameNode controls the DataNode by replying the heartbeats

  35. OUTLINE • Introduction • Architecture • Hadoop Distribution File System • Architecture of HDFS • NameNode • DataNode • HDFS Client • Replica Management

  36. Block Writing HDFS Client NN DN DN Request DN List DN DN DN Write Cluster NN: NameNode DN: DataNode

  37. Writing a Block

  38. File Appending File Data Read Read Client Client Write Appended Data Read Read Client Client

  39. Block Reading HDFS Client NN DN DN Request DN List DN DN DN Read Cluster NN: NameNode DN: DataNode

  40. OUTLINE • Introduction • Architecture • Hadoop Distribution File System • Architecture of HDFS • NameNode • DataNode • HDFS Client • Replica Management

  41. Topology Example Rack0 Rack1 N00 N01 N02 N10 N11 N12

  42. Read Example Rack0 Rack1 Client BR BR BR BR N00 N01 N02 N10 N11 N12 Selected Replica Block Replica

  43. Distance Example 1 Distance is 4 Rack0 Rack1 Client BR BR BR N00 N01 N02 N10 N11 N12 Selected Replica Block Replica

  44. Distance Example 2 Distance is 2 Rack0 Rack1 Client BR BR BR N00 N01 N02 N10 N11 N12 Selected Replica Block Replica

  45. Block Placement Rack0 Rack1 Client BR BR BR N00 N01 N02 N10 N11 N12 Block Replica

  46. Only one replica at one node

  47. Most two replicas in the same rack If the number of nodes Is twice the number of racks

  48. Replication Management Over-Replicated Under-Replicated

  49. Over-Replicated Rack0 Rack1 50% 51% 50% BR BR BR BR N00 N01 N02 N10 N11 N12 Disk Space Utilization Block Replica

  50. Under-Replicated Rack0 Rack1 BR BR BR N00 N01 N02 N10 N11 N12 Block Replica

More Related