inter process communication in hadoop n.
Skip this Video
Download Presentation
Inter-process Communication in Hadoop

Loading in 2 Seconds...

play fullscreen
1 / 14

Inter-process Communication in Hadoop - PowerPoint PPT Presentation

  • Uploaded on

Inter-process Communication in Hadoop. Different types of network interactions found in Hadoop (source Cloudera ).

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Inter-process Communication in Hadoop' - goro

Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
different types of network interactions found in hadoop source cloudera
Different types of network interactions found in Hadoop(source Cloudera)

1. Hadoop RPC calls – These are performed by clients using the Hadoop API, by MapReduce jobs, and among Hadoop services (JobTracker, TaskTrackers, NameNodes, DataNodes).

2. HDFS data transfer – Done when reading or writing data to HDFS, by clients using Hadoop API, by MapReduce jobs and among Hadoop services. HDFS data transfers are done using TCP/IP sockets directly.

3. MapReduce Shuffle – The shuffle part of a MapReduce job is the process of transferring data from the Map tasks to Reducer tasks. As this transfer is typically between different nodes in the cluster, the shuffle is done using the HTTP protocol.

4. Web UIs – The Hadoop daemons provide Web UIs for users and administrators to monitor jobs and the status of the cluster. The Web UIs use the HTTP protocol.

5. FSImage operations – These are metadata transfers between the NameNode and the Secondary NameNode. They are done using the HTTP protocol.

This means that Hadoop uses three different network communication protocols:

---Hadoop RPC, Direct TCP/IP, HTTP

ipc in hadoop distributed system
IPC in Hadoop Distributed System
  • IPC: Inter-process communication
  • Based on: Using Hadoop IPC/RPC for distributed applications, by Kelvin Tan
  • Hadoop IPC is the underlying mechanism is used whenever there is a need for out-of-process applications to communicate with one another.
hadoop ipc
Hadoop IPC
  • Hadoop IPC

1. uses binary serialization using and, unlike SOAP or XML-RPC.

2. is a simple low-fuss RPC mechanism.

3. is unicast does not support multi-cast.

  • Why use Hadoop IPC over RMI or
  • RMI: Remote method invocation
  • RPC: Remote Procedure Call
server code
Server code

Configuration conf = new Configuration();

Server server = RPC.getServer(this, "localhost", 16000, conf);

// start a server on localhost:16000


client code
Client code

Configuration conf = new Configuration();

InetSocketAddressaddr = new InetSocketAddress("localhost",16000);

// the server's inetsocketaddress

ClientProtocol client = (ClientProtocol)


ClientProtocol.versionID, addr, conf);

clientprotocol interface
ClientProtocol Interface

interface ClientProtocol extends org.apache.hadoop.ipc.VersionedProtocol {

public static final long versionID = 1L;

HeartbeatResponse heartbeat();


Note: HeartbeatReponse is a class


public class HeartbeatResponse implements {

String status;

public void write(DataOutput out) throws IOException {

UTF8.writeString(out, status);


public void readFields(DataInput in) throws IOException {

this.status = UTF8.readString(in);




1. A server which implements an interface (which itself extends VersionedProtocol)

2. 1 or more clients which call the interface methods.

3. Any method arguments or objects returned by methods must implement

Learn about RPC, RMI, …TCP/IP, .. SOAP, REST…

Also see ;)

now to answer troy s and saurab s questions
Now to answer Troy’s and Saurab’s questions:
  • Output of mapper is buffered
  • When the buffer is 80% full it is transferred to “local file system” (spill files) not to HDFS …no need… no need for replication. It is after all local…
  • “The map outputs are copied to the reduce tasktracker’s memory if they are small enough (the buffer’s size is controlled by mapred.job.shuffle.input.buffer.percent, which specifies the proportion of the heap to use for this purpose); otherwise, they are copied to disk. …”, T.White, Hadoop Definitive Guide
  • So the question is: How will you design/decide the size of the input split to the mapper? How many mappers?
  • Memory(RAM) size/heapsize, bandwidth between machines are important
there was a question about partitioner
There was a question about partitioner
  • Here is the code:

public interface Partitioner<K, V> extends JobConfigurable {

intgetPartition(K key, V value, intnumPartitions);


Return value is the partition # , that is the reducer#

hadoop task flow
Hadoop Task flow


#Mapper X #Reducertimes




Local Disk

What sort could the partiton, shuffle use?

  • parameters to tune in are:
  • io.sort.mb This is the output buffer of a mapper. When this buffer is full the data is sorted and spilled to disk. Ideally you avoid having to many spills. Note that this memory is part of the maptask heap size.
  • This is the heap size of a map task, the higher this is the higher you can put output buffer size.
  • In principle the number of reduce tasks also influences the shuffle speed. The number of reduce rounds is the total number of reduce slots / the number of reduce tasks. Note that the initial shuffle (during the map phase) will only shuffle data to the active reducers. So mapred.reduce.tasks is also relevant.
  • io.sort.factor is the number threads performing the merge sort, both on the map and the reduce side.
  • Compression also has a large impact (it speeds up the transfer from mapper to reducer but the compr/decompr comes at a cost!
  • mapred.job.shuffle.input.buffer.percent is the percentage of the reducer's heap to store map output in memory.
more info
More info
  • Increasing the sort buffer memory and the performance gain from that will depend on:
  • a)The size/total Number of the Keys being emitted by the mapper
  • b) The Nature of the Mapper Tasks : (IO intensive, CPU intensive)
  • c) Available Primary Memory, Map/Reduce Slots in the given Node
  • d) Data skewness