Inter process communication in hadoop
This presentation is the property of its rightful owner.
Sponsored Links
1 / 14

Inter-process Communication in Hadoop PowerPoint PPT Presentation


  • 119 Views
  • Uploaded on
  • Presentation posted in: General

Inter-process Communication in Hadoop. Different types of network interactions found in Hadoop (source Cloudera ).

Download Presentation

Inter-process Communication in Hadoop

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Inter process communication in hadoop

Inter-process Communication in Hadoop


Different types of network interactions found in hadoop source cloudera

Different types of network interactions found in Hadoop(source Cloudera)

1. Hadoop RPC calls – These are performed by clients using the Hadoop API, by MapReduce jobs, and among Hadoop services (JobTracker, TaskTrackers, NameNodes, DataNodes).

2. HDFS data transfer – Done when reading or writing data to HDFS, by clients using Hadoop API, by MapReduce jobs and among Hadoop services. HDFS data transfers are done using TCP/IP sockets directly.

3. MapReduce Shuffle – The shuffle part of a MapReduce job is the process of transferring data from the Map tasks to Reducer tasks. As this transfer is typically between different nodes in the cluster, the shuffle is done using the HTTP protocol.

4. Web UIs – The Hadoop daemons provide Web UIs for users and administrators to monitor jobs and the status of the cluster. The Web UIs use the HTTP protocol.

5. FSImage operations – These are metadata transfers between the NameNode and the Secondary NameNode. They are done using the HTTP protocol.

This means that Hadoop uses three different network communication protocols:

---Hadoop RPC, Direct TCP/IP, HTTP


Ipc in hadoop distributed system

IPC in Hadoop Distributed System

  • IPC: Inter-process communication

  • Based on: Using Hadoop IPC/RPC for distributed applications, by Kelvin Tan

  • http://www.supermind.org/blog/520/using-hadoop-ipcrpc-for-distributed-applications

  • Hadoop IPC is the underlying mechanism is used whenever there is a need for out-of-process applications to communicate with one another.


Hadoop ipc

Hadoop IPC

  • Hadoop IPC

    1. uses binary serialization using java.io.DataOutputStream and java.io.DataInputStream, unlike SOAP or XML-RPC.

    2. is a simple low-fuss RPC mechanism.

    3. is unicast does not support multi-cast.

  • Why use Hadoop IPC over RMI or java.io.Serialization?

  • RMI: Remote method invocation

  • RPC: Remote Procedure Call


Server code

Server code

Configuration conf = new Configuration();

Server server = RPC.getServer(this, "localhost", 16000, conf);

// start a server on localhost:16000

server.start();


Client code

Client code

Configuration conf = new Configuration();

InetSocketAddressaddr = new InetSocketAddress("localhost",16000);

// the server's inetsocketaddress

ClientProtocol client = (ClientProtocol)

RPC.waitForProxy(ClientProtocol.class,

ClientProtocol.versionID, addr, conf);


Clientprotocol interface

ClientProtocol Interface

interface ClientProtocol extends org.apache.hadoop.ipc.VersionedProtocol {

public static final long versionID = 1L;

HeartbeatResponse heartbeat();

}

Note: HeartbeatReponse is a class


Heartbeatresponse

HeartbeatResponse

public class HeartbeatResponse implements org.apache.hadoop.io.Writable {

String status;

public void write(DataOutput out) throws IOException {

UTF8.writeString(out, status);

}

public void readFields(DataInput in) throws IOException {

this.status = UTF8.readString(in);

}

}


Summary

Summary

1. A server which implements an interface (which itself extends VersionedProtocol)

2. 1 or more clients which call the interface methods.

3. Any method arguments or objects returned by methods must implement org.apache.hadoop.io.Writable

Learn about RPC, RMI, …TCP/IP, .. SOAP, REST…

Also see ;)

http://www.cse.buffalo.edu/~bina/cse486/spring2008/RMIJan30.ppt


Now to answer troy s and saurab s questions

Now to answer Troy’s and Saurab’s questions:

  • Output of mapper is buffered

  • When the buffer is 80% full it is transferred to “local file system” (spill files) not to HDFS …no need… no need for replication. It is after all local…

  • “The map outputs are copied to the reduce tasktracker’s memory if they are small enough (the buffer’s size is controlled by mapred.job.shuffle.input.buffer.percent, which specifies the proportion of the heap to use for this purpose); otherwise, they are copied to disk. …”, T.White, Hadoop Definitive Guide

  • So the question is: How will you design/decide the size of the input split to the mapper? How many mappers?

  • Memory(RAM) size/heapsize, bandwidth between machines are important


There was a question about partitioner

There was a question about partitioner

  • Here is the code:

    public interface Partitioner<K, V> extends JobConfigurable {

    intgetPartition(K key, V value, intnumPartitions);

    }

    Return value is the partition # , that is the reducer#


Hadoop task flow

Hadoop Task flow

Copy

#Mapper X #Reducertimes

Buffer/Cache

HDFS

HDFS

Local Disk

What sort could the partiton, shuffle use?


Improvements

Improvements?

  • parameters to tune in mapred.site.xml are:

  • io.sort.mb This is the output buffer of a mapper. When this buffer is full the data is sorted and spilled to disk. Ideally you avoid having to many spills. Note that this memory is part of the maptask heap size.

  • mapred.map.child.java.opts This is the heap size of a map task, the higher this is the higher you can put output buffer size.

  • In principle the number of reduce tasks also influences the shuffle speed. The number of reduce rounds is the total number of reduce slots / the number of reduce tasks. Note that the initial shuffle (during the map phase) will only shuffle data to the active reducers. So mapred.reduce.tasks is also relevant.

  • io.sort.factor is the number threads performing the merge sort, both on the map and the reduce side.

  • Compression also has a large impact (it speeds up the transfer from mapper to reducer but the compr/decompr comes at a cost!

  • mapred.job.shuffle.input.buffer.percent is the percentage of the reducer's heap to store map output in memory.


More info

More info

  • Increasing the sort buffer memory and the performance gain from that will depend on:

  • a)The size/total Number of the Keys being emitted by the mapper

  • b) The Nature of the Mapper Tasks : (IO intensive, CPU intensive)

  • c) Available Primary Memory, Map/Reduce Slots in the given Node

  • d) Data skewness


  • Login