Read And Write Operation In HDFS

HDFS is hadoop distributed file system; in simple terms a file is stored in a distributed machines. The Hadoop framework was designed considering reliability, throughput, network I/O, and disk I/O; but compromised with latency, which is best in RDBMS.

These points are discussed in part I ie; HDFS Architecture HDFS Architecture .

How Read and Write operations are performed in HDFS

HDFS Write

• By default the replication factor(multiple copiesof blocks) for a block is 3.
• As Name Node receives write request from HDFS client (JVM), Name Node checks whether file is available or not as well as whether client is authorized or not (performs various checks) and returns multiple nodes.
• Step 3,4 and 5 will get repeated until the whole file gets written on HDFS.
• In case of Data Node failure-
• The data is written on the remaining two nodes.
• Name node notices under replication and arranges for replication.
• Same is the case with multiple node failure.

HDFS Write- Selection of the Data Nodes

How Data Nodes are selected by Name Nodes

• Any node within the cluster is chosen as the first node but it should not be too busy or overloaded.
• Second node is chosen as the first node is chosen.
• Third node is chosen to be on the same rack as the second one.This forms the pipeline.

Simulation on block distribution

• File is broken into blocks (64 mb) and then replicated and distributed across the file system.
  
  
  

• If one of the node/rack fails then also the replication of ( ) that block is available on other racks .
• Failure of multiple racks are more serious but less probable.
• Also, the whole procedure of selection and replication happens behind a curtain, no developer or client is able to see all this or has to worry about what happens in the background.

Node Distance

How is distance calculated in HDFS?

Idea of distance is based on bandwidth.

The only possible cases for calculating distance are-

• D=0, blocks on same node, same rack.
• D=2, blocks on different node,same rack.
• D=4, blocks on a node having different rack.
• D=6, blocks on a node having different data centre.

HDFS Read

If D7 fails, next D8 is picked

Failure cases,

• Data block is corrupted,
• Next node in the list is picked up.
• Data Node fails,
• Next node in the list is picked up.
• That node is not tried for the later blocks.