feedburner
Enter your email address:

Delivered by FeedBurner

feedburner count

Hadoop File System Java Tutorial

Labels: , , ,

Hadoop Distributed File System (HDFS) Java tutorial.


This Java tutorial contains examples and Java code on how to create, rename, delete and do much more on Hadoop Distributed File System using the Haddop Java API.

Copy a file from the local file system to HDFS


The srcFile variable needs to contain the full name (path + file name) of the file in the local file system. The dstFile variable needs to contain the desired full name of the file in the Hadoop file system.
  Configuration config = new Configuration();
  FileSystem hdfs = FileSystem.get(config);
  Path srcPath = new Path(srcFile);
  Path dstPath = new Path(dstFile);
  hdfs.copyFromLocalFile(srcPath, dstPath);

Create HDFS file


The fileName variable contains the file name and path in the Hadoop file system. The content of the file is the buff variable which is an array of bytes.
  //byte[] buff - The content of the file
  Configuration config = new Configuration();
  FileSystem hdfs = FileSystem.get(config);
  Path path = new Path(fileName);
  FSDataOutputStream outputStream = hdfs.create(path);
  outputStream.write(buff, 0, buff.length);

Rename HDFS file


In order to rename a file in Hadoop file system, we need the full name (path + name) of the file we want to rename. The rename method returns true if the file was renamed, otherwise false.
  Configuration config = new Configuration();
  FileSystem hdfs = FileSystem.get(config);
  Path fromPath = new Path(fromFileName);
  Path toPath = new Path(toFileName);
  boolean isRenamed = hdfs.rename(fromPath, toPath);

Delete HDFS file


In order to delete a file in Hadoop file system, we need the full name (path + name) of the file we want to delete. The delete method returns true if the file was deleted, otherwise false.
  Configuration config = new Configuration();
  FileSystem hdfs = FileSystem.get(config);
  Path path = new Path(fileName);
  boolean isDeleted = hdfs.delete(path, false);

Recursive delete:
  Configuration config = new Configuration();
  FileSystem hdfs = FileSystem.get(config);
  Path path = new Path(fileName);
  boolean isDeleted = hdfs.delete(path, true);

Get HDFS file last modification time


In order to get the last modification time of a file in Hadoop file system, we need the full name (path + name) of the file.
  Configuration config = new Configuration();
  FileSystem hdfs = FileSystem.get(config);
  Path path = new Path(fileName);
  FileStatus fileStatus = hdfs.getFileStatus(path);
  long modificationTime = fileStatus.getModificationTime

Check if a file exists in HDFS


In order to check the existance of a file in Hadoop file system, we need the full name (path + name) of the file we want to check. The exists methods returns true if the file exists, otherwise false.
  Configuration config = new Configuration();
  FileSystem hdfs = FileSystem.get(config);
  Path path = new Path(fileName);
  boolean isExists = hdfs.exists(path);

Get the locations of a file in the HDFS cluster


A file can exist on more than one node in the Hadoop file system cluster for two reasons:
  1. Based on the HDFS cluster configuration, Hadoop saves parts of files on different nodes in the cluster.
  2. Based on the HDFS cluster configuration, Hadoop saves more than one copy of each file on different nodes for redundancy (The default is three).
  Configuration config = new Configuration();
  FileSystem hdfs = FileSystem.get(config);
  Path path = new Path(fileName);
  FileStatus fileStatus = hdfs.getFileStatus(path);

  BlockLocation[] blkLocations = hdfs.getFileBlockLocations(path, 0, fileStatus.getLen());
    
  int blkCount = blkLocations.length;
  for (int i=0; i < blkCount; i++) {
    String[] hosts = blkLocations[i].getHosts();
    // Do something with the block hosts
  }

Get a list of all the nodes host names in the HDFS cluster


This method casts the FileSystem Object to a DistributedFileSystem Object. This method will work only when Hadoop is configured as a cluster. Running Hadoop on the local machine only, in a non cluster configuration will cause this method to throw an Exception.
  Configuration config = new Configuration();
  FileSystem fs = FileSystem.get(config);
  DistributedFileSystem hdfs = (DistributedFileSystem) fs;
  DatanodeInfo[] dataNodeStats = hdfs.getDataNodeStats();
  String[] names = new String[dataNodeStats.length];
  for (int i = 0; i < dataNodeStats.length; i++) {
      names[i] = dataNodeStats[i].getHostName();
  }


Did you like it? Digg it!

9 comments:
gravatar
Unknown said...
February 11, 2009 at 1:12 PM  

i put the code inside local localtohdfs.java
bt i get error message typically

Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
100 errors

how to rectify this?

gravatar
Anonymous said...
February 11, 2009 at 2:35 PM  

@nirvana74v
Thanks for your response. I wrote the original HDFS tutorial for Hadoop 0.14.
Although The delete() and getFileCacheHints() are deprecated you can still use this code. I made some modifications to the examples and now it no longer contains deprecate code.

gravatar
doolittle said...
June 2, 2010 at 3:22 PM  

BlockLocation[] blkLocations = hdfs.getFileBlockLocations(path, 0, fileStatus.getLen());

seem should be

BlockLocation[] blkLocations = hdfs.getFileBlockLocations(filestatus, 0, fileStatus.getLen());

gravatar
David said...
March 28, 2011 at 11:11 PM  

Do you have a post for map/reducing after adding files using online java code not command line? Thanks a lot, David

gravatar
David said...
March 28, 2011 at 11:12 PM  

Thanks for the post. Do you have more snippets for running the map/reduce using online java code (not command line)? Thanks again, David

gravatar
Moses said...
August 14, 2011 at 3:07 PM  

Good tutorial!
It helps me so much!

gravatar
Karol said...
April 11, 2012 at 3:50 PM  

Nice post but how do you read a files inside directory if you dont know the names of the files but you know directory is not empy ?

gravatar
Kumar said...
April 18, 2012 at 7:07 AM  

Hi,
I am looking at appending data to an existing file in HDFS, is there any program to perform the same.

Thanks,
Kumar

gravatar
MANOJ BABU said...
July 21, 2012 at 4:47 AM  

Good stuff. Really helps.

Post a Comment