Access to HDFS files from all computers of a cluster

Question

My hadoop the program originally was launched in a local mode, and now my purpose became start in completely distributed mode. For this purpose it is necessary to provide access to the files which reading is executed in the reducer and mapper functions, from all computers of a cluster and therefore I asked a question on http://answers.mapr.com/questions/4444/syntax-of-option-files-in-hadoop-script (also as it will be not known on what computer to be executed the mapper function (mapper from logic of the program there will be only one and the program will be launched only with one mapper), it is necessary to provide also access on all cluster to the file arriving on an input of the mapper function). In this regard I had a question: Whether it is possible to use hdfs-files directly: that is to copy beforehand files from file system of Linux in file system of HDFS (thereby as I assume, these files become available on all computers of a cluster if it not so, correct please) and then to use HDFS Java API for reading these files, in the reducer and mapper functions which are executing on computers of a cluster?

If on this question the response the positive, give please a copying example from file system of Linux in file system of HDFS and reading these files in java to the program by means of HDFS Java API and and record of its contents at java-string.

Answer 1

Copy all your input files to the master node (this can be done using scp ). Then login to your master node ( ssh ) and execute something like following to copy files from local filesystem to hdfs:

hadoop fs -put $localfilelocation $destination

Now in your hadoop jobs, you may use the input to be hdfs:///$destination . No need to use any extra API to read from HDFS.

If you really want to read files from HDFS and use as addiotional information other than the input files, then refer this .