无法在Java Spark中读取文件

Question

I am trying to run the spark program on java using eclipse. Its is running if i simply print something on console but I am not able to read any file using textFile function. I have read somewhere that reading a file can only be done using HDFS but I am not able to do in my local system. Do let me know how to access/read file , if using HDFS then how to install HDFS in my local system so that i can rad the text file.

Here's a code on which I am testing , though this program is working fine but it is unable to read file saying Input path does not exist.

package spark;

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.sql.DataFrame;
import org.apache.spark.sql.SQLContext;

import org.apache.spark.api.java.function.Function;

public class TestSpark {

    public static void main(String args[])
    {
        String[] jars = {"D:\\customJars\\spark.jar"};
        System.setProperty("hadoop.home.dir", "D:\\hadoop-common-2.2.0-bin-master");
        SparkConf sparkConf = new SparkConf().setAppName("spark.TestSpark")
                .setMaster("spark://10.1.50.165:7077")
                .setJars(jars);

        JavaSparkContext jsc = new JavaSparkContext(sparkConf);
        SQLContext sqlcon = new SQLContext(jsc);
        String inputFileName = "./forecaster.txt" ;
        JavaRDD<String> logData = jsc.textFile(inputFileName);
        long numAs = logData.filter(new Function<String, Boolean>() {

            @Override
            public Boolean call(String s) throws Exception {
                return s.contains("a");
            }
        }).count();

        long numBs = logData.filter(new Function<String, Boolean>() {
              public Boolean call(String s) { return s.contains("b"); }
            }).count();

         System.out.println("Lines with a: " + numAs + ", lines with b: " + numBs);
        System.out.println("sadasdasdf");

        jsc.stop();
        jsc.close();
    }

}

My File Struture :

Answer 1

Update : you don't have .txt extension in file name and you are using it in your application. You should use it as String inputFileName = "forecaster" ;

If file is in same folder as java class TestSpark ( $APP_HOME ):

String inputFileName = "forecaster.txt" ;

If file is in Data dir under your project of spark:

String inputFileName = "Data\\forecaster.txt" ;

Or use fully qualified Path log says from below testing:

16/08/03 08:25:25 INFO HadoopRDD: Input split: file:/C:/Users/user123/worksapce/spark-java/forecaster.txt
~~~~~~~
String inputFileName = "file:/C:/Users/user123/worksapce/spark-java/forecaster.txt" ;

For example: I copied your code and ran on my local environment:

this is how my project step up is, and I run it as:

 String inputFileName = "forecaster.txt" ;

Test File:

this is test file
aaa
bbb
ddddaaee
ewwww
aaaa
a
a
aaaa
bb

Code that I used:

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.Function;

public class TestSpark {

    public static void main(String args[])
    {
       // String[] jars = {"D:\\customJars\\spark.jar"};
       // System.setProperty("hadoop.home.dir", "D:\\hadoop-common-2.2.0-bin-master");
        SparkConf sparkConf = new SparkConf().setAppName("spark.TestSpark").setMaster("local");
                //.setMaster("spark://10.1.50.165:7077")
                //.setJars(jars);

        JavaSparkContext jsc = new JavaSparkContext(sparkConf);
        //SQLContext sqlcon = new SQLContext(jsc);
        String inputFileName = "forecaster.txt" ;
        JavaRDD<String> logData = jsc.textFile(inputFileName);
        long numAs = logData.filter(new Function<String, Boolean>() {

            @Override
            public Boolean call(String s) throws Exception {
                return s.contains("a");
            }
        }).count();

        long numBs = logData.filter(new Function<String, Boolean>() {

            public Boolean call(String s) { return s.contains("b"); }
            }).count();

         System.out.println("Lines with a: " + numAs + ", lines with b: " + numBs);
        System.out.println("sadasdasdf");

        jsc.stop();
        jsc.close();
    }

}

Answer 2

Spark needs schema and proper path in order to understand how to read the file. So if you are reading from HDFS, you should use:

jsc.textFile("hdfs://host/path/to/hdfs/file/input.txt");

If you are reading local file (local to the worker node, not the machine the driver is running), you should use:

jsc.textFile("file://path/to/hdfs/file/input.txt");

For reading Hadoop Archive File (HAR), you should use:

jsc.textFile("har://archive/path/to/hdfs/file/input.txt");

And so on.