使用Java的Apache Spark流自定义接收器（文本文件）

Question

I'm new to Apache Spark.

I need to read the log files from local/mounted directory. Some external source writing the files into local/mounted directory. Eg External source writing logs into combined_file.txt file and once file writing completed the external source create new file with prefix 0_ , like 0_combined_file.txt . Then i need to read the combined_file.txt log file and process it. So I'm trying to write the custom receiver to check log file writing into local/mounted directory is completed and then read the completed file.

Here is my code

@Override
    public void onStart() {
        Runnable th = () -> {
            while (true) {
                try {
                    Thread.sleep(1000l);
                    File dir = new File("/home/PK01/Desktop/arcflash/");
                    File[] completedFiles = dir.listFiles((dirName, fileName) -> {
                        return fileName.toLowerCase().startsWith("0_");
                    });
                    //metaDataFile --> 0_test.txt
                    //completedFiles --> test.txt
                    for (File metaDataFile : completedFiles) {
                        String compFileName = metaDataFile.getName();
                        compFileName = compFileName.substring(2, compFileName.length());
                        File dataFile = new File("/home/PK01/Desktop/arcflash/" + compFileName);
                        if (dataFile.exists()) {
                            byte[] data = new byte[(int) dataFile.length()];
                            fis.read(data);
                            fis.close();
                            store(new String(data));
                            dataFile.delete();
                            metaDataFile.delete();
                        }
                    }
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }
        };
    new Thread(th);
    }

I'm trying to process the data like below.

JavaReceiverInputDStream<String> data = jssc.receiverStream(receiver);
data.foreachRDD(fileStreamRdd -> {
                        processOnSingleFile(fileStreamRdd.flatMap(streamBatchData -> {
                        return Arrays.asList(streamBatchData.split("\\n")).iterator();
                    }));
});

But getting below exception

18/01/19 12:08:39 WARN RandomBlockReplicationPolicy: Expecting 1 replicas with only 0 peer/s.
18/01/19 12:08:39 WARN BlockManager: Block input-0-1516343919400 replicated to only 0 peer(s) instead of 1 peers
18/01/19 12:08:40 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 1)
java.lang.OutOfMemoryError: Java heap space
    at com.esotericsoftware.kryo.io.Output.<init>(Output.java:60)
    at org.apache.spark.serializer.KryoSerializer.newKryoOutput(KryoSerializer.scala:91)
    at org.apache.spark.serializer.KryoSerializerInstance.output$lzycompute(KryoSerializer.scala:308)
    at org.apache.spark.serializer.KryoSerializerInstance.output(KryoSerializer.scala:308)
    at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:312)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:383)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
18/01/19 12:08:40 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker for task 1,5,main]
java.lang.OutOfMemoryError: Java heap space
    at com.esotericsoftware.kryo.io.Output.<init>(Output.java:60)
    at org.apache.spark.serializer.KryoSerializer.newKryoOutput(KryoSerializer.scala:91)
    at org.apache.spark.serializer.KryoSerializerInstance.output$lzycompute(KryoSerializer.scala:308)
    at org.apache.spark.serializer.KryoSerializerInstance.output(KryoSerializer.scala:308)
    at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:312)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:383)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
18/01/19 12:08:40 WARN TaskSetManager: Lost task 0.0 in stage 1.0 (TID 1, localhost, executor driver): java.lang.OutOfMemoryError: Java heap space
    at com.esotericsoftware.kryo.io.Output.<init>(Output.java:60)
    at org.apache.spark.serializer.KryoSerializer.newKryoOutput(KryoSerializer.scala:91)
    at org.apache.spark.serializer.KryoSerializerInstance.output$lzycompute(KryoSerializer.scala:308)
    at org.apache.spark.serializer.KryoSerializerInstance.output(KryoSerializer.scala:308)
    at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:312)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:383)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

Anybody can help me to resolve the error here.

Any help will be appreciate

Answer 1

18/01/19 12:08:40 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker for task 1,5,main] java.lang.OutOfMemoryError: Java heap space

The above show that you are hitting an out of memory error.Increase the memory explicitly while submitting the spark job