避免在大型数据集上使用收集

Question

We're using Apache Spark for processing. We have several steps where it is necessary to use collect() to to a JavaRDD to a list, but we are wanting to avoid doing this in order to operate on a list. We know we want to avoid this because it brings everything back to the driver. It ends up and we run out of memory because we are processing anywhere from 5million - 200 million records. Here's an example of what we have so far.

private InputStream createCSVObject(JavaRDD<Object[]> args) {
        System.out.println("inside createCSVObject");
        try {
            StringBuilder value = new StringBuilder(CHUNK_SIZE);

            args.collect().forEach(i -> {
                value.append(i[0].toString());
                for (int j = 1; j < i.length; ++j) {
                    value.append("," + i[j]);
                }
                value.append("\n");
            });
            System.out.println("Out of createCSVObject for loops");
            byte[] strBytes = value.toString().getBytes();

            InputStream myInputStream = new ByteArrayInputStream(strBytes);
            return (myInputStream);
        } catch (Exception e) {
            System.err.println(String.format("ERROR: FileWriterService - writeFile: %s", e.getMessage()));
            return null;
        }
    }

I've searched for this over and over across SO and google, and can't come up with anything definitive. Does anyone have any ideas???

Note: the COLLECT at args.collect()

EDIT:

After looking into the proposed answer below we devised a simple proof of concept for it, and what we came up with does one iteration through every 40s. The logic is not complex, why is it so slow?

        System.out.println("inside createCSVObject");
        try {
            StringBuilder value = new StringBuilder();
            System.out.println("args length " + args.toLocalIterator().next().length);

             while (args.toLocalIterator().hasNext()) {
                 Object[] objects = args.toLocalIterator().next();
                 System.out.println("Inside iterator");
                 value.append(objects[0].toString());
                 for (int j = 1; j < objects.length; ++j) {
                     value.append("," + objects[j]);
                 }
                 value.append("\n");
             }

            System.out.println("Out of createCSVObject for loops");
            byte[] strBytes = value.toString().getBytes();

            InputStream myInputStream = new ByteArrayInputStream(strBytes);
            return (myInputStream);
        } catch (Exception e) {
            System.err.println(String.format("ERROR: FileWriterService - writeFile: %s", e.getMessage()));
            e.printStackTrace();
            return null;
        }

Answer 1

You can use JavaRDD.toLocalIterator() to iterate through the entire RDD on the driver without collecting it all into a list. Instead, it brings each partition over to the driver one at a time, so doesn't use more memory than the size of the largest partition ( documentation ).

Obviously, in the exmple you've given, you still have the problem that you're collecting everything into a massive byte array, which will still use quite a lot of memory. Instead, you could write a custom InputStream class that wraps an Iterator (as returned by toLocalIterator ), and only buffers one element at a time, calling next() on the iterator only when InputStream.read() demands more data.