具有Groovy的Nifi ExecuteScript：OOM异常

Question

I am using an ExecuteScript processor in NiFi to execute a PUT command with the flowfile I have specified. However, right now I am getting an OutOfMemory exception when the file is greater than 2GB. My memory settings on NiFi are much higher. I've also seen much larger flow files work with some of the other provided Nifi processor.

I cannot use InvokeHTTP as I have 2 different return codes with conflicting relationships that I need to handle.

Currently my JVM settings for NiFi are

java.arg.2=-Xms16g
java.arg.3=-Xmx32g

Here is a snippet of how the data is getting PUT.

    InputStream i = session.read(flowFile)
    def baseUrl = new URL(Location)
    def connection = baseUrl.openConnection()
    connection.setDoOutput(true)
    connection.setRequestMethod('PUT')
    connection.connect()

    OutputStream os = connection.getOutputStream()
    os << i
    i.close()

    os.flush()
    os.close()

Is there a setting for groovy in NiFi that needs to be set to increase this memory limit? Currently running NiFi 1.9.2

Answer 1

Issue was solved using an inputstream read buffer AND adding ChunkedStreamingMode

connection.setChunkedStreamingMode(1024)

byte[] buffer = new byte[1024];
int len;
while ((len = i.read(buffer)) != -1) {
    os.write(buffer, 0, len);
    os.flush;
}