在Nifi中从ExecuteScript传输多个flowFiles

Question

I am trying to generate multiple flowfiles from one flowfile using an ExecuteScript processor in python.

The ouputs flowfiles depend on one attribute for configuration and the input flowfile (xml content).

I tried many things but I always ends with error like :

• this flowfile is already marked for transfer
• transfer relationship not specified

Below the last version :

from org.apache.commons.io import IOUtils
from java.nio.charset import StandardCharsets
from org.apache.nifi.processor.io import StreamCallback
import java.io
from org.python.core.util import StringUtil

class PyStreamCallback(StreamCallback):
    def __init__(self, flowFile):
        global matched
        self.parentFlowFile = flowFile
        pass

    def process(self, inputStream, outputStream):
        try:
            text_content = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
            flowfiles_list = []

            new_xml = "blabla"
            outputStream.write(bytearray(new_xml.encode('utf-8')))

            for n in range(0,5):
                flowFile = session.create(self.parentFlowFile)
                if (flowFile != None):
                    flowFile = session.write(flowFile, "Nothing")
                    flowfiles_list.append(flowFile)

            for flow in flowfiles_list:
                session.transfer(flow, REL_SUCCESS)
        except:
            print('Error inside process')
            raise

originalFlowFile = session.get()
if(originalFlowFile != None):
    try :
        originalFlowFile = session.write(originalFlowFile, PyStreamCallback(originalFlowFile))
        session.remove(originalFlowFile)

    except Exception as e:
        originalFlowFile = session.putAttribute(originalFlowFile,'python_error', str(e))
        session.transfer(originalFlowFile, REL_FAILURE)

Can someone tell me what I am doing wrong and how to achieve what I want to do?

Answer 1

Here are some notes on your script:

1) You are subclassing StreamCallback and writing to the original flow file, but then you remove it later. StreamCallback is for when you want to overwrite the contents of the existing flow file. If you don't need to do that, you can use InputStreamCallback as the base class, it won't take an outputStream arg but you wouldn't need it in that case. You'd also use session.read on the original flow file rather than session.write .

2) The line flowFile = session.write(flowFile, "Nothing") isn't valid because session.write needs an OutputStreamCallback or StreamCallback as the argument (same as where you call it with PyStreamCallback below). When that throws an error, it gets raised all the way to the top level of the script, but by then you've created a flow file and didn't reach the statement that transfers the flowfiles_list to REL_SUCCESS. Consider adding a try/except around the session.write , then you could remove the newly created flow file and then raise the exception.

3) If you want to read the entire content of the incoming flow file into memory (which you are currently doing), then remove the original flow file and instead create new flow files from it, consider instead using the version of session.read() that returns an InputStream (ie doesn't require an InputStreamCallback ). Then you can save the contents into a global variable and/or pass it into an OutputStreamCallback when you want to do write something to the created flow files. Something like:

inputStream = session.read(originalFlowFile)
text_content = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
inputStream.close()
flowfiles_list = []

for n in range(0,5):
    flowFile = session.create(originalFlowFile)
    if (flowFile != None):
        try:
            flowFile = session.write(flowFile, PyStreamCallback(text_content))
            flowfiles_list.append(flowFile)
        except Exception as e:
            session.remove(flowFile)
            raise

for flow in flowfiles_list:
    session.transfer(flow, REL_SUCCESS)

session.remove(originalFlowFile)

This doesn't include the refactor of PyStreamCallback to be an OutputStreamCallback that takes a string arg instead of a FlowFile in the constructor.