[英]Python ExecuteScript in NiFi: Transform flowfile attributes & content
我正在嘗試在 NiFi 中創建一個 Python 腳本:
到目前為止我所做的:
import json
import java.io
from org.apache.commons.io import IOUtils
from java.nio.charset import StandardCharsets
from org.apache.nifi.processor.io import StreamCallback,InputStreamCallback, OutputStreamCallback
class OutputWrite(OutputStreamCallback, obj):
def __init__(self):
self.obj = obj
def process(self, outputStream):
outputStream.write(bytearray(json.dumps(self.obj).encode('utf')))
###end class###
flowfile = session.get()
if flowfile != None:
**#1) Get flowfile attributes**
headers = {
'Accept-Encoding': 'gzip, deflate, br',
'Accept': 'application/json, text/plain, */*',
'Cache-Control': 'no-cache',
'Ocp-Apim-Trace': 'true',
'Authorization': flowfile.getAttribute('Authorization')
}
collection = flowfile.getAttribute('collection')
dataset = flowfile.getAttribute('dataset')
**#2)Get flowfile content**
stream_content = session.read(flowfile)
text_content = IOUtils.toString(stream_content, StandardCharsets.UTF_8)
json_content = json.loads(text_content)
records = json_content['result']['count']
pages = records/10000
**#3) Write flowfile attributes**
flowfile = session.putAttribute(flowfile, 'collection', collection)
flowfile = session.putAttribute(flowfile, 'dataset', dataset)
**#API operations: output_json with desired data**
output_json = {some data}
**#4) Write final JSON data to output flowfile**
flowfile = session.write(flowfile, OutputWrite(output_json))
session.transfer(flowfile, REL_SUCCESS)
session.commit()
我的問題是我找不到將所需 output_json 對象的引用作為參數傳遞給 OutputStreamCallback 類的方法。 關於如何解決這個問題或更好的方法的任何想法?
在這種情況下,在類的 process 函數中執行所有 API 操作是否可能更容易,但是我如何訪問 process 函數中的傳入流文件屬性(需要會話或流文件對象)?
非常感謝任何幫助!
你可以試試這樣的——
import json
import sys
import traceback
from java.nio.charset import StandardCharsets
from org.apache.commons.io import IOUtils
from org.apache.nifi.processor.io import StreamCallback
from org.python.core.util import StringUtil
class TransformCallback(StreamCallback):
def __init__(self):
pass
def process(self, inputStream, outputStream):
try:
# Read input FlowFile content
input_text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
input_obj = json.loads(input_text)
# Transform content
output_obj = input_obj #your input content
#perform Data tranformation on output_obj
# Write output content
output_text = json.dumps(outputJson)
outputStream.write(StringUtil.toBytes(output_text))
except:
traceback.print_exc(file=sys.stdout)
raise
flowFile = session.get()
if flowFile != None:
flowFile = session.write(flowFile, TransformCallback())
# Finish by transferring the FlowFile to an output relationship
session.transfer(flowFile, REL_SUCCESS)
我在下面包含了示例 Python 代碼,該代碼允許自定義PyStreamCallback
類,該類實現邏輯以轉換來自Matt Burgess 的關於該主題的博客文章的流文件內容中的 JSON,但我鼓勵您考慮使用本機處理器進行UpdateAttribute
和EvaluateJSONPath
執行相關活動,並且僅在特別需要執行 NiFi 無法立即處理的任務時才使用自定義代碼。
import json
import java.io
from org.apache.commons.io import IOUtils
from java.nio.charset import StandardCharsets
from org.apache.nifi.processor.io import StreamCallback
class PyStreamCallback(StreamCallback):
def __init__(self):
pass
def process(self, inputStream, outputStream):
text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
obj = json.loads(text)
newObj = {
"Range": 5,
"Rating": obj['rating']['primary']['value'],
"SecondaryRatings": {}
}
for key, value in obj['rating'].iteritems():
if key != "primary":
newObj['SecondaryRatings'][key] = {"Id": key, "Range": 5, "Value": value['value']}
outputStream.write(bytearray(json.dumps(newObj, indent=4).encode('utf-8')))
flowFile = session.get()
if (flowFile != None):
flowFile = session.write(flowFile,PyStreamCallback())
flowFile = session.putAttribute(flowFile, "filename", flowFile.getAttribute('filename').split('.')[0]+'_translated.json')
session.transfer(flowFile, REL_SUCCESS)
更新:
要在回調中訪問流文件的屬性,只需將其作為參數傳遞給構造函數,將其存儲為字段,並在process
方法中引用它。 這是一個非常簡單的示例,它將屬性my_attr
的值連接到傳入的流文件內容並將其寫回:
import json
import java.io
from org.apache.commons.io import IOUtils
from java.nio.charset import StandardCharsets
from org.apache.nifi.processor.io import StreamCallback
class PyStreamCallback(StreamCallback):
def __init__(self, flowfile):
self.ff = flowfile
pass
def process(self, inputStream, outputStream):
text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
text += self.ff.getAttribute('my_attr')
outputStream.write(bytearray(text.encode('utf-8')))
flowFile = session.get()
if (flowFile != None):
flowFile = session.write(flowFile,PyStreamCallback(flowFile))
session.transfer(flowFile, REL_SUCCESS)
傳入流文件:
--------------------------------------------------
Standard FlowFile Attributes
Key: 'entryDate'
Value: 'Tue Mar 13 13:10:48 PDT 2018'
Key: 'lineageStartDate'
Value: 'Tue Mar 13 13:10:48 PDT 2018'
Key: 'fileSize'
Value: '30'
FlowFile Attribute Map Content
Key: 'filename'
Value: '1690494181462176'
Key: 'my_attr'
Value: 'This is an attribute value.'
Key: 'path'
Value: './'
Key: 'uuid'
Value: 'dc93b715-50a0-43ce-a4db-716bd9ec3205'
--------------------------------------------------
This is some flowfile content.
傳出流文件:
--------------------------------------------------
Standard FlowFile Attributes
Key: 'entryDate'
Value: 'Tue Mar 13 13:10:48 PDT 2018'
Key: 'lineageStartDate'
Value: 'Tue Mar 13 13:10:48 PDT 2018'
Key: 'fileSize'
Value: '57'
FlowFile Attribute Map Content
Key: 'filename'
Value: '1690494181462176'
Key: 'my_attr'
Value: 'This is an attribute value.'
Key: 'path'
Value: './'
Key: 'uuid'
Value: 'dc93b715-50a0-43ce-a4db-716bd9ec3205'
--------------------------------------------------
This is some flowfile content.This is an attribute value.
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.