mergecontent处理器未提供预期结果

Question

I have json data in multiple small files (some times only one line in a file or a null file).

I want to merge all small files in to single large file.

I am getting a large file in an unexpected format.

ex:

file 1:

{"code"="1", "color"="green"}
{"code"="2", "color"="blue"}
{"code"="3", "color"="orange"}

file 2:

{"code"="4", "color"="yellow"}
{"code"="5", "color"="red"}

I am getting the below output after using MergeContent

{"code"="1", "color"="green"}
{"code"="2", "color"="blue"}
{"code"="3", "color"="orange"}{"code"="4", "color"="yellow"}
{"code"="5", "color"="red"}

Expected output

{"code"="1", "color"="green"}
{"code"="2", "color"="blue"}
{"code"="3", "color"="orange"}
{"code"="4", "color"="yellow"}
{"code"="5", "color"="red"}

Any help is appreciated Thank you

Answer 1

This is likely because file-1 does not have a new-line character after the last line. The bin-packing merge is literally writing the raw bytes of each flow file one after another with no concept of what is in the bytes, so the bytes of the second file go right after the bytes of the first file.

The are properties for Header, Demarcator, and Footer which will get inserted around the bytes accordingly. So using "Delimiter Strategy" of "Text" and entering shift+enter into the "Demarcator" value will tell it to enter a new-line in between each batch of bytes.

Keep in mind if some files do end in new-lines then you will sometimes get two new-lines in a row with this approach. You could probably filter that out after the fact using RouteText, or try to clean it up before hand using ReplaceText.