NiFi-1.0.0 - ExecuteSQL CrateIO

Question

I am trying to execute an sql statement (ExecuteSql proc) for creating a table, table's name is based on a filename attribute, and when ExecuteSQl gets executed, it creates the table but does not route the file to any relationship.

I think this is in part due to the fact that in order to create a table in CrateIO you need to put ";" at the end. If I do that I also get an error.

Any ideas ?

Answer 1

ExecuteSQL is intended for queries that return a ResultSet. Try PutSQL for things like table creation (and other DDL statements). You shouldn't need a semicolon there either, but you can try it both ways to see what CrateIO likes.

EDIT (reply to comments): If you are specifying the query in ExecuteSQL, it probably looks something like:

CREATE TABLE ${filename} (myColumn1 STRING, myColumn2 INT, ...)

You can do a similar thing with a ReplaceText processor after the GetFile and before PutSQL. ReplaceText will set the content of the flow file to your SQL query, which is what PutSQL wants.

EDIT 2 : If you want custom logic, you could consider a scripting processor vs writing a full custom processor. Then you could do the generation of a CREATE TABLE followed by a translation of CSV rows to INSERT statements. If you find yourself wanting to connect to a DB, check my blog post on how to use DBCP Connection Pools in a scripting processor.