How best to interact with AWS RDS Postgres via NiFi

Question

I have a FlowFile and I want to insert the attributes into RDS. If this was a local machine, I'd create a DBCPConnectionPool, reference a JDBC driver, etc.

With RDS, what am I supposed to do? Something similar (how would I do this on AWS)? Or am I stuck using ExecuteScript? If it's the later, is there a Python example for how to do this?

Answer 1

Question might not have been clear based on the feedback, but here is the answer to get a NiFi (running on an AWS EC2 instance) communicating with an Amazon RDS instance:

1. On the EC2 instance, download the latest JDBC driver ( wget "https://driver.jar" )
2. (If needed) Move the JDBC driver into a safe folder.
3. Create the DBCPConnectionPool, referencing the fully-resolved file path to the driver.jar (helpful: use readlink -f driver.jar to get the path).
4. Don't forget -- under your AWS Security Groups, add an inbound rule that allows your EC2 instance to access RDS (under Source , you should put the security group of your EC2 instance).

Answer 2

assume that all the values that you want to insert into database are in flowfile attributes.

use ReplaceText processor to replace content of the flow file with sql insert statement:

insert into mytable (id, message, start_date) values(?, ?, ?)

then use PutSQL processor with following properties to specify three parameters ( ? ) of sql statement

sql.args.1.type   = 2               # NUMERIC
sql.args.1.value  = ${MY_ATTR_ID}   # get value from attribute or context variable MY_ATTR_ID

sql.args.2.type   = 12              # VARCHAR
sql.args.2.value  = ${MY_ATTR_MSG}

sql.args.3.type   = 93              # TIMESTAMP
sql.args.3.value  = ${MY_ATTR_DATE}
sql.args.3.format = yyyy-MM-dd HH:mm:ss.SSS

all the jdbc data types are listed here:

https://docs.oracle.com/javase/7/docs/api/constant-values.html#java.sql.Types.ARRAY