How to pass SQL query as a context variable in Talend Open Studio job

Question

I am creating a job to pull data from a database to CSV file using talend open studio. There are 100 of tables, the data types and no of columns differ in the tables, I want to pull the data from database tables with a single job and customizable SQL query. I know how to create and use context variables.

Answer 1

If I understood you correctly you should be using tMap's reload at each row -option and defining table names in Excel sheet or in tFixedFlowInput.

tMap settings

Whole job and results

SQL Script:

"SELECT TOP(1) Name,  Code from mdm." + (String)globalMap.get("row4.table")

I used Microsoft SQL Server for example but same script works as well with MySQL.

Answer 2

You can simply use a context-variable which you set via the --context_param argument in a tWhicheverDatabaseInput. Eg define a context variable "my_sql" which you can set in the commandline as

my_job.sh --context_param my_sql="select a,b,c from a_test_table"

and then use context.my_sql as the SQL in you database input component.

However, as garpitmzn already mentioned, you will need dynamic schemas to actually work with this unknown structure in Talend. This feature only exists in the enterprise version.

If the enterprise version is available to you, simply declare a single column of type "Dynamic", which you can pass through the rest of your flow.

Answer 3

1. Declare a local context say query as type string.

2. Prepare a context file with variable query: query=select name from employee

3. Excecute the query: toraclecomponent use context.query

Query is throwing some error when you have where conditions with string type. Need to investigate more on that. Otherwise it works.