Talend并行化和Java范围

Question

I am creating a job to create a complex multi level document for mongoDB from relational data.

I read 'product' records in from Oracle.

I have a tJavaRow and I use the mongoDB API to create a product document (BasicDBObject) using the product details coming in. I store this document in the global map (call this 'product_doc')..as I need to embed a sub-document in this later in the sub job.

I use a tFlowToIterate to store the product_id in the globalMap.

I then have another Oracle input which uses the product_id from the global map as a parameter in the sql, so getting the many part of the relationship to products (call this 'product_orders').

I build a java List of 'product_order' documents and write the List to the globalMap, let's call this 'product_orders'.

I then insert the 'product_documents' List as a sub document to the 'product' document in a tJava component. And I write 'product' to mongoDB and then I move on to the next product row from Oracle.

It is more complex than this, creating a 5 level hierarchy...but this s the basic idea - but it takes 3 hours to run.

So,I want to set the job to run parallelized, so each product row from Oracle gets despatched onto a new thread...Round Robin style.

However, I have a heavy dependency on the globalMap to store objects for later use in the flow....and I know the threads will trample all over each other. I assume each thread maintains the same variable scope across the sub job...

I can identify the thread_id using a global variable in the globalMap "tCollector_1_THREAD_ID" I think.

So I had considered doing this when I add documents/objects into the globalMap.

globalMap.put("product_doc_" + globalMap.get(" tCollector_1_THREAD_ID"))

So that everything I put in the globalMap is thread specific and tagged...but I don't know how tCollector_1_THREAD_ID gets populated, if it is in the globalMap then surely each thread can trample over this value also?

It didn't work...I was getting a load of Null Errors.

So I guess my question is about variable scope and use of globalMap when using tJavaRow components in a parallelized data flow, when you need to maintain references in each thread.

---- UPDATE ------

For clarity if you look at this page it states you can get the thread ID from the variable tCollector_1_Thread_ID. BUt it gets that variable from the globalMap.

Surely the globalMap is a global variable so how can the multiple threads not be all changing this global variable all the time and interfering with each other?

https://help.talend.com//pages/viewpage.action?pageId=265114338

Answer 1

Here's a few approaches that I am using successfully for parallel executions:

If possible create another job that you run parallel, that helps understanding the tasks:

In this example I use the "Use or Register a shared Db connection" feature, so I re-use my connections. tJavaFlex just contains a simple try{ } catch block, so I handle / hide the errors. GpOutput uses a connection that was created outside of the threads.

I more prefer this approach where I create a separate job and use the context parameters to pass information to the job.

Here you can see how the globalMap is used.

I found that the tPartition/Departitioner in talend is very hard to use. I prefer to use more controlled ways to handle parallel executions. Such as a Loop that splits the workload on 20 parallel threads.

  " WHERE mod(num,20) = " + context.i "