Processing ArrayList in Mule Batch

Question

I am building an aggregation process for upserting records. I have a scatter-gather subflow where each path queries a Salesforce environment, maps the data into a consistent output List of Maps, and after the scatter-gather passes the message into the Combine Collections transformer. The output is an ArrayList of all of the values I need to the upsert as part of batch processing.

The issue is that when the Arraylist is passed to the Process step Mule throws the following error:

Object "org.mule.transport.NullPayload" not of correct type. It must be of type "{interface java.lang.Iterable,interface java.util.Iterator,interface org.mule.routing.MessageSequence,interface java.util.Collection}" (java.lang.IllegalArgumentException) (com.mulesoft.module.batch.exception.BatchException). Message payload is of type: NullPayload

Using Mule Debugger I can see that the payload at the completion of the Input phase is not null and is of type ArrayList. If I simplify the flow, remove the scatter-gather so that the input phase is simply a query and DataMapper I still receive the same error.

How can you take a list of maps and process them using the Batch processing feature?

Input phase:

                </processor-chain>
                <processor-chain>
                    <sfdc:query config-ref="Salesforce-B" query="dsql:SELECT Amount,CloseDate,CreatedDate,Id,LeadSource,Name,OwnerId,Primary_Source__c,Probability,StageName FROM Opportunity WHERE lastModifiedDate &gt; #[lastUpdate]" doc:name="B" fetchSize="10"/>
                    <logger message="B has #[payload.size().toString()] Opportunities for upsert" level="INFO" doc:name="Logger"/>
                               <data-mapper:transform config-ref="List_Opportunity__To_Map_B" doc:name="Opp to Sales B"/>

                </processor-chain>
            </scatter-gather>
    <combine-collections-transformer doc:name="Combine Collections"/>

Answer 1

I cannot really picture your flow so attaching your xml file would really help me understand what you are trying to achieve here. Hence my answer might not be totally true for your scenario.

The batch processor is able to process a list of maps, the following is an example of this:

   <batch:job name="batchlistofmapsBatch1">
        <batch:input>
            <poll doc:name="Poll">
                <fixed-frequency-scheduler frequency="1" timeUnit="SECONDS"/>
                <set-payload value="#[[['Name':'Andre'],['Name':'Mike']]]" doc:name="Set Payload"/>
            </poll>
        </batch:input>
        <batch:process-records>
            <batch:step name="Batch_Step">
                <logger message="#[message.payload]" level="INFO" doc:name="Logger"/>
            </batch:step>
        </batch:process-records>
    </batch:job>

That said, As far as I know the SFDC query (unless it is a query single) returns a consumeriterator and not an arraylist and it might be that somewhere you are consuming this. I suggest that you generate a list from the iterator, this can be done by using the iteratorUtils ( https://commons.apache.org/proper/commons-collections/javadocs/api-2.1.1/org/apache/commons/collections/IteratorUtils.html )

Answer 2

Andre, thank you for the assistance. I ended up opening a case with Mule Support and discovered that when a Polling endpoint is used to collect the input it is able to successfully pass into the Processing step.

When using a http endpoint and triggering the subflow via the Batch execute reference or when the payload is an ArrayList of ConsumerIterators, the payload is dropped and throws an error as the type becomes NullPayload.

Mule confirmed this is a bug and will be updating.