Spring Batch ItemWriter错误与PostgreSQL函数

Question

I am attempting to setup a Spring Batch ItemWriter to call a function in PostgreSQL to insert the provided objects. I initially tried using a JdbcBatchItemWriter and specifying the SQL in the configuration, but when that failed, I switched to a custom ItemWriter class to attempt to debug further, but I get the same error. The following are the components of my setup for my ItemWriter (I'm going to ask that folks assume that the ItemReader works properly, since I have no problems with it, so I will not be providing the details of that implementation):

-Source database table and data:

CREATE TABLE test_user
(
    test_user_sys_id numeric NOT NULL,
    ssn character varying(9) NOT NULL,
    create_user_id character varying(30) NOT NULL,
    create_ts timestamp(6) without time zone NOT NULL DEFAULT now()
)
WITH (
OIDS=FALSE
)
DISTRIBUTED BY (ssn);

INSERT INTO test_user (test_user_sys_id, ssn, create_user_id)  VALUES (1,'111111111','DataAdmin');
INSERT INTO test_user (test_user_sys_id, ssn, create_user_id)  VALUES (2,'222222222','DataAdmin');
INSERT INTO test_user (test_user_sys_id, ssn, create_user_id)  VALUES (3,'333333333','DataAdmin');

-Target database table:

CREATE TABLE test_user_result
(
  test_user_result_sys_id numeric NOT NULL,
  ssn character varying(9) NOT NULL,
  job_id numeric NOT NULL
)
WITH (
  OIDS=FALSE
)
DISTRIBUTED BY (ssn);

-Database function to save the results:

CREATE OR REPLACE FUNCTION test_user_result_insert_func(
p_id NUMERIC,
p_ssn CHARACTER VARYING(9),
p_job_id NUMERIC
)
  RETURNS VOID AS
$BODY$

BEGIN
INSERT INTO test_user_result (test_user_result_sys_id,ssn,job_id)
VALUES (p_id,p_ssn,p_job_id);
END;
$BODY$
  LANGUAGE plpgsql VOLATILE;

-localLaunchContext.xml (envirnoment-specific info):

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    xsi:schemaLocation="
               http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">

    <!-- Import the Spring Batch config files -->
    <import resource="springBatchConfig.xml" />

    <!-- Define the PostgreSQL source -->
    <bean id="postgresql_dataSource"
        class="org.apache.commons.dbcp.BasicDataSource">
        <property name="driverClassName" value="org.postgresql.Driver"/>
        <property name="url" value="jdbc:postgresql://host:port/schema"/>
        <property name="username" value="user"/>
        <property name="password" value="password"/>
        <property name="defaultAutoCommit" value="false"/>
    </bean>

    <!-- Define a resourceless transaction manager for the in-memory job repository -->
    <bean id="repositoryTransactionManager"
    class="org.springframework.batch.support.transaction.ResourcelessTransactionManager" />

    <!-- Define a transaction manager for the data source -->
    <bean id="transactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
        <property name="dataSource" ref="postgresql_dataSource"/> 
    </bean>

    <!--  Define the writer JDBC template -->
      <bean id="outputJdbcTemplate" class="org.springframework.jdbc.core.JdbcTemplate">    
          <property name="dataSource" ref="postgresql_dataSource"/> 
       </bean>

    <!-- Define in-memory job repository  -->
    <bean id="jobRepository" class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean">
        <property name="transactionManager" ref="repositoryTransactionManager"/>
    </bean>

    <!-- Define the synchronous job launcher -->
    <bean id="syncJobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
        <property name="jobRepository" ref="jobRepository" />
    </bean>

</beans>

-springBatchConfig.xml (Job info):

<?xml version="1.0" encoding="UTF-8"?> 
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:batch="http://www.springframework.org/schema/batch" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:util="http://www.springframework.org/schema/util"
    xsi:schemaLocation="http://www.springframework.org/schema/beans 
                    http://www.springframework.org/schema/beans/spring-beans-3.0.xsd 
                    http://www.springframework.org/schema/batch 
                    http://www.springframework.org/schema/batch/spring-batch-2.1.xsd
                    http://www.springframework.org/schema/util 
                    http://www.springframework.org/schema/util/spring-util-3.0.xsd">

    <!-- ================================================== -->
    <!-- Components for TestUser Job           -->
    <!-- ================================================== -->

    <!-- Test User Stored Procedure ItemReader -->
    <bean id="testUserItemReader"
        class="org.springframework.batch.item.database.StoredProcedureItemReader"
        scope="step">
        <property name="dataSource" ref="postgresql_dataSource" />
        <property name="procedureName" value="get_user_func_no_arg" />
        <property name="parameters">
            <list>
                <bean class="org.springframework.jdbc.core.SqlParameter">
                    <constructor-arg index="0" value="p_id_min" />
                    <constructor-arg index="1">
                        <util:constant static-field="java.sql.Types.NUMERIC" />
                    </constructor-arg>
                </bean>
                <bean class="org.springframework.jdbc.core.SqlOutParameter">
                    <constructor-arg index="0" value="resultCursor" />
                    <constructor-arg index="1">
                        <util:constant static-field="java.sql.Types.OTHER" />
                    </constructor-arg>
                </bean>
            </list>
        </property>
        <property name="refCursorPosition" value="2" />
        <property name="rowMapper">
            <bean class="dao.mapper.TestUserRowMapper" />
        </property>
        <property name="preparedStatementSetter" ref="preparedStatementSetter" />
    </bean>

    <!-- ItemProcessor is not needed, since the stored procedure provides the results -->

    <!-- TestUser ItemWriter -->
      <bean id="testUserItemWriter" class="dao.writer.TestUserDbItemWriter" scope="step">
        <property name="jdbcTemplate" ref="outputJdbcTemplate"/>
        <property name="jobId" value="#{stepExecution.jobExecution.id}"/>
    </bean>

    <!-- TestUser Job definition -->
    <batch:job id="TestUserJob" incrementer="jobParametersIncrementer">
        <batch:step id="TestUser_step1">
            <batch:tasklet>
                <batch:chunk reader="testUserItemReader" reader-transactional-queue="true"
                    writer="testUserItemWriter" commit-interval="1" />
            </batch:tasklet>
        </batch:step>
    </batch:job>

    <!-- ================================================== -->
    <!-- Common Beans that are used in multiple scenarios -->
    <!-- ================================================== -->

    <!-- Increments the Job ID -->
    <bean id="jobParametersIncrementer" class="org.springframework.batch.core.launch.support.RunIdIncrementer" />

    <!-- Prepared statement setter to provide Tax Year for the stored procedures. -->
    <bean id="preparedStatementSetter" class="dao.setter.TestUserPreparedStatementSetter"
        scope="step">
        <property name="minId">
            <value>#{jobParameters['minId']}</value>
        </property>
    </bean>
</beans>

-TestUser.java (domain data object):

package domain;

// Data object to support a user
public class TestUser {

    private int id;
    private String ssn;
    public int getId() {
        return id;
    }
    public void setId(int id) {
        this.id = id;
    }
    public String getSsn() {
        return ssn;
    }
    public void setSsn(String ssn) {
        this.ssn = ssn;
    }

    @Override
    public String toString() {
        return "TestUser [id=" + id + ", ssn=" + ssn + "]";
    }
}

-TestUserDbItemWriter.java (custom ItemWriter implementation):

package dao.writer;

import java.util.List;
import java.sql.PreparedStatement;
import java.sql.SQLException;

import org.springframework.batch.item.ItemWriter;   
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.core.BatchPreparedStatementSetter;

import domain.TestUser;

public class TestUserDbItemWriter  implements ItemWriter<TestUser> 
{
    private JdbcTemplate jdbcTemplate;
    private int jobId;

    @Override
    public void write(final List<? extends TestUser> chunk) throws Exception {

        String sql = "select test_user_result_insert_func(?, ?, ?);";
        try
        {
            getJdbcTemplate().setSkipResultsProcessing(true);
            getJdbcTemplate().setSkipUndeclaredResults(true);
            getJdbcTemplate().batchUpdate(sql, 
                new BatchPreparedStatementSetter() { 
                    @Override
                    public void setValues(PreparedStatement ps, int i) throws SQLException {
                        TestUser testUser = chunk.get(i);
                        ps.setInt(1, testUser.getId());
                        ps.setString(2, testUser.getSsn());
                        ps.setInt(3, getJobId());
                    }
                    @Override
                    public int getBatchSize() {
                        return chunk.size();
                    }
                });
        }
        catch(org.springframework.dao.DataIntegrityViolationException  ex)
        {
            System.out.println("data integrity ex="+ex.getMessage());
            Throwable innerex = ex.getMostSpecificCause();
            if(innerex instanceof java.sql.BatchUpdateException)
            {
                java.sql.BatchUpdateException batchex = (java.sql.BatchUpdateException) innerex ;
                SQLException current = batchex;
                int count=1;
                   do {

                       System.out.println("inner ex " + count + " =" + current.getMessage());
                       count++;

                   } while ((current = current.getNextException()) != null);
            }

            throw ex;
        }
        catch(Exception  ex)
        {
            System.out.println("ex="+ex.getMessage());
            throw ex;
        }
    }

    public JdbcTemplate getJdbcTemplate() {
        return jdbcTemplate;
    }

    public void setJdbcTemplate(JdbcTemplate jdbcTemplate) {
        this.jdbcTemplate = jdbcTemplate;
    }

    public int getJobId() {
        return jobId;
    }

    public void setJobId(int jobId) {
        this.jobId = jobId;
    }
}

-Invocation command:

java -classpath ".;lib\*;bin" org.springframework.batch.core.launch.support.CommandLineJobRunner localLaunchContext.xml TestUserJob minId=1

When I run this command, I get the following results from the println statements in my custom ItemWriter:

data integrity ex=PreparedStatementCallback; SQL [select test_user_result_insert_func(?, ?, ?);]; Batch entry 0 select test_user_result_insert_func(3, '333333333', 0) was aborted.  Call getNextException to see the cause.; nested exception is java.sql.BatchUpdateException: Batch entry 0 select test_user_result_insert_func(3, '333333333', 0) was aborted.  Call getNextException to see the cause.
inner ex 1 =Batch entry 0 select test_user_result_insert_func(3, '333333333', 0) was aborted.  Call getNextException to see the cause.
inner ex 2 =A result was returned when none was expected.
data integrity ex=PreparedStatementCallback; SQL [select test_user_result_insert_func(?, ?, ?);]; Batch entry 0 select test_user_result_insert_func(2, '222222222', 0) was aborted.  Call getNextException to see the cause.; nested exception is java.sql.BatchUpdateException: Batch entry 0 select test_user_result_insert_func(2, '222222222', 0) was aborted.  Call getNextException to see the cause.
inner ex 1 =Batch entry 0 select test_user_result_insert_func(2, '222222222', 0) was aborted.  Call getNextException to see the cause.
inner ex 2 =A result was returned when none was expected.

This is occuring using JDK 1.7, PostgreSQL JDBC driver JAR postgresql-9.3-1102.jdbc4, and PostgreSQL 8.2.15 (under Greenplum 4.2.8.1 build 2)

I have done a lot investigation and searching online, and I've found the following so far:

• Indication that this is due to something in the batchUpdate() chain not liking that 'select' is being used, since it isn't expecting a result. PostgreSQL doesn't have both stored procedures and functions (as Oracle and other databases do), so a function is the only option of which I'm aware
• As the current TestUserDbItemWriter.write() method stands, I've experimented with settings on the JdbcTemplate to skip results and undeclared results, but I see no change in the behavior.
• I've found suggestions to change the SQL string used by the ItemWriter to use 'call' instead of 'select'. Whenever I attempt that, I get an SQL syntax error.

Although this is a simple example where a function might be considered overkill versus just using the SQL to do the insert, my real application is going to have more complicated data to save across multiple tables, and I would like to be able to use a function to abstract those details from the Java code. Any suggestions would be appreciated.

Thanks in advance

Answer 1

Instead of calling getJdbcTemplate().batchUpdate, please try with getJdbcTemplate().execute(sql);

This works for me at least in a similar requirement.