在PostgreSQL中批量存储过程

Question

I need to save a lot of entities into a database. Saving an entity involves adding rows to different tables with keys autogenerated by inserting a row in one table being used for inserting some row into another table. Such a logic made me create and use a stored procedure. Calling this stored procedure for every entity separately (ie via statement.execute(...)) works fine except there'll be billions of entities to save. So I made an attempt to do this in batches. However, in case of batches a batch execution leads to throwing org.postgresql.util.PSQLException with a message 'A result was returned when none was expected.'

My stored procedure is such:

CREATE OR REPLACE FUNCTION insertSentence(warcinfoID varchar, recordID varchar, sentence varchar,
    sent_timestamp bigint, sect_ids smallint[]) RETURNS void AS $$
DECLARE
    warcinfoIdId integer := 0;
    recordIdId integer := 0;
    sentId integer := 0;
    id integer := 0;
BEGIN
    SELECT warcinfo_id_id INTO warcinfoIdId FROM warcinfo_id WHERE warcinfo_id_value = warcinfoID;
    IF NOT FOUND THEN
        INSERT INTO warcinfo_id (warcinfo_id_value) VALUES (warcinfoID)
            RETURNING warcinfo_id_id INTO STRICT warcinfoIdId;
    END IF;
    SELECT record_id_id INTO recordIdId FROM record_id WHERE record_id_value = recordID;
    IF NOT FOUND THEN
        INSERT INTO record_id (record_id_value) VALUES (recordID)
            RETURNING record_id_id INTO STRICT recordIdId;
    END IF;
    LOOP
        SELECT sent_id INTO sentId FROM sentence_text
            WHERE md5(sent_text) = md5(sentence) AND sent_text = sentence;
        EXIT WHEN FOUND;
        BEGIN
            INSERT INTO sentence_text (sent_text) VALUES (sentence) RETURNING sent_id INTO STRICT sentId;
        EXCEPTION WHEN unique_violation THEN
            sentId := 0;
        END;
    END LOOP;
    INSERT INTO sentence_occurrence (warcinfo_id, record_id, sent_id, timestamp, sect_ids)
        VALUES (warcinfoIdId, recordIdId, sentId, TO_TIMESTAMP(sent_timestamp), sect_ids)
        RETURNING entry_id INTO STRICT id;
END;
$$ LANGUAGE plpgsql;

and the Scala code is such:

def partition2DB(iterator: Iterator[(String, String, String, Long, Array[Int])]): Unit = {
  Class.forName(driver)
  val conn = DriverManager.getConnection(connectionString)

  try {
    val statement = conn.createStatement()
    var i = 0
    iterator.foreach(r => {
      i += 1
      statement.addBatch(
        "select insertSentence('%s', '%s', '%s', %d, '{%s}');".format(
          r._1, r._2, r._3.replaceAll("'", "''"), r._4, r._5.mkString(","))
      )
      if (i % 1000 == 0) statement.executeBatch()
    })
    if (i % 1000 != 0) statement.executeBatch()
  } catch {
    case e: SQLException => println("exception caught: " + e.getNextException());
  } finally {
    conn.close
  }
}

Strangely, even though statement.executeBatch() throw an exception, it saves entities before this. So this workaround makes things work:

def partition2DB(iterator: Iterator[(String, String, String, Long, Array[Int])]): Unit = {
  Class.forName(driver)
  val conn = DriverManager.getConnection(connectionString)

  try {
    var statement = conn.createStatement()
    var i = 0
    iterator.foreach(r => {
      i += 1
      statement.addBatch(
        "select insertSentence('%s', '%s', '%s', %d, '{%s}');".format(
          r._1, r._2, r._3.replaceAll("'", "''"), r._4, r._5.mkString(","))
      )
      if (i % 1000 == 0) {
        i = 0
        try {
          statement.executeBatch()
        } catch {
          case e: SQLException => statement = conn.createStatement()
        }
      }
    })
    if (i % 1000 != 0) {
      try {
        statement.executeBatch()
      } catch {
        case e: SQLException => statement = conn.createStatement()
      }
    }
  } catch {
    case e: SQLException => println("exception caught: " + e.getNextException());
  } finally {
    conn.close
  }
}

However, I would want not to rely on an undocumented feature of PostgreSQL I use currently. I see that other people also came across this problem:

• https://www.postgresql.org/message-id/20050313204039.02ef22fb@emach
• http://grokbase.com/t/postgresql/pgsql-jdbc/113g9ygydb/problem-with-executebatch-and-a-result-was-returned-when-none-was-expected

Can someone suggest a solution?

Answer 1

Strangely, even though statement.executeBatch() throw an exception, it saves entities before this.

That's because you didn't wrap the batch in a transaction. The JDBC spec doesn't IIRC make it clear whether batches should be implicitly wrapped in a transaction if one isn't already in progress, or fired as individual statements. Nor whether the implementation should carry on with the batch after an error or not.

To get well defined behaviour (and better performance), wrap the batch in a transaction.

  statement.addBatch( "select insertSentence('%s', '%s', '%s', %d, '{%s}');".format( r._1, r._2, r._3.replaceAll("'", "''"), r._4, r._5.mkString(",")) ) 

No! Step away from the keyboard! Come on, you're not a PHP programmer :p

You know better than to interpolate strings into SQL. Don't do that. Use PreparedStatement . In addition to being safer and more secure, it'll also be faster since PgJDBC only has to send one statement to parse, then re-uses it. PreparedStatement is ideally suited for use in JDBC batches.

Now, stepping back a bit...

Saving an entity involves adding rows to different tables with keys autogenerated by inserting a row in one table being used for inserting some row into another table. Such a logic made me create and use a stored procedure.

That's the simple way to write it, but it won't perform wonderfully. You're doing lots of independent operations on different tables, lots of piecemeal index updates, etc. There's also procedure call overhead, the overhead of each individual query, etc. Each BEGIN ... EXCEPTION ... block in pl/pgsql has a nontrivial overhead too.

You're going to have problems with hundreds of thousands or millions of rows with this approach, let alone billions.

Relational databases think best in sets. If you're really looking a billions of rows, a proc-based approach will not work. You'll want to take batches of the raw input, insert them into a temp table, then use a series of queries over the temp data to insert them into the destination table(s).

You'll need to get familiar with INSERT INTO ... SELECT ... , UPDATE ... FROM ... , data-modifying common-table expressions etc. If you're on PostgreSQL 9.5 you'll benefit from using INSERT ... ON CONFLICT ... for your upsert-like operations.

It'll hurt to think this way for a while, but it's well worth it, you won't believe the performance you get when you work in sets not individual items.

I can't write the whole thing for you - you've shown no raw data, no schema, and no explanation for the details. Which is fine, since that wasn't your question. It'd be too long anyway, and SO isn't a code-it-for-me site.

Answer 2

Okay, I got rid of stored procedures to prevent batch from failing and as a result relying on undocumented behaviour in case of batch failure. Batches are now wrapped in transactions, and Statement were replaced with PreparedStatement (in fact, it didn't lead to a better speed performance in this script). I made use of INSERT INTO ... SELECT ... and INSERT ... ON CONFLICT ... so a lot of logic moved from a stored procedure to SQL commands.

Now it looks like this:

def partition2DB(iterator: Iterator[(String, String, String, Long, Array[Short])]): Unit = {
  val batchSize = 1000
  val nRetries = 10

  def updStatements(item: (String, String, String, Long, Array[Short]), c: Connection, statement1: PreparedStatement,
                    statement2: PreparedStatement, statement3: PreparedStatement, statement4: PreparedStatement) = {
    val sentence = if (item._3.length > 2712) item._3.substring(0, 2712) else item._3
    statement1.setString(1, item._1)
    statement2.setString(1, item._2)
    statement3.setString(1, sentence)
    statement4.setString(1, item._1)
    statement4.setString(2, item._2)
    statement4.setString(3, sentence)
    statement4.setString(4, sentence)
    statement4.setLong(5, item._4)
    statement4.setArray(6, c.createArrayOf("int4", item._5.map(new Integer(_)).asInstanceOf[Array[Object]]))
    statement1.addBatch()
    statement2.addBatch()
    statement3.addBatch()
    statement4.addBatch()
  }
  def executeStatements(statement1: PreparedStatement, statement2: PreparedStatement,
                        statement3: PreparedStatement, statement4: PreparedStatement) = {
    statement1.executeBatch()
    statement2.executeBatch()
    statement3.executeBatch()
    statement4.executeBatch()
  }

  Class.forName(driver)
  var conn: Connection = null

  try {
    conn = DriverManager.getConnection(connectionString)
    conn.setAutoCommit(false)
    val statement1 = conn.prepareStatement("INSERT INTO warcinfo_id (warcinfo_id_value) VALUES (?) ON CONFLICT (warcinfo_id_value) DO NOTHING;")
    val statement2 = conn.prepareStatement("INSERT INTO record_id (record_id_value) VALUES (?) ON CONFLICT (record_id_value) DO NOTHING;")
    val statement3 = conn.prepareStatement("INSERT INTO sentence_text (sent_text) VALUES (?) ON CONFLICT (sent_text) DO NOTHING;")
    val statement4 = conn.prepareStatement(
      """
        |INSERT INTO sentence_occurrence (warcinfo_id, record_id, sent_id, timestamp, sect_ids) VALUES (
        |    (SELECT warcinfo_id_id FROM warcinfo_id WHERE warcinfo_id_value = ?),
        |    (SELECT record_id_id FROM record_id WHERE record_id_value = ?),
        |    (SELECT sent_id FROM sentence_text WHERE md5(sent_text) = md5(?) AND sent_text = ?),
        |    TO_TIMESTAMP(?),
        |    ?
        |)
      """.stripMargin)
    var i = 0
    val batch = ListBuffer[(String, String, String, Long, Array[Short])]()
    conn.setAutoCommit(false)

    def executeBatch() = {
      var attempts = 0
      while (attempts < nRetries) {
        try {
          for (item <- batch) updStatements(item, conn, statement1, statement2, statement3, statement4)
          executeStatements(statement1, statement2, statement3, statement4)
          conn.commit()
          batch.clear()
          attempts += nRetries
        } catch {
          case e: SQLException => {
            attempts += 1
            println("exception caught: " + e.getNextException)
            conn.rollback()
          }
        }
      }
    }

    iterator.foreach(r => {
      i += 1
      batch += r
      if (i % batchSize == 0) {
        executeBatch()
      }
    })
    if (i % batchSize != 0) {
      executeBatch()
    }
  } catch {
    case e: SQLException => println("exception caught: " + e)
  } finally {
    conn.close()
  }
}

This code doesn't seem quite neat for me though...

Data are stream of items corresponding to some sentence with its timestamp and some identifiers. So content of the r variable is something like this: ('4af93233-3515-43da-8b47-71b0dad99ccc', 'd5ea8a14-be65-4281-9a87-24dcbdc3f879', 'the definitive guide being hater the internet', 1362484800, [1])

Each item is stored tables 'sentence_occurrence', and if needed in 'warcinfo_id', 'record_id', 'sentence_text'.

Schema is following:

statement.executeUpdate(
  """
    |CREATE TABLE warcinfo_id (
    |    warcinfo_id_id serial PRIMARY KEY,
    |    warcinfo_id_value char(36) UNIQUE NOT NULL
    |);
  """.stripMargin)
statement.executeUpdate(
  """
    |CREATE TABLE record_id (
    |    record_id_id serial PRIMARY KEY,
    |    record_id_value char(36) UNIQUE NOT NULL
    |);
  """.stripMargin)
statement.executeUpdate(
  """
    |CREATE TABLE sentence_text (
    |    sent_id serial PRIMARY KEY,
    |    sent_text varchar UNIQUE NOT NULL
    |);
  """.stripMargin)
statement.executeUpdate(
  """
    |CREATE TABLE sentence_occurrence (
    |    entry_id serial PRIMARY KEY,
    |    warcinfo_id integer NOT NULL,
    |    record_id integer NOT NULL,
    |    sent_id integer NOT NULL,
    |    timestamp timestamp NOT NULL,
    |    sect_ids smallint ARRAY
    |);
  """.stripMargin)

Added after Craig's comment:

Thanks, Craig. What is operation on sets of input? Could you post a link to some example?

Additionally, I've got the following problem. If two batches attempt to insert the same record in some table simultaneously, I get an java.sql.BatchUpdateException with a message like this "ERROR: deadlock detected. Detail: Process 31959 waits for ShareLock on transaction 24298876; blocked by process 31955. Process 31955 waits for ShareLock on transaction 24298877; blocked by process 31959." What is the right solution for such cases? I can think of retrying the failed attempt until it succeeds or reaches the limit for number of retries, storing duplicated and then producing the final result table with SELECT DISTICT..., playing around with isolation levels (for example trying 'Read uncommitted'). However, all of them seem dangerous workarounds (number of retries reaches the limit, I run out of disk space, I get some wrong data in the database).