Azure：已准备的语句超出了每个会话20 MB的内存限制

Question

I'm executing lot's of batches, containing prepared insert statements

public static void main(String... args) throws Exception {
    Class.forName("com.microsoft.sqlserver.jdbc.SQLServerDriver");
    BufferedReader csv = new BufferedReader(new InputStreamReader(Main.class.getClassLoader().getResourceAsStream("records.csv")));
    String line;
    createConnectionAndPreparedStatement();
    while ((line = csv.readLine()) != null) {
        tupleNum++;
        count++;
        List<String> row = new ArrayList<String>(Arrays.asList(line.split(";")));

        tupleCache.add(row);
        addBatch(row, ps);
        if (count > BATCH_SIZE) {
            count = 0;
            executeBatch(ps);
            tupleCache.clear();
        }
    }
}

protected static void createConnectionAndPreparedStatement() throws SQLException {
    System.out.println("Opening new connection!");
    con = DriverManager.getConnection(jdbcUrl, jdbcUser, jdbcPassword);
    con.setAutoCommit(true);
    con.setAutoCommit(false);
    ps = con.prepareStatement(insertQuery);

    count = 0;
}


private static void executeBatch(PreparedStatement ps) throws SQLException, IOException, InterruptedException {
    try {
        ps.executeBatch();
    } catch (BatchUpdateException bue) {
        if (bue.getMessage() != null && bue.getMessage().contains("Exceeded the memory limit")) {
            // silently close the old connection to free resources
            try {
                con.close();
            } catch (Exception ex) {}
            createConnectionAndPreparedStatement();
            for (List<String> t : tupleCache) {
                addBatch(t, ps);
            }
            // let's retry once
            ps.executeBatch();
        }
    }
    System.out.println("Batch succeeded! -->" + tupleNum );
    con.commit();
    ps.clearWarnings();
    ps.clearBatch();
    ps.clearParameters();
}

private static void addBatch(List<String> tuple, PreparedStatement ps) throws SQLException {
    int sqlPos = 1;
    int size = tuple.size();
    for (int i = 0; i < size; i++) {
        String field = tuple.get(i);
        //log.error(String.format("Setting value at pos [%s] to value [%s]", i, field));
        if (field != null) {
            ps.setString(sqlPos, field);
            sqlPos++;
        } else {
            ps.setNull(sqlPos, java.sql.Types.VARCHAR);
            sqlPos++;
        }
    }
    ps.addBatch();
}

So in standalone application everything is fine and no exceptions occur after 700k batch insertions. But when I execute actually same code in custom pig StoreFunc after about 6-7k batch insertions I get the following exception:

java.sql.BatchUpdateException: 112007;Exceeded the memory limit of 20 MB per session for prepared statements. Reduce the number or size of the prepared statements.
    at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeBatch(SQLServerPreparedStatement.java:1824)

And only restarting connection helps. Can someone help me with ideas why it's happening and how to fix it?

Answer 1

According to your description & the error information, per my experience, I think the issue was caused by the configuration about memory at the server side of SQL Azure, such as memory limits for connections within the server resource pool.

I tried to follow the clue to search for the specific explaination about connection memory limits, but failed, besides the content below from here .

Connection Memory

SQL Server sets aside three packet buffers for every connection made from a client. Each buffer is sized according to the default network packet size specified by the sp_configure stored procedure. If the default network packet size is less than 8KB, the memory for these packets comes from SQL Server's buffer pool. If it's 8KB or larger, the memory is allocated from SQL Server's MemToLeave region.

And I continued to search for packet size & MemToLeave and view them.

Based on the above information, I guess that "Exceeded the memory limit of 20 MB per session for prepared statements" means all memory used of parallel connections over the max memory buffer pool of SQL Azure instance.

So there are two solutions I suggested which you can try.

1. Recommended to reduce the value of BATCH_SIZE variable to make the server memory cost less than the max size of memory buffer pool.
2. Try to scale up your SQL Azure instance.

Hope it helps.

---

Here are two new suggestions.

1. I'm really not sure that the MS jdbc driver whether supports the current scenario using Apache Pig to do this like a paralleled ETL job. Please try to use jtds jdbc driver instead of the MS one.
2. A better way I think is using more professional tools to do this, such as sqoop or kettle .

Answer 2

I am running into the same issue when I tried to write a pandas dataframe to the Azure SQL data warehouse. I specified the chunksize, assigned the load user with the largest resource class. However, the issue still occurs.

According to the documentation, INSERT VALUE statement by default only uses the smallrc resource class .

The only solution I can think of is to scale up the DWU but it's not an optimal solution as the cost will go very high.