Time efficient program for importing an excel sheet and then storing it into DB in servlet

Question

I have written a code to import .xls and store the data into db. It works fine for small amount of data ie some hundreds of rows of excel. But if the .xls has say 5000 rows, the importing and storing takes about 1.5 - 2 minutes. Here is the piece of code:

sql = new StringBuffer("insert into bookdetails(bookno");
for (int i = 0; i < header.length; i++) sql.append("," + header[i]);

sql.append(") values(?");
for (int i = 0; i < header.length; i++) sql.append(",?");

sql.append(")");
System.out.println(sql);
psmt = con.prepareStatement(sql.toString());

columnCount = s.getColumns();

// Reading Individual Row Content
for (int i = 1; i < rowCount; i++) {
  // Get Individual Row

  auto =
      getAutoIncrement(
          con,
          "bookdetails"); // getAutoIncrement() just increments the value(as the name suggests).
  psmt.setString(1, auto);

  // s is sheet
  if (s.getCell(0, 0).getContents().length() != 0) {
    for (int j = 0; j < columnCount; j++) {
      rowData = s.getCell(j, i).getContents();

      System.out.println(
          "Header====" + header[j] + "==rowData[j]==" + rowData + "==i==" + i + "==j==" + j);
      // let's say our excel has 4 columns[sno, hello, asd, column]
      if (header[j].equalsIgnoreCase("sno")) {
        psmt.setString(2, rowData);
      } else if (header[j].equalsIgnoreCase("hello")) {
        psmt.setString(3, rowData);
      } else if (header[j].equalsIgnoreCase("asd")) {
        psmt.setString(4, rowData);
      } else if (header[j].equalsIgnoreCase("column")) {
        psmt.setString(5, rowData);
      }
    }
    psmt.addBatch();
    psmt.executeBatch();
  }
}

what measures can I take to make this storing faster.
Can I store the .xls sheet all at once temporarily instead of row-by-row(which takes a lot of time). Any suggestions are welcome.

Answer 1

You are doing an executeBatch after each addBatch . Execute the batch at the end. It might be better to do not batches of thousands, so below I do batches of 100 (less memory usage and less heavy data transport by the driver).

int batchSize = 0;

for (int i = 1; i < rowCount; i++) {
  if (batchSize >= 100) {
      batchSize = 0;
      psmt.executeBatch();
      psmt.clearBatch();
  }

  // Get Individual Row

  auto =
      getAutoIncrement(
          con,
          "bookdetails"); // getAutoIncrement() just increments the value(as the name suggests).
  psmt.setString(1, auto);

  // s is sheet
  if (s.getCell(0, 0).getContents().length() != 0) {
    for (int j = 0; j < columnCount; j++) {
      rowData = s.getCell(j, i).getContents();

      System.out.println(
          "Header====" + header[j] + "==rowData[j]==" + rowData + "==i==" + i + "==j==" + j);
      // let's say our excel has 4 columns[sno, hello, asd, column]
      if (header[j].equalsIgnoreCase("sno")) {
        psmt.setString(2, rowData);
      } else if (header[j].equalsIgnoreCase("hello")) {
        psmt.setString(3, rowData);
      } else if (header[j].equalsIgnoreCase("asd")) {
        psmt.setString(4, rowData);
      } else if (header[j].equalsIgnoreCase("column")) {
        psmt.setString(5, rowData);
      }
    }
    psmt.addBatch();
  }
  psmt.executeBatch(); // At the end.
  psmt.close(); // Close
}

Irrelevant, but you should use a StringBuilder io the old, slower StringBuffer .