将海量数据写入excel文件的Java最佳方法

Question

我在2个PostgreSQL数据库中有大量数据，我想将其写入excel文件，我的代码工作得很好，但是问题是需要很长时间。 另外，当我选择数据周期超过3个月时，服务器中将出现“ Java堆空间”错误，因为它多于200万行。 我需要一种更有效的方式将数据写入excel文件，而且timestamp列需要在写入文件之前将其更改为DateTime列。 我将文件制作为服务器，然后将文件路径返回到客户端以进行下载。

请编写示例代码来解决我的问题。

public String exportRmsValues(String path, String query, String query2) {
    System.out.println(query);
    System.out.println(query2);
    dateFormat = new SimpleDateFormat("yyyy/MM/dd HH:mm:ss");
    Calendar cal = Calendar.getInstance();
    System.out.println("start exportRmsValues \n" + dateFormat.format(cal.getTime()));
    Connection c = null;
    Statement st = null;
    Connection c2 = null;
    Statement st2 = null;

    XSSFWorkbook wb = new XSSFWorkbook();
    XSSFSheet sheet = wb.createSheet("RMS Data Sheet 1");
    XSSFSheet sheet2 = wb.createSheet("RMS Data Sheet 2");
    XSSFSheet sheet3 = wb.createSheet("RMS Data Sheet 3");
    //XSSFCellStyle my_style = wb.createCellStyle();
    //XSSFFont my_font = wb.createFont();
    //my_font.setBoldweight(XSSFFont.BOLDWEIGHT_BOLD);
    //my_style.setFont(my_font);

    Row rowhead = sheet.createRow(0);       
    rowhead.createCell(0).setCellValue("Va");
    rowhead.createCell(1).setCellValue("Vb");
    rowhead.createCell(2).setCellValue("Vc");
    rowhead.createCell(3).setCellValue("Ia");
    rowhead.createCell(4).setCellValue("Ib");
    rowhead.createCell(5).setCellValue("Ic");
    rowhead.createCell(6).setCellValue("datatime");

    Row rowhead2 = sheet2.createRow(0);     
    rowhead2.createCell(0).setCellValue("Va");
    rowhead2.createCell(1).setCellValue("Vb");
    rowhead2.createCell(2).setCellValue("Vc");
    rowhead2.createCell(3).setCellValue("Ia");
    rowhead2.createCell(4).setCellValue("Ib");
    rowhead2.createCell(5).setCellValue("Ic");
    rowhead2.createCell(6).setCellValue("datatime");

    Row rowhead3 = sheet3.createRow(0);     
    rowhead3.createCell(0).setCellValue("Va");
    rowhead3.createCell(1).setCellValue("Vb");
    rowhead3.createCell(2).setCellValue("Vc");
    rowhead3.createCell(3).setCellValue("Ia");
    rowhead3.createCell(4).setCellValue("Ib");
    rowhead3.createCell(5).setCellValue("Ic");
    rowhead3.createCell(6).setCellValue("datatime");

    try {
        Class.forName("org.postgresql.Driver");
        String conString = "jdbc:postgresql://" + host + ":" + port + "/" + DBName
                + "?user=" + user + "&pass=" + pass;
        String conString1 = "jdbc:postgresql://" + host + ":" + port2 + "/" + DBName2
                + "?user=" + user + "&pass=" + pass;
        c = DriverManager.getConnection(conString);
        c2 = DriverManager.getConnection(conString1);
        st = c.createStatement();
        st2 = c2.createStatement();
        String file_name = "RMS_"+dateFormat2.format(cal.getTime())+".xlsx";
        //path = "/opt/jetty/files/"+file_name;
        path = path + file_name;
        List<ResultSet> resultSets = new ArrayList<>();
        resultSets.add(st.executeQuery(query));
        resultSets.add(st2.executeQuery(query2));
        ResultSets rs = new ResultSets(resultSets);
        int index = 1; // row index 0 for columns name
        int index2 = 1;
        int index3 = 1;
        int sheetCount = 1;
        Row row = null;
        Calendar calendar = Calendar.getInstance();
        //TimeZone tz = TimeZone.getDefault();
        while (rs.next()) {
            //XSSFRow row = sheet.createRow((long) index);
            if ( sheetCount <= 1000000 ) {
                row = sheet.createRow(index);
                index++;
            }
            if ( sheetCount > 1000000 && sheetCount <= 2000000) {
                row = sheet2.createRow(index2);
                index2++;
            }
            if ( sheetCount > 2000000 && sheetCount <= 3000000) {
                row = sheet3.createRow(index3);
                index3++;
            }

            /*else {
                row = sheet2.createRow(index2);
                index2++;
                if ( sheetCount >= 2000000 ) {
                    row = sheet3.createRow(index3);
                    index3++;
                }
            }*/

            row.createCell(0).setCellValue(rs.getDoubleValues("va"));
            row.createCell(1).setCellValue(rs.getDoubleValues("vb"));
            row.createCell(2).setCellValue(rs.getDoubleValues("vc"));
            row.createCell(3).setCellValue(rs.getDoubleValues("ia"));
            row.createCell(4).setCellValue(rs.getDoubleValues("ib"));
            row.createCell(5).setCellValue(rs.getDoubleValues("ic"));
            long datatime = rs.getLongValues("datatime");
            calendar.setTimeInMillis(datatime * 1000);
            SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
            Date currenTimeZone = (Date) calendar.getTime();
            row.createCell(6).setCellValue(sdf.format(currenTimeZone));
            sheetCount++;
        }

        FileOutputStream fileOut = new FileOutputStream(path);
        wb.write(fileOut);
        fileOut.close();
        System.out.println("Data is saved in excel file.");
        rs.close();
        st.close();
        c.close();
        st2.close();
        c2.close();
    } catch (Exception e) {
        System.out.println(e);
        e.printStackTrace();
    }
    Calendar cal1 = Calendar.getInstance();
    System.out.println("finish exportRmsValues \n" + dateFormat.format(cal1.getTime()));
    return path;
}

Answer 1

我有好的方法而且很快

query = "select name, to_timestamp(datatime) from x ";
CopyManager copyManager = new CopyManager((BaseConnection) c);
File file = new File(path);
FileOutputStream fileOutputStream = new FileOutputStream(file);

//and finally execute the COPY command to the file with this method:

copyManager.copyOut("COPY (" + query + ") TO STDOUT WITH (FORMAT CSV)", fileOutputStream);

这会将查询中的所有数据写入CSV文件。

Answer 2

您可以使用BCP Utility来实现。 命令行查询只是将查询返回的数据复制为所需格式而已。 在计算机上安装BCP实用程序后，运行以下代码片段：

public class ExcelExport {

    public static void main(String[] args) {

        String excelFileName = "excel/myExcelFile.xls";  // Other formats also supported like .csv, .xlsx etc
        String query = "The SQL Query goes here";
        String databaseName = "DatabseName";
        String ServerUrl = "DatabaseServerUrl/IP";
        String userName = "DatabaseUsername";
        String password = "DatabasePassword";
        String bcpCommand = "bcp \"" + query + "\" queryout \"" + excelFileName + "\" -c -d \"" + databaseName
                + "\" -S " + ServerUrl + " -U \"" + userName + "\" -P \"" + password + "\"";
        System.out.println("BCP Command : " + bcpCommand);

        // Executes BCP command using command line
        ProcessBuilder processBuilder = new ProcessBuilder("cmd.exe", "/c", bcpCommand);

        try {

            Process process = processBuilder.start();
            BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream()));
            String line;

            while (true) {
                line = reader.readLine();

                if (line == null) {
                    break;
                }

                System.out.println(line);
            }

            System.out.println("Excel File Created !!!");

        } catch (IOException ioException) {
            ioException.printStackTrace();
            System.out.println("Failed to export data to excel !!!");
        }
    }
}

注意：它不能像Apache POI一样很好地格式化excel文件，但是要导出数百万个数据。 这样可以节省大量时间。 您可以手动进行格式化。 该代码复制查询返回的所有数据。 因此，请记住，您的记录数不应超过excel文件的行限制，即当前版本为1,048,576行，Office 2003之前为65536行。因此，对于200万行，如果需要，可以创建多个文件。