在列表中生成大量数据并生成文件

Question

I have a table in the database that has about 2.3 million records.

I need to export this data to a csv file through a java / web application.

I'm using JPA / Hibernate. But I'm having trouble.

HibernateEntityManager hem = this.getEntityManager().unwrap(HibernateEntityManager.class);
    Session session = hem.getSession();
    org.hibernate.Query qu = 
        session.createSQLQuery("select li.* from ligacoes li").
            setFetchSize(Integer.MIN_VALUE).setReadOnly(true);
    ScrollableResults r = qu.scroll(ScrollMode.FORWARD_ONLY);

    int count = 0;
    while (r.next()) {

But when the program reads about 200,000 records it throws an exception:

Exception in thread "ContainerBackgroundProcessor[StandardEngine[Catalina]]"  
java.lang.OutOfMemoryError: Java heap space

I've tried other means, but I'm still having difficulties. I can't export the data directly from the database. The data must be exported through this application.

Can anyone help?

Answer 1

I solved the question.

I split the query. Instead of bringing a very large list, I bring a minor list several times. Pagination.

        HibernateEntityManager hem = this.getEntityManager().unwrap(HibernateEntityManager.class);
        Session session = hem.getSession();

        int pageIn = 1;
        int pageFim = 20000;
        boolean fim = false;
        int count = 0;

        while (!fim) {
            ++count;
            org.hibernate.Query qu = session.createSQLQuery("select li.* from ligacoes li where li.operadora='81' "
                    + "and length(li.numero_destino) < 8 and li.valor_contrato <> '0'");
            qu.setFirstResult(pageIn - 1);
            qu.setMaxResults(pageFim);
            List<Ligacoes> chamadasVivo = new ArrayList<>(objectToLigacoes(qu.list()));

            count += pageFim;

            if (chamadasVivo.size() <= 0) {
                break;
            }

            for (Ligacoes li : chamadasVivo) {
                //append the file...

            }

            if (count % 100000 == 0) {
                session.flush();
                session.clear();
            }

            pageIn += pageFim;
        }