Trying to get data from large dataset

Question

I am developing an application for one of our clients using Java 1.7 + GAE, and using Google cloud sql as well as database.

I am trying to get data from a huge resultset, but the only thing i get its the dreaded timeout.

So far, the code i have is the following :

public List <String> leerMasivo(String cadena, String [] valores) throws SQLException{
    log.info("Entra en la función leer(String cadena, String [] valores)");
    this.conectar();
    long nRowsNumber = 1;
    long nId = 0;
    List <String> listaDatos = new ArrayList <String> ();
    try{
        while (nRowsNumber > 0) {
            nRowsNumber = 0;    
    PreparedStatement stmt = con.prepareStatement(cadena, ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY);
    stmt.setMaxRows(10);

    System.out.println("Número de filas maximo "+stmt.getMaxRows());        
    if(valores != null){
        for(int i=0; i< valores.length;i++){
            stmt.setString(i+1,valores[i]);
        }
    }
    ResultSet rs = stmt.executeQuery();
    ResultSetMetaData rsmd = (ResultSetMetaData) rs.getMetaData();
    int numeroColumnas= rsmd.getColumnCount();

    while (rs.next()){
        ++nRowsNumber;
        nId = rs.getLong(1);
        if (numeroColumnas >1){
            String fila ="";
            for (int i=1;i<=numeroColumnas;i++){
                    fila=fila + "::" + rs.getString(i);
            }
            fila=fila+"%%";
            listaDatos.add(fila);
        }else{
            listaDatos.add(rs.getString(1));
            }
        }
    }
    log.info("Los valores leidos son: " + listaDatos.toString());
    }catch(Exception e){
        log.info("Ha ocurrido un error" );
        log.info("el error es " + e.getMessage().toString());
        con.close();
        e.printStackTrace();
    }finally{
        this.cerrarConexion();
    }
     log.info("Sale de la función leer(String cadena, String [] valores)");

    return listaDatos;

}

I require, if you have the time, to tell me whats left so i can get the data i need without getting the timeout error. Right now i am clueless on what i could be missing.

Thank you for your time,

Kind regards,

Answer 1

You have several options here:

• consider using MapReduce library to operate on large datasets
• run this task on module with manual or basic scaling so there will be no time constraints