Why fetching data from sqlite block Node.JS?

Question

I want to fetch and export to csv file huge amount (5 - 12 milions rows) of archive data from Sqlite database. While doing this the whole server is blocked. No other connection can be handled by server (for example I couldn't open website in another tab in browser).

Node.JS server part:

function exportArchiveData(response, query){                                                                                                    
  response.setHeader('Content-type', 'text/csv');                             
  response.setHeader('Content-disposition', 'attachment; filename=archive.csv');                      
  db.fetchAllArchiveData(                                                     
      query.ID,                                                               
      function(error, data){
          if(!error)                                           
             response.write(data.A + ';' + data.B + ';' + data.C + '\n');           
      },                                                                      
      function(error, retrievedRows){
          response.end();                                                     
      });                                                                     
};            

Sqlite DB module:

 module.exports.SS.prototype.fetchAllArchiveData = function (          
     a, callback, complete) {                                                  

     var self = this;                                                            

 //      self.sensorSqliteDb.all(                                                
         self.sensorSqliteDb.each(                                               
             'SELECT A, B, C '+                            
             'FROM AD WHERE '+                                          
             ' A="' + a + '"'+                                                
             ' ORDER BY C ASC' +                                         
             ';'                                                                 
             ,
             callback,                                                               
             complete                                                            
         );                                                                      
 };        

I also create index on AD like CREATE INDEX IAD ON AD(A, C) and EXPLAIN QUERY PLAN show that this index is used by sqlite engine.

Still, when I call exportArchiveData server send the data properly but no other action can be performed during this. I have a huge amount of data (5 - 12 milions of rows to send) so it takes ~3 minutes.

How can I prevent this from blocking whole server?

I thought that if I use EACH and there will be callback's the server will be more responsive. Also Memory usage is huge (about 3GB and even more). Can I prevent this somehow?

In answer to comments, I would like to add some clarifications:

I use node-sqlite3 from developmentseed . It should be asynchronous and non-blocking. And it is. When statement is prepared I can request main page. But when server start serving data, then Node.js server is blocked. I guess thats because request for home page is one request to call some callback while there are milions request for callback handling archive data "EACH".

If I use sqlite3 tool from linux command line I do not get rows immediately but that is not the problem as long as node-sqlite3 is non-blocking.

Yes. I'm hitting CPU max. What is worse, when I request twice as much data the whole memory is used, and then server freeze forever.

Answer 1

OK. I handle this problem this way.

Instead of using Database#each I use Database#prepare with multiple Statement#get.

What is more, I investigate that running out of memory was caused by full buffer of response. So now, I call for next row when I get previous and when response buffer have place for new data. Working perfect. And Now server is not blocked (only during preparing statement).

Sqlite module:

module.exports.SS.prototype.fetchAllArchiveData = function (                  
 a) {                                                          

 var self = this;                                                                    
 var statement = self.Db.prepare(                                                                                          
         'SELECT A, B, C '+                                    
         'FROM AD WHERE '+                                                  
         ' A="' + a + '"'+                                                        
         ' ORDER BY C ASC' +                                                 
         ';'                                                                         
         ,                                                                           
         function(error){                                                            
             if(error != null){                                                      
                 console.log(error);                                                 
             }                                                                       
         }                                                                                                                                           
     );                                                                              
 return statement;                                                                   
};              

Server side:

function exportArchiveData(response, query){                                    

 var respRet = null;                                                         
 var i = 0;                                                                  
 var statement = db.fetchAllArchiveData(                                     
     query.ID);                                                               
 var getcallback = function(err, row){                                       
     if(err != null){                                                        
         console.mylog(err);                                                 
         return;                                                             
     }                                                                       
     if(typeof(row) != 'undefined'){                                         
         respRet = response.write(row.A + ';' + row.B + ';' + row.C + '\n');
         console.log(i++ + ' ' + respRet);                                   
         if(respRet){                                                        
             statement.get(getcallback);                                     
         }else{                                                              
             console.log('should wait on drain');                            
             response.on('drain', function(){                                
                 console.log('drain - set on drain to null, call statement');
                 response.on('drain', function(){});                         
                 statement.get(getcallback);                                 
             });                                                             
         }                                                                   
     }else{                                                                  
         response.end();                                                     
     }                                                                       
 };                                                                          

 statement.get(function(err, row){                                           
     response.setHeader('Content-type', 'text/csv');                         
     response.setHeader('Content-disposition', 'attachment; filename=archive.csv');                   
     getcallback(err, row);                                                  
 });                                                                         
 };