MongoDB Find Query Not Returning All Values

Question

I have a mongoDB collection of sensor values where I'm trying to find all the documents in the collection that are within a certain date range. I've read the documents and have actually implemented the code to find all of the data. However, I'm then writing all of the documents to a CSV file so that I can download them from the web. The issue I'm running into is that the query seems to be looking for all of the data within the correct date range (I've printed the dates on both the server and client sides and both match up as to the correct date range) however, the CSV file only returns about half of the data (it's never consistent). For example, let's say I'm querying my collection to return all the documents stored between:

(Thu May 09 2013 07:57:06 GMT-0400) and (Tue May 14 2013 00:40:43 GMT-0400)

When I click the download data button, it sends a request to the server to find all the documents within this range and then stream the documents to a text file and serves it back to the client. I print to the console the dates on both the server and client, and everything matches... so I'm pretty sure the mongodb find function is using the correct dates in its query.

However, when I open up the CSV file for this query, it gives me a list of documents spanning from (Thu May 09 2013 07:57:15 -> the first date entry gte to the start date)... but only returns values to (Sat May 11 2013 01:04:09). This is about three days worth of data that seems to be missing from the query. Since my sensors are uploading data to the collection every 8 seconds, this is thousands of entries. Is there some sort of data limit on the amount of documents that can be retrieved using the find function? The CSV file has almost 6,850 entries (between Thu to Sat)... but why wouldn't it be returning the full list of values between Thu to Tue? I've checked the database and know it has stored data consistently during this time range... so I don't think that would be the issue. Does anyone have any ideas?

Here is the code for my find function:

collection.find({"addr": Number(json.addr), 
                 "Time": {$gte:new Date(json.startDate), 
                 $lte:new Date(json.endDate)}},
                 proj,
                 function (err, docs) 
                 {
                   if( err || !docs) console.log("No data found");
                   else 
                   {
                     var tempname=json.type+".csv";
                     var stream=fs.createWriteStream("pages/"+tempname);
                     stream.on('error', function(e){console.error(e);});

                     docs.each(function(err2, doc) 
                     {
                       if (!doc) 
                       {
                         console.log("done");
                         stream.end();
                         res.writeHead(200, {"Content-Type": "application/json"});
                         res.write(JSON.stringify({fname:tempname}));
                         res.end();
                         return;
                       }
                       stream.write(doc.Time+","+doc[json.type]+"\n");
                     });
                   }
                });

Answer 1

stream.write() returns false when the buffer is full, so you need to check for that.

Also, see [Writing large files with Node.js] for more info.