Piping (big) CSV download from Cloudant noSQL database through Node.js in IBM Cloud not working in production environment

Question

I developed an application in Node.js and I need to download raw data from a noSQL database in the cloud in CSV format (Cloudant noSQL - IBM Cloud). Cloudant allows me to download all data from a database through an API . I want the user to be able to download this same file but through my Node.js App. What I did is, pipe the response from the database API to the client's response. This works just fine when I do it locally, but when I upload the application to the IBM Cloud and try to download the same file (40 mb) it is never downloaded (but it does work with small files, ej. 5 mb).

I've tried 3 different approaches, one with request module, and two other with the https module.

1. Request Module

    request({
      url: database.credentials.url + path,
      method: 'GET'
    }).pipe(res);

2. Https Module (1st try)

    res.setHeader('content-Type', 'text/csv');
    res.setHeader('transfer-encoding','chunked');
    res.setHeader('strict-transport-security','max-age=31536000');

    https.get(database.credentials.url + path, (csv_res) => {
      console.log('Download raw data db headers');
      console.log(csv_res.headers);

      csv_res.on('data', (d) => {
        res.write(d);
        process.stdout.write(".");
      });
      csv_res.on('end', () => {
        res.end();
      });
      }).on('error', (e) => {
        console.error(e);
      });

3. Https Module (2nd try)

    var options = {
      hostname:  database.credentials.host,
      port: 443,
      path: path,
      method: 'GET',
      headers: {
        'Authorization': 'Basic ' + new Buffer(database.credentials.username + ':' + database.credentials.password).toString('base64')
     }
    };
    var proxy = https.request(options, function (csv_res) {
      console.log(csv_res.headers)
      res.writeHead(csv_res.statusCode, csv_res.headers)
      csv_res.pipe(res, {
        end: true
      }).on('error', (e) => {
        console.log("ERROR piping to res: " + e)
      })
    });
    req.pipe(proxy, {
      end: true
    }).on('error', (e) => {
      console.log("ERROR piping from req" + e)
    })

When I run the app locally, the file is downloaded but when I do it in the cloud, the file is never downloaded and after a while the browser shows the file with Network error. Why is this happening?

Answer 1

check the monitoring dashboard while you do this. You might be getting rate limited. I'm also posting a utility that has export functionality for cloudant, https://github.com/glynnbird/couchimport .

If you have have further questions or concerns, just drop us a line through IBM Cloud support and we can help you out!