Exporting data from Cloudant/CouchDB to CSV

Question

Having some minor issues exporting data to CSV from Cloudant. Currently using the CSV function found here: https://developer.ibm.com/clouddataservices/2015/09/22/export-cloudant-json-as-csv-rss-or-ical/

The issue is some data was added later with 2-3 more fields. When this downloads the documents it just places the information one right after the other and it can't account for some older data missing fields so data gets misaligned.

I've tried creating functions that try and detect if the field exists, and if it doesn't set it to an empty string.

Here's what I have tried which gives me the error: {"error":"compilation_error","reason":"Expression does not eval to a function.

 // output HTTP headers
 start({
   headers: {  'Content-Type': 'text/csv'  },
 });


 // iterate through the result set
 while(row = getRow()) {

   // get the doc (include_docs=true)
   var doc = row.doc;

   // if this is the first row
   if (first) {

     // output column headers
     send(Object.keys(doc).join(',') + 'n');
     first = false;
   }

   // build up a line of output
   var line = '';

   // iterate through each row
   //for(var i in doc) {

     if (doc.hasOwnProperty('customerNumber')) {
     } else {
       doc.customerNumber = '';
     }

     if (doc.hasOwnProperty('affiliateNumber')) {
     } else {
       doc.affiliateNumber = '';
     }

     if (doc.hasOwnProperty('dunsNumber')) {
     } else {
       doc.dunsNumber = '';
     }

     if (doc.hasOwnProperty('dunsDomestic')) {
     } else {
       doc.dunsDomestic = '';
     }

     if (doc.hasOwnProperty('dunsGlobal')) {
     } else {
       doc.dunsGlobal = '';
     }

     if (doc.hasOwnProperty('countryCode')) {
     } else {
       doc.countryCode = '';
     }

     if (doc.hasOwnProperty('companyName')) {
     } else {
       doc.companyName = '';
     }

     if (doc.hasOwnProperty('countryPreapprovedAmt')) {
     } else {
       doc.countryPreapprovedAmt = '';
     }

     if (doc.hasOwnProperty('preapprovedAmt')) {
     } else {
       doc.preapprovedAmt = '';
     }

     if (doc.hasOwnProperty('currency')) {
     } else {
       doc.currency = '';
     }

     if (doc.hasOwnProperty('expirationDate')) {
     } else {
       doc.expirationDate = '';
     }

     if (doc.hasOwnProperty('blacklist')) {
     } else {
       doc.blacklist = '';
     }

     line += doc.customerNumber + ',' + doc.affiliateNumber + ',' + doc.dunsNumber+ ',' + doc.dunsDomestic+ ',' + doc.dunsGlobal+ ',' + doc.countryCode+ ',' + doc.companyName+ ',' + doc.countryPreapprovedAmt+ ',' + doc.preapprovedAmt+ ',' + doc.currency+ ',' + doc.expirationDate+ ',' + doc.blacklist;
   //}
   line += 'n';

   // send  the line
   send(line);
 }
};

When it comes across a piece of data that doesn't have all of those fields, it will detect it. Assign an empty string so the data aligns when downloading the csv.

Answer 1

If you want to generate CSV files from relatively flat documents from CouchDB/Cloudant, you have two options:

1. Using CouchDB "list" functions, as described in the blog post you linked.
2. Use a utility to export the data as JSON and convert to CSV on the client side

Option 2 can be facilitated by the open-source couchimport utility which comes with a couchexport companion tool:

Data is exported on the command line and can be piped to a file for further analysis.

couchexport --db mydb > mydb.csv

If the documents are not in the correct format a "filter" function can be provided to coerce the data into the correct form.