Reading an arbitrary number of csvs with javascript/D3

Question

I am building a webapp that receives a list of filenames, and based on that input reads in the files, concatenates them, and displays the data in various charts. The tricky thing here is that the files are rather large before the relevant information is extracted (~6-10mb), and I don't know exactly how many the user will request. (Lets say that a typical request will be greater than 3 but less than 20.)

Insofar as I can tell, there is no obvious way of reading in an arbitrary number of files with d3. Mike Bostock has answered a similar question here . The problem with these solutions is that they require you to know the number of files ahead of time - I don't have that luxury.

What I would like to do is have some kind of recursive function like:

function concatFiles(fileNameArray) {
    function recursiveConcat(fileNameArray, concatedFile) {
        var temp;
        if (fileNameArray.length === 0) {
            return concatedFile;
        } else {
            d3.csv(fileNameArray.shift(), function(err, csv) {
                temp = csv;
            }
            return recursiveConcat(fileNameArray, concatedFile.concat(temp));
        }
    }
    return recursiveConcat(fileNameArray, []);
}

The nice thing about this kind of solution is that I don't have to create a variable for each file or store intermediate results in an array. However, I'm not sure that this thing will play nice with Javascripts' asynchronous nature (I'm new to Javascript). How can I map between arrayOfFileNames -> concatenatedFile effectively?

Answer 1

d3-queue seems to be perfectly fine for this scenario. The following snippet should be enough to handle the loading of an arbitrary number of files:

function loadFiles(files) {

  // Create a new queue to handle the loading of all files.
  var q = d3.queue();

  // Iterate over the array of all files to be loaded.
  files.forEach(function(f) {
    // Add a loading task for each file to the queue.
    q.defer(d3.csv, f)  
  });

  // When all loading tasks have finished, you may process the
  // contents of all files.
  q.awaitAll(function(error, contents) {
    // Do your concatenation here.
    // contents contains an array of the parsed results of all files
    // console.log(contents);
  })
}

var filesToLoad = ["file1.csv", "whatever.csv", "another.csv"];
loadFiles(filesToLoad);      // load all files in the array

Note, that this does not make any assumptions about the number of files in the array filesToLoad . Have a look at this Plunk for a working example.