DownloadDir with node-s3-client failing due to illegal operation on a directory

Question

I am using a node s3 client ( https://github.com/andrewrk/node-s3-client#clientdownloaddirparams ) to sync an entire directory from S3 to a local directory.

As per the documentation, my code is as follows:

    var s3 = require('s3');     
    var client = s3.createClient({
        s3Options: {
            accessKeyId: Config.accessKeyId,
            secretAccessKey: Config.secretAccessKey
        }
    });

    var downloader = client.downloadDir({
        localDir: 'images/assets',
        deleteRemoved: true,
        s3Params: {
            Bucket: Config.bucket,
            Prefix: Config.bucketFolder
        }
    });

    downloader.on('error', function(err) {
        console.error("unable to download: ", err);
    });

    downloader.on('progress', function() {
        console.log("progress", downloader.progressMd5Amount, downloader.progressAmount, downloader.progressTotal);
    });

    downloader.on('end', function(data) {
        console.log("done downloading", data);
    });

This begins syncing and the folder begins downloading, but eventually returns this:

progress 0 0 0
...
progress 1740297 225583 5150000
unable to download:  { Error: EISDIR: illegal operation on a directory, open 'images/assets'
    at Error (native)
  errno: -21,
  code: 'EISDIR',
  syscall: 'open',
  path: 'images/assets' }

The directory does indeed exist. I've tried moving directory location, path, etc, but nothing seems to do the trick. I have researched this error and have found out that it occurs when you try to open a file, but the path given is a directory. Not sure why this s3-client is trying to open a file instead of a directory. Any help or advice would be awesome. Thanks!

Answer 1

Remember that in S3, you can create a directory that has the same name as a file. Based on the error you're getting, I would say that in S3 you have a file named images and a folder named images. This would be illegal on the file system but not in S3.

Answer 2

I just determined that download speeds were causing this issue. Unfortunately, I was on a network with .5 up and down. I just switched over to 25/10 and its working fine.

Answer 3

Use getS3Params function to resolve this :

getS3Params: function getS3Params(localFile, s3Object, callback) {
    if (path.extname(localFile) === '') {  callback(null, null); }
    else { callback(null, {}); }
}

https://github.com/andrewrk/node-s3-client/issues/80