Building a File Tree from Amazon S3 to be stored in Firebase

Question

I am wanting to be able to retrieve the key/value pairs from Amazon s3 and then store some of the information retrieved in Firebase so I can build a file system of sorts in AngularJS.

This question is only going as far as the storing in firebase.

Now, obviously you need to be able to create an unlimited amount of folders and put an unlimited amount of files within those folders.

The way my code works right now If I want to add another subfolder, I have to add another IF/ELSE statement. I need something more dynamic.

I have a script I made in AngularJS that retrieves the Key/Value pairs from Amazon S3 and then breaks it down and stores them in Firebase based on the Key returned.

I'm going to break the bits of code up so I can explain it however, I created a jsFiddle with the entire block in it: https://jsfiddle.net/Lcr5p6qm/2/

First, I get the key/value pairs from amazon using a service.

storageService.getBucketDirectory({
   'bucket' : bucket,
   'directory' : 'code/'
}).then(function(code){

Second, I remove the previous tree from Firebase and start the loop through each key/value pair from S3.

projectRef.child('code').remove();
angular.forEach(code, function(value, key){

   var item = value.Key.replace('code/', '');

Third, if the pair signifies it's at the base I store it as such. we alter the key a little to create a type and name for the file.

if(!item.includes('/'))
{

    var name = value.Key.replace('code/', '').replace('/', '').replace('+', ' ');

    if(name.indexOf('.') == 0)
    {
       var type = null;
    }
    else
    {
       var type = value.Key.substr(value.Key.indexOf('.') + 1)
    }

    projectRef.child('code').child('/').push({
       'Name'           : name,
       'Type'           : type,
       'Key'           : value.Key,
       'LastModified'  : value.LastModified,
       'Size'           : value.Size
    })
}

Fourth, if the pair isn't at the base and also has subfolders in the key, we alter the key and then store it as such. If it doesn't have subfolders, then we just store it inside the folder.

else {
  var folder = item.substr(0, item.indexOf('/'));

  if ((item.replace(folder + '/', '')).includes('/')) {
    var subfolder = item.replace(folder + '/', '');

    subfolder = subfolder.substr(0, subfolder.indexOf('/'));

    var name = value.Key.replace('code/', '').replace(folder + '/', '').replace(subfolder + '/', '').replace('+', ' ');

    if (name.indexOf('.') == 0) {
        var type = null;
    } else {
        var type = value.Key.substr(value.Key.lastIndexOf('.') + 1)
    }

    projectRef.child('code').child(folder).child(subfolder).push({
        'Name': name,
        'Type': type,
        'Key': value.Key,
        'LastModified': value.LastModified,
        'Size': value.Size
    })
  }
  else {
     var name = value.Key.replace('code/', '').replace(folder + '/', '').replace('+', ' ');

     if (name.indexOf('.') == 0) {
         var type = null;
     } else {
         var type = value.Key.substr(value.Key.indexOf('.') + 1)
     }

     projectRef.child('code').child(folder).push({
        'Name': name,
        'Type': type,
        'Key': value.Key,
        'LastModified': value.LastModified,
        'Size': value.Size
     })

   }
}

Finally, this turns into something like what is below.

• css
  • main.css
  • normalize.css
• doc
  • TOC.md
  • css.md
  • extend.md
  • faq.md
  • html.md
  • js.md
  • misc.md
  • usage.md
• img
  • .gitignore
• js
  • main.js
  • plugins.js
  • vendor
    • jquery-1.12.0.min.js
    • modernizr-2.8.3.min.js

Great, it works, right? Well if you don't want anything more than second level directories and we all know that isn't enough.

My problem is is if we would have another folder inside the /js/vendor/ folder, it would break the system and the name and type would include the subfolder name.

I need a way to create an unlimited amount of sub-folders and have this work.

UPDATE

I came up with some "cleaner" code that has been added to my JSFiddle in the link above. It's under the comment "Second Attempt". This gets me closer to what I want, yet isn't getting the sub folders in.

Bonus: What about empty folders?

Answer 1

Firebase will accept a path-like string as the parameter for .child() , so you only need to concern yourself with manipulating the Key string to get this path and the filename. You can do something like this:

var testKey = 'code/some/long/path/filename.txt';

var lastSlash = testKey.lastIndexOf('/');

var filepath = testKey.substring(0, lastSlash); // 'code/some/long/path'
var filename = testKey.substr(lastSlash+1);  // 'filename.txt'

projectRef.child(filepath).push({
    'Name': filename
});