How to download files in /tmp folder of Google Cloud Function and then upload it in Google Cloud Storage

Question

So I need to deploy a Google Cloud Function that allow me to make 2 things.

The first one is to DOWNLOAD any files on SFTP/FTP server on /tmp local directory of the Cloud Function. Then, the second step, is to UPLOAD this file in a bucket on the Google Cloud Storage.

Actually I know how to upload but I don't get how to DOWNLOAD files from ftp server to my local /tmp directory.

So, actually I have written a GCF that receive in parameters (on the body), the configuration (config) to allow me to connect on the FTP server, the filename and the path.

For my test I used the following ftp server test: https://www.sftp.net/public-online-sftp-servers with this configuration.

{
    config:
    {
        hostname: 'test.rebex.net',
        username: 'demo',
        port: 22,
        password: 'password'
    },
    filename: 'FtpDownloader.png',
    path: '/pub/example'
}

After my DOWNLOAD, I start my UPLOAD. For that I check if I found the DOWNLOAD file in '/tmp/filename' before to UPLOAD but the file is nerver here.

See the following code:

exports.transferSFTP = (req, res) =>
{
    let body = req.body;
    if(body.config)
    {
        if(body.filename)
        {
            //DOWNLOAD
            const Client = require('ssh2-sftp-client');
            const fs = require('fs');

            const client = new Client();

            let remotePath
            if(body.path)
                remotePath = body.path + "/" + body.filename;
            else
                remotePath = "/" + body.filename;

            let dst = fs.createWriteStream('/tmp/' + body.filename);

            client.connect(body.config)
            .then(() => {
                console.log("Client is connected !");
                return client.get(remotePath, dst);
            })
            .catch(err => 
                {
                    res.status(500);
                    res.send(err.message);   
                })
           .finally(() => client.end());


           //UPLOAD
            const {Storage} = require('@google-cloud/storage');

            const storage = new Storage({projectId: 'my-project-id'});

            const bucket = storage.bucket('my-bucket-name');

            const file = bucket.file(body.filename);

            fs.stat('/tmp/' + body.filename,(err, stats) =>
            {
                if(stats.isDirectory())
                {
                    fs.createReadStream('/tmp/' + body.filename)
                        .pipe(file.createWriteStream())
                        .on('error', (err) => console.error(err))
                        .on('finish', () => console.log('The file upload is completed !!!'));

                    console.log("File exist in tmp directory");
                    res.status(200).send('Successfully executed !!!')
                }
                else
                {
                    console.log("File is not on the tmp Google directory");
                    res.status(500).send('File is not loaded in tmp Google directory')
                }
            });
        }
        else res.status(500).send('Error: no filename on the body (filename)');
    }
    else res.status(500).send('Error: no configuration elements on the body (config)');
}

So, I received the following message: "File is not loaded in tmp Google directory" because after fs.stat() method, stats.isDirectory() is false. Before I use the fs.stats() method to check if the file is here, I have just writen files with the same filenames but without content. So, I conclude that my upload work but without DONWLOAD files is really hard to copy it in the Google Cloud Storage.

Thanks for your time and I hope I will find a solution.

Answer 1

The problem is that your not waiting for the download to be completed before your code which performs the upload starts running. While you do have a catch() statement, that is not sufficient.

Think of the first part (the download) as a separate block of code. You have told Javascript to go off an do that block asynchronously. As soon as your script has done that, it immediately goes on to do the the rest of your script. It does not wait for the 'block' to complete. As a result, your code to do the upload is running before the download has been completed.

There are two things you can do. The first would be to move all the code which does the uploading into a 'then' block following the get() call (BTW, you could simplify things by using fastGet()). eg

client.connect(body.config)
 .then(() => {
   console.log("Client is connected !");
   return client.fastGet(remotePath, localPath);
 })
 .then(() => {
    // do the upload
  }) 
  .catch(err => {
     res.status(500);
     res.send(err.message);   
  })
 .finally(() => client.end());

The other alternative would be to use async/await, which will make your code look a little more 'synchronous'. Something along the lines of (untested)

async function doTransfer(remotePath, localPath) {
  try {
    let client - new Client();
    await client.connect(config);
    await client.fastGet(remotePath, localPath);
    await client.end();
    uploadFile(localPath);
  } catch(err) {
    ....
   }
}

Answer 2

here is a github project that answers a similar issue to yours.

here they deploy a Cloud Function to download the file from the FTP and upload them directly to the bucket, skipping the step of having the temporal file.

The code works, the deployment way in this github is not updated so I'll put the deploy steps as I suggest and i verified they work:

1. Activate Cloud Shell and run:

2. Clone the repository from github: git clone https://github.com/RealKinetic/ftp-bucket.git

3. Change to the directory: cd ftp-bucket

4. Adapt your code as needed

5. Create a GCS bucket, if you dont have one already you can create one by gsutil mb -p [PROJECT_ID] gs://[BUCKET_NAME]

6. Deploy: gcloud functions deploy importFTP --stage-bucket [BUCKET_NAME] --trigger-http --runtime nodejs8

In my personal experience this is more efficient than having it in two functions unless you need to do some file editing within the same cloud function