Putting s3 object data into an array - node

Question

I'm trying to fetch data from an S3 object and put it into an array. I plan to map through this array and display the data on a React front end in grid/list whatever. I'm struggling with nested functions though, so I'd appreciate some help.

const dataFromS3 = async (bucket, file) => {
let lines = [];

const options = {
    Bucket: bucket,
    Key: file
  };

s3.getObject(options, (err, data) => {
if (err) {
  console.log(err);
} else {
  let objectData = data.Body.toString('utf-8');
  lines.push(objectData);
  console.log(lines);
  return lines;
}
  });
};

Formatting is a bit weird but this is my function to get data from s3. I want to take the output of this function in the form of an array and pass it to my '/' route which I'm testing:

app.get('/', async (req, res, next) => {


try {
    let apolloKey = await dataFromS3(s3Bucket, apolloKeywords);
    res.send(apolloKey);
  } catch (err) {
    console.log('Error: ', err);
  }
});

It seems that the return value of lines in the s3.getObject function needs to be returned within the first function so that I can access it in app.get but I can't seem to do it after some attempts. The value in lines turns into an empty array if I return it at the end of datafromS3() and I can't find a way to return it. I've tried using promises also using a method found here - How to get response from S3 getObject in Node.js? but I get a TypeError: Converting Circular Structure to JSON...

Thank you

Answer 1

You need to make your dataFromS3 func like htis. You were not returning anything from that. AWS also provided promise based function.

const dataFromS3 = async (bucket, file) => {
  const lines = [];

  const options = {
    "Bucket": bucket,
    "Key": file
  };

  const data = await s3.getObject(options).promise();
  const objectData = data.Body.toString("utf-8");
  lines.push(objectData); // You might need to conversion here using JSON.parse(objectData);
  console.log(lines);
  return lines;
};