[英]How to parse CSVs from s3 bucket to use in a javascript AWS Lambda function
I'm trying to read in a CSV from an s3 bucket using the csvtojson library in AWS Lambda, but it's not working properly.我正在尝试使用 AWS Lambda 中的 csvtojson 库从 s3 存储桶中读取 CSV,但它无法正常工作。 Locally, my code works.在本地,我的代码有效。 But when I upload it to Lambda, it doesn't return anything.但是当我把它上传到 Lambda 时,它没有返回任何东西。 There are no errors in the Lambda console, so I'm having a hard time debugging. Lambda console 没有错误,所以我调试很费力。 My code is below.我的代码如下。
const AWS = require('aws-sdk');
const csvtojson = require('csvtojson');
const s3 = new AWS.S3();
const params = {
Bucket: bucketName,
Key: pathToFile
};
const stream = s3.getObject(params).createReadStream();
csvtojson()
.fromStream(stream)
.then((json) => {
console.log('Locally, this returns the CSV as JSON. On Lambda, it does not.');
});
Does csvtojson not work on Lambda for some reason? csvtojson 是否出于某种原因在 Lambda 上不起作用? Should I be using a different method to parse the CSV?我应该使用不同的方法来解析 CSV 吗? Thanks!谢谢!
Your lambda is finishing before the Promise has completed.您的 lambda 在 Promise 完成之前完成。 Replace the last section with this:将最后一节替换为:
const json = await csvtojson().fromStream(stream);
console.log('Locally, this returns the CSV as JSON. On Lambda, it does not.');
Use promise to solve this problem.使用 promise 来解决这个问题。
import * as AWS from 'aws-sdk';
const s3 = new AWS.S3();
const csv = require('@fast-csv/parse');
const params = {
Bucket: 'bucket name',
Key: 'uploads/img-links.csv'
};
const csvFile = s3.getObject(params).createReadStream();
let csvParsePromise = new Promise((resolve, reject) => {
const parser = csv.parseStream(csvFile, { headers: true }).on("data", function (data) {
parser.pause(); // can pause reading using this at a particular row
console.log('One line from .csv >> ', data);
parser.resume(); // to continue reading
}).on("end", function () {
resolve('csv parse process finished')
}).on("error", function () {
reject('csv parse process failed')
});
});
try { await csvParsePromise; }
catch(err) {
console.log('an error has occurred');
}```
Get CSV content is simple获取CSV 内容很简单
Try this尝试这个
export async function getS3Object(
bucket: string,
key: string,
s3: Pick<S3, "getObject">
): Promise<string> {
const { Body } = await s3
.getObject({
Bucket: bucket,
Key: key,
})
.promise();
if (!Body) {
throw new Error(`S3: Empty body for ${bucket} - ${key}`);
}
return Body.toString("utf-8");
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.