[英]How to insert bulk data to postgresql db from CSV file?
I have to insert more than 100 records which are present in CSV file to PostgreSQL db. 我必须将CSV文件中存在的100多个记录插入到PostgreSQL数据库中。 So I have tried the below mentioned code, it is reading the data from the file but unable to insert them to PostgreSQL table so is there any other way to perform this? 所以我尝试了下面提到的代码,它正在从文件中读取数据,但是无法将它们插入到PostgreSQL表中,所以还有其他方法可以执行此操作吗? Like csvtojson etc.? 像csvtojson等?
const csv = require('csv');
var csvParser = require('csv-parse');
Controller.uploadCsv = async(data) => {
fs.createReadStream(data.path)
.pipe(csvParser({
delimiter: '\t',
endLine: '\n',
escapeChar: '"',
enclosedChar: '"'
}))
.on('data', function(data) {
console.log(data)// returning in console mentioned below
console.log(data.name) // is undefined
const add = {
name: data.name,
address: data.address,
phoneNo: data.phoneNumber,
email: data.email,
created_at: new Date(),
updated_at: new Date()
};
const result = await models.table.create(add);
})
.on('end', function(data) {
console.log('reading finished')
})
}
router.js router.js
router.post('/file', upload.single('file'),(req, res, next) => {
Controller.uploadCsv(req.file)
.then((result) => res.json(result))
.catch(next)
})
console data 控制台数据
[ 'name',
'address'
'phoneNumber',
'email',
'created_at',
'updated_at']
[ 'aaa',
'delhi',
'1102558888',
'test@gmail.com',
'2017-10-08T06:17:09.922Z',
'2018-10-08T06:17:09.922Z',]
[ 'Oreo',
'bgl',
'1112589633',
'test123@gmail.com',
'2017-10-08T06:17:09.922Z',
'2018-10-08T06:17:09.922Z' ]
TL;DR. TL; DR。 Your code has a minor error that may be causing your problem - it's when you use await, in order to run it you'd need to put async
before the function
on the data
handler - it may work for small files, but please read on it's not the right solution - I added one of the proper ways below . 您的代码有一个小错误,可能会引起您的问题-当您使用await时,要运行它,您需要在data
处理程序中的function
前放置async
它可能适用于小文件,但请继续阅读这不是正确的解决方案-我在下面添加了一种适当的方法 。
ES6 async/await is a language construct that allows you to await
for resolution of a Promise and continue executing the code in an async
function. ES6异步/等待是一种语言构造,可让您await
Promise的解析并继续执行async
功能中的代码。 In your code you do have an async function
declaration, however you added await
in a non-async function. 在您的代码中确实有一个async function
声明,但是您在非异步函数中添加了await
。 To clarify - await
keyword will only be allowed when the closest function() {
is async
- in your case it's not. 需要说明的是-仅在最接近的function() {
是async
时才允许使用await
关键字-在您的情况下则不允许。
I actually don't think your code would even compile and after some changes you'd fall straight to a problem mentioned in this question - this is because you're trying to run an asynchronous operation on a synchronous event handler in node. 其实,我不认为你的代码,甚至会编写一些更改后,你会直接落在中提到的一个问题这个问题 -这是因为你想在节点同步事件处理程序运行的异步操作。 This asynchronous insert to the database will get run, but the end
event will fire before the operations are completed . 该对数据库的异步插入将开始运行,但是end
事件将在操作完成之前触发 。
In order to do this correctly - you could use a transform stream or abandon streaming altogether and simply use an array from CSV (there's more than enough good modules for that). 为了正确执行此操作-您可以完全使用转换流或完全放弃流式传输,而仅使用CSV中的数组(为此有足够多的好模块)。 I am however the author of the scramjet framework and I also think this should simply work as you wrote it, or maybe even simpler. 但是,我是scramjet框架的作者,我也认为这应该像您编写它时那样简单,甚至可能更简单。
Here's a code that will do what you want: 这是将执行您想要的代码:
const {StringStream} = require('scramjet');
Controller.uploadCsv = async(data) =>
fs.createReadStream(data.path)
.pipe(new StringStream('utf-8'))
.CSVParse({
delimiter: '\t',
newline: '\n',
escapeChar: '"',
quoteChar: '"'
})
.map(data => ({
name: data.name,
address: data.address,
phoneNo: data.phoneNumber,
email: data.email,
created_at: new Date(),
updated_at: new Date()
}))
.each(async entry => await models.table.create(entry))
.each(result => log(result)) // if it's worth logging
.run();
Scramjet simply uses streams (all classes extend built-in node.js streams) underneath, but exposes an interface similar to synchronous ones on Array etc. You can run your async operations and it returns a Promise from run
operation. Scramjet只是在下面使用流(所有类都扩展了内置的node.js流),但是在Array等上公开了类似于同步接口的接口。您可以运行异步操作,并从run
操作返回Promise。
Insert the async keyword on the OnData function. 在OnData函数上插入async关键字。 Remember, it's not sequencial execution, so the records may be inserted on a completely diferent order between one program execution and another. 请记住,它不是顺序执行,因此记录可以在一个程序执行与另一个程序执行之间以完全不同的顺序插入。
Replace: 更换:
.on('data', function(data) {
With: 附:
.on('data', async function(data) {
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.