[英]NodeJS fs.unlinkSync not freeing up hard-drive space on aws ec2 server
I have a NodeJS application on Elastic Beanstalk.我在 Elastic Beanstalk 上有一个 NodeJS 应用程序。
I'm downloading a large number of large zips, opening and processing them, then deleting the zip files via fs.unlinkSync
我正在下载大量大 zip,打开并处理它们,然后通过
fs.unlinkSync
删除 zip 文件
When testing I see the files get deleted, but apparently unlink won't actually deallocate the space while a process is using it?测试时我看到文件被删除,但显然 unlink 在进程使用它时实际上不会释放空间?
How can I instruct fs to free up the memory allocation?如何指示 fs 释放 memory 分配?
I am writing the file via我正在通过写文件
let writeStream = fs.createWriteStream(file.name);
writeStream.on('finish', () => {
// a bunch of logic that eventually returns a promise
and inside its resolve/reject并在其解决/拒绝中
fs.unlinkSync(file.name)
I've gone through and those are the only times I'm touching the file, everything else is closed.我已经完成了,这是我唯一一次触摸文件,其他所有内容都已关闭。 Is there some other event I need to wait for for the writeStream to close?
我需要等待 writeStream 关闭的其他事件吗?
-- to fully explain the workflow -- 充分解释工作流程
for (let i = 0; i < records.length; i++) {
await processZip(records[i]);
fs.unlinkSync(records[i].name);
}
inside the processZip method在 processZip 方法中
return new Promise(async (resolve, reject) => {
var readStream = this.s3Access.getObjectStream(guid);
let writeStream = fs.createWriteStream(file.name);
readStream.pipe(writeStream);
writeStream.on('finish', () => {
resolve();
}
}
According to NodeJS docs unlinkSync()
is calling C/C++ API call unlink
.根据 NodeJS 文档
unlinkSync()
正在调用 C/C++ API 调用unlink
。
Here is what docs says here这是文档在这里所说的
unlink() deletes a name from the filesystem.
unlink() 从文件系统中删除一个名称。 If that name was the last link to a file and no processes have the file open, the file is deleted and the space it was using is made available for reuse.
如果该名称是文件的最后一个链接,并且没有进程打开该文件,则该文件将被删除,并且它正在使用的空间可供重用。
If the name was the last link to a file but any processes still have the file open, the file will remain in existence until the last file descriptor referring to it is closed.
如果名称是文件的最后一个链接,但任何进程仍然打开该文件,则该文件将一直存在,直到引用它的最后一个文件描述符关闭。
If the name referred to a symbolic link, the link is removed.
如果名称引用了符号链接,则删除该链接。
If the name referred to a socket, FIFO, or device, the name for it is removed but processes which have the object open may continue to use it.
如果名称引用了套接字、FIFO 或设备,则删除它的名称,但打开 object 的进程可以继续使用它。
Basically all links should be closed for a file to be deleted.基本上应该关闭所有链接以删除文件。
Try to think of other processes that may use the files you are trying to delete (files opened with fs.open/openSync()
which were not closed etc.)尝试考虑可能使用您尝试删除的文件的其他进程(使用
fs.open/openSync()
打开但未关闭的文件等)
UPDATED更新
It looks like the file is deleted after the promise returned by processZip
is resolved successfully. processZip返回的
processZip
解析成功后,文件似乎被删除了。
It would be useful to also delete the file if the promise was rejected (ie error thrown in async function).如果 promise 被拒绝(即异步函数中抛出错误),删除该文件也会很有用。
for (let i = 0; i < records.length; i++) {
try {
await processZip(records[i]);
} finally {
// delete file after success or error
fs.unlinkSync(records[i].name);
}
}
Additionally 'error'
event handlers can be added to both streams in processZip
.此外,可以将
'error'
事件处理程序添加到processZip
中的两个流中。
I've found this note in stream.pipe()
docs , maybe it's related to the issue:我在
stream.pipe()
docs中找到了这个注释,可能与问题有关:
One important caveat is that if the
Readable
stream emits an error during processing, theWritable
destination is not closed automatically.一个重要的警告是,如果
Readable
stream 在处理过程中发出错误,Writable
目标不会自动关闭。 If an error occurs, it will be necessary to manually close each stream in order to prevent memory leaks.如果发生错误,则需要手动关闭每个 stream 以防止 memory 泄漏。
return new Promise(async (resolve, reject) => {
var readStream = this.s3Access.getObjectStream(guid);
let writeStream = fs.createWriteStream(file.name);
readStream.pipe(writeStream);
writeStream.on('finish', () => {
resolve();
}
readStream.on('error', err => {
readStream.end();
writeStream.end();
reject(err);
}
writeStream.on('error', err => {
readStream.end();
writeStream.end();
reject(err);
}
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.