[英]Chrome File System API hanging
disclaimer, self-answered post to hopefully save others time.免责声明,自我回答的帖子,希望能节省别人的时间。
Setup :设置:
I've been using chrome's implementation of the file systems API, [1] [2] [3] .我一直在使用 chrome 的文件系统 API [1] [2] [3]实现。
This requires enabling the flag chrome://flags/#native-file-system-api .这需要启用标志chrome://flags/#native-file-system-api 。
For starters I want to recursively read a directory and obtain a list of files.对于初学者,我想递归地读取目录并获取文件列表。 This is simple enough:这很简单:
paths = [];
let recursiveRead = async (path, handle) => {
let reads = [];
// window.handle = handle;
for await (let entry of await handle.getEntries()) { // <<< HANGING
if (entry.isFile)
paths.push(path.concat(entry.name));
else if (/* check some whitelist criteria to restrict which dirs are read*/)
reads.push(recursiveRead(path.concat(entry.name), entry));
}
await Promise.all(reads);
console.log('done', path, paths.length);
};
chooseFileSystemEntries({type: 'openDirectory'}).then(handle => {
recursiveRead([], handle).then(() => {
console.log('COMPLETELY DONE', paths.length);
});
});
I've also implemented a non-recursive while-loop-queue version.我还实现了一个非递归 while-loop-queue 版本。 And lastly, I've implemented a node fs.readdir
version.最后,我实现了一个节点fs.readdir
版本。 All 3 solutions work fine for small directories.所有 3 种解决方案都适用于小目录。
The problem:问题:
But then I tried running it on some sub-directories of the chromium source code ('base', 'components', and 'chrome');但后来我尝试在 chromium 源代码的一些子目录('base'、'components' 和 'chrome')上运行它; together the 3 sub-dirs consist of ~63,000 files. 3 个子目录共包含约 63,000 个文件。 While the node implementation worked fine (and surprisingly it utilized cached results between runs, resulting in instantaneous runs after the first), both browser implementations hung.虽然节点实现工作正常(令人惊讶的是它在运行之间使用了缓存的结果,导致在第一次运行后立即运行),但两个浏览器实现都挂了。
Attempted debugging:尝试调试:
Sometimes, they would return the full 63k files and print 'COMPLETLEY DONE'
as expected.有时,他们会返回完整的 63k 文件并按预期打印'COMPLETLEY DONE'
。 But most often (90% of the time) they would read 10k-40k files before hanging.但大多数情况下(90% 的时间)他们会在挂起之前读取 10k-40k 文件。
I dug deeper into the hanging, and apparently the for await
line was hanging.我更深入地研究了悬挂,显然for await
线正在悬挂。 So I added the line window.handle = handle
immediately before the for loop;所以我在 for 循环之前添加了行window.handle = handle
; when the function hung, I ran the for loop directly in the browser console, and it worked correctly.当函数挂起时,我直接在浏览器控制台中运行 for 循环,它运行正常。 So now I'm stuck.所以现在我被困住了。 I have seemingly working code that randomly hangs.我有看似随机挂起的有效代码。
Solution:解决方案:
I tried skipping over directories that would hang:我尝试跳过会挂起的目录:
let whitelistDirs = {src: ['base', 'chrome', 'components', /*'ui'*/]}; // 63800
let readDirEntry = (handle, timeout = 500) => {
return new Promise(async (resolve, reject) => {
setTimeout(() => reject('timeout'), timeout);
let entries = [];
for await (const entry of await handle.getEntries())
entries.push(entry);
resolve(entries);
});
};
let readWhile = async entryHandle => {
let paths = [];
let pending = [{path: [], handle: entryHandle}];
while (pending.length) {
let {path, handle} = pending.pop();
await readDirEntry(handle)
.then(entries =>
entries.forEach(entry => {
if (entry.isFile)
paths.push({path: path.concat(entry.name), handle: entry});
else if (path.length || !whitelistDirs[handle.name] || whitelistDirs[handle.name].includes(entry.name))
pending.push({path: path.concat(entry.name), handle: entry});
}))
.catch(() => console.log('skipped', handle.name));
console.log('paths read:', paths.length, 'pending remaining:', pending.length, path);
}
console.log('read complete, paths.length');
return paths;
};
chooseFileSystemEntries({type: 'openDirectory'}).then(handle => {
readWhile(handle).then(() => {
console.log('COMPLETELY DONE', paths.length);
});
});
And the results showed a pattern.结果显示出一种模式。 Once a directory read hung and was skipped, the subsequent ~10 dir reads would likewise hang and be skipped.一旦一个目录读取挂起并被跳过,随后的 ~10 个目录读取同样会挂起并被跳过。 Then the following reads would resume functioning properly until the next similar incident.然后接下来的读取将恢复正常运行,直到下一次类似事件发生。
// begins skipping
paths read: 45232 pending remaining: 49 (3) ["chrome", "browser", "favicon"]
VM60:25 skipped extensions
VM60:26 paths read: 45239 pending remaining: 47 (3) ["chrome", "browser", "extensions"]
VM60:25 skipped enterprise_reporting
VM60:26 paths read: 45239 pending remaining: 46 (3) ["chrome", "browser", "enterprise_reporting"]
VM60:25 skipped engagement
VM60:26 paths read: 45266 pending remaining: 45 (3) ["chrome", "browser", "engagement"]
VM60:25 skipped drive
VM60:26 paths read: 45271 pending remaining: 44 (3) ["chrome", "browser", "drive"]
// begins working properly again
So the issue seemed temporal.所以这个问题似乎是暂时的。 I added a simple retry wrapper with a 500ms wait between retries, and the reads began working fine.我添加了一个简单的重试包装器,重试之间有 500 毫秒的等待时间,读取开始正常工作。
readDirEntryRetry = async (handle, timeout = 500, tries = 5, waitBetweenTries = 500) => {
while (tries--) {
try {
return await readWhile(handle, timeout);
} catch (e) {
console.log('readDirEntry failed, tries remaining:', tries, handle.name);
await sleep(waitBetweenTries);
if (!tries)
return e;
}
}
};
Conclusion:结论:
The non-standard Native File System API hangs when reading large directories.读取大型目录时,非标准的本机文件系统 API 会挂起。 Simply retrying after waiting resolves the issue.只需在等待后重试即可解决问题。 Took me a good week to arrive at this solution, so thought it'd be worth sharing.我花了一周的时间才得出这个解决方案,所以认为它值得分享。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.