替换大文件中的字符串

Question

I have a server-client application where clients are able to edit data in a file stored on the server side. The problem is that the file is too large in order to load it into the memory (8gb+). There could be around 50 string replacements per second invoked by the connected clients. So copying the whole file and replacing the specified string with the new one is out of question.

I was thinking about saving all changes in a cache on the server side and perform all the replacements after reaching a certain amount of data. After reaching that amount of data I would perform the update by copying the file in small chunks and replace the specified parts.

This is the only idea I came up with but I was wondering if there might be another way or what problems I could encounter with this method.

Answer 1

When you have more than 8GB of data which is edited by many users simultaneously, you are far beyond what can be handled with a flatfile.

You seriously need to move this data to a database. Regarding your comment that "the file content is no fit for a database": sorry, but I don't believe you. Especially regarding your remark that "many people can edit it" - that's one more reason to use a database. On a filesystem, only one user at a time can have write access to a file. But a database allows concurrent write access for multiple users.

We could help you to come up with a database schema, when you open a new question telling us how your data is structured exactly and what your use-cases are.

Answer 2

您可以对数据使用某种形式的索引（在单独的文件中），以允许快速访问此巨大文件的相关部分（我们已经成功地对大型文件进行了此操作（〜200-400gb），但是正如Phillipp所述您应该将该数据移动到数据库中，尤其是用于读/写访问；某些框架（例如OSG ）已经带有用于3dterrain数据的数据库后端，因此您可以在那里查看它们的工作方式。