具有Java库的随机访问结构化归档文件格式

Question

My team and I have been given a requirement of a file format, with Java library support, that holds various metadata about some larger file. In fact, the powers that be would like us to wrap the large file (maybe 100MB) and the other related file (metadata, non-destructive edits, etc.) into one bundled archive file.

For a one-off creation, that's a breeze: just throw everything in a Zip file. But we want to be able to constantly update the metadata, the non-destructive edits, etc. We don't want dump the whole >100MB contents to a temporary directory and then zip everything back up just to add a line to one of the metadata files.

There are some projects (eg TrueVFS ) that on the surface sound ideal by claiming to abstract a zip file or other archive file format as a file system. But on closer inspection it would seem that the only in-place update functionality we get is simple appending new files and not actually changing or appending to individual files.

What we need is some file format that's in between a Zip file and a relational database. Something with a hierarchical structure would be great. It must efficiently support reasonably large files (over 100MB) and allow random access to add, remove, and change individual files within the archive. I was surprised to be unable to find anything. Any suggestions?

PS I've had bad experiences years ago with the Microsoft compound file format getting corrupted. I don't know if something like Apache POIFS is reliable and efficient with large files.

Answer 1

I do not believe that what you are asking for is easily doable for one simple reason: filesystems do not generally support inserting data in the middle of a file - not without truncating and rewriting the remainder. This means that a simple append on a plain file turns into a truncate-rewrite operation when that file is stored in an archive.

You would have to find some block-based format that would essentially replicate much of the functionality of an actual filesystem, in order to allow such operations.

I would look at refactoring the whole system to enforce some structure on that big data file. That would allow you to turn it into something that can be stored in a database. For example, line based text could be stored in a table with two columns - a line number as a primary key and the line text. Any line based operation would easily turn into a DB-based one.

You could then just use an embedded database such as SQLite to keep everything in the same file without depending on an external server.

Answer 2

Depending on what platforms you want to run your application on, you can use our Solid File System - this is a virtual file system backed by an automatically resizable container file. It's written in Ansi C and has Java JNI wrapper for Android (and this wrapper can be brought to other platforms on request - we just didn't have such goal before).

There also exists Codebase File System, which as I understand also offers a JNI for Java.