Ruby: Create files with metadata

Question

We're creating an app that is going to generate some text files on *nix systems with hashed filenames to avoid too-long filenames.

However, it would be nice to tag the files with some metadata that gives a better clue as to what their content is.

Hence my question. Does anyone have any experience with creating files with custom metadata in Ruby?

I've done some searching and there seem to be some (very old) gems that read metadata: https://github.com/kig/metadata http://oai.rubyforge.org/

I also found: system file, read write o create custom metadata or attributes extended which seems to suggest that what I need may be at the system level, but dropping down there feels dirty and scary.

Anyone know of libraries that could achieve this? How would one create custom metadata for files generated by Ruby?

Answer 1

A very old but interesting question with no answers!

In order for a file to contain metadata, it has to have a format that has some way (implicitly or explicitly) to describe where and how the metadata is stored.

This can be done by the format, such as having a header that says where the "main" data is stored and where the "metadata" is stored, or perhaps implicitly, such as having a length to the "main" data, and storing metadata as anything beyond the "main" data.

This can also be done by the OS/filesystem by storing information along with the files, such as permission info, modtime, user, and more comprehensive file information like "icon" as you would find with iOS/Windows.

(Note that I am using "quotes" around "main" and "metadata" because the reality is that it's all data, and needs to be stored in some way that tools can retrieve it)

A true text file does not contain any headers or any such file format, and is essentially just a continuous block of characters (disregarding how the OS may store it). This also means that it can be generally opened by any text editor, which will merely read and display all the characters it finds.

So the answer in some sense is that you can't, at least not on a true text file that is truly portable to multiple OS.

A few thoughts on how to get around this:

Use binary at the end of the text file with hope/requirements that their text editor will ignore non-ascii.

Store it in the OS metadata for the file and make it OS specific (such as storing it in the "comments" section that an OS may have for a file.

Store it in a separate file that goes "along with" the file (ie, file.txt and file.meta) and hope that they keep the files together.

Store it in a separate file and zip the text and the meta file together and have your tool be zip aware.

Come up with a new file format that is not just text but has a text section (though then it can no longer be edited with a text editor).

Store the metadata at the end of the text file in a text format with perhaps comments or some indicator to leave the metadata alone. This is similar to the technique that the vi/vim text editor uses to embed vim commands into a file, it just puts them as comments at the beginning or end of the file.

I'm not sure there are many other ways to accomplish what you want, but perhaps one of those will work.