Django filesystem storage for large number of files

Question

Scenario: A Django app generates a lot of small files related to objects in different models.

I've done a lot of search for avoiding generation of large number of files in a single directory, when using the default Filestorage.

Is django-fsfield the only open source solution for this? Anything else you would recommend for fixing the large number of inodes in a dir?

Thank you!

Answer 1

Consider storing the content in a MongoDB collection.

Since the files are small, you can create MongoDB documents with a single binary field containing the file contents. "Small" in this case meaning less than the MongoDB document size limit, currently 16MB.

If you need to store big files, Mongo supports that with gridfs .

See also When to use GridFS .

Answer 2

I am using django-castor .

django-castor is a re-usable app for Django which provides a content-addressable storage backend . The main class, djcastor.storage.CAStorage , is a type of FileSystemStorage which saves files under their SHA-1 digest.

• No matter how many times the same file is uploaded, it will only ever be stored once, thus eliminating redundancy.

• Filenames are pseudorandom and made up only of hexadecimal characters, so you don't have to worry about filename collisions or sanitization.

• django-castor shards files in the uploads directory based on their digests; this prevents filesystem issues when too many files are in a single directory.

Answer 3

也可以使用自定义存储管理器，如果文件很小，请将其保存在带二进制字段的数据库模型中，并保存更多16MB的小文件，不需要使用其他数据库。