文件如何存储在Amazon S3 ..中？

Question

We all know that Dropbox uses Amazon S3 to Backup the Files and it's been said that every 15 mins, 1 Million files are handled by Dropbox. Then the Amazon S3 reads and Writes must be huge.

Any body have a clue how S3 can handle that many file descriptors ..? Is there any Different File System behind S3 ? I mean whether S3 creates a new File Descriptor for each and every file ..? Or Open a File ,write many files till it reaches some size say 1 GB and so on.

Answer 1

(Disclaimer: I do not actually know any of the implementation details of S3):

It sounds like you are making two incorrect assumptions:

1. Every dropbox "file" == one OS "file".
2. All files are being read/written on a single machine.

It's possible that each DropBox file maps to a single OS file, but it is also possible that they do something like the Google File System , which would break up files into multiple smaller files with a fixed max size.

Secondly, these files are not going to all be stored on a single machine, but rather stored on lots of different machines. So, you would not have that many file descriptors open on any given machine.

Thirdly, you may have forgotten that the default maximum number of file descriptors open at any given time can be raised by ulimit (from the commandline) or setrlimit (from C code). It would not surprise me if the limits were raised beyond the standard defaults to maximize available descriptors.