How does the backend on sites like Megaupload work?

Question

There are numerous megaupload type sites out there, such as rapidshare, fileserve, filesonic, wupload etc. I'm curious as to how things work behind the scenes. Once a user uploads the file, how is it stored? Is some sort of distributed file system used? How does the site scale as it becomes more popular? Obviously they add more servers, but how does the load balancing work? Thanks.

Answer 1

One technique they employ is file referencing, which is a great way to scale as it becomes more popular. The basic concept is that when a file is uploaded multiple times, it simply adds a reference to the first original upload (even Dropbox does this). This is actually a major complaint of the RIAA / MPAA in the MegaUpload case because if a file is requested to be taken down, only the reference is removed. The other references and the original file itself stays because MegaUpload does not know whether these were uploaded by the copyright holder or by copyright infringers.

Answer 2

Using Amazon Web Services S3 is one approach. You might be interested in real world examples described in HighScalability such as this one for Instagram