S3 Bucket Amazon issue

Question

1) I have upload form

2) It uploads file to my local storage move_uploaded_file.

3) It uses zend putObject function to move file to s3 object.

Everything works ok till I have file size of around 30Mb to 40 Mb. The problem is when I try uploading larger files like 80 Mb, 100 Mb or so, the file moving to s3 takes ages to complete the upload. My code is something like this:

$orginalPath = APPLICATION_PATH."/../storage/".$fileName;
move_uploaded_file($data['files']['tmp_name'], "$orginalPath");

$s3 = new Zend_Service_Amazon_S3($accessKey, $secretKey);

$s3->putObject($path, file_get_contents($orginalPath),
    array(Zend_Service_Amazon_S3::S3_ACL_HEADER =>Zend_Service_Amazon_S3::S3_ACL_PUBLIC_READ));

Can you help how to handle large files move quickly I tried using streamWrapper like this

$s3->registerStreamWrapper("s3");


file_put_contents("s3://my-bucket-name/orginal/$fileName", file_get_contents($orginalPath));

But no luck, it take same long time to move file.

Hence, is there an efficient way to move file quickly to s3 bucket?

Answer 1

The answer is a worker process. You can start a PHP worker script via PHP CLI on server boot, perhaps with a GearmanClient php extension and gearman server running on your box. Then you queue a background job to upload the file to S3 while your main site PHP code returns success after issuing the job and the file happily uploads in the background while your foreground site continues on it's merry way. Another way of doing this is making another server do all of this task while your main site remains utilization free of this process. I am doing this now. It works well.

Answer 2

You could consider using the more direct POST to S3 feature . The AWS SDK for PHP has a class to help generate the data for the form .