绕过PHP CURL 2GB下载限制

Question

I am trying to download a 7GB file using php-curl however it seems that it will only download the first 2GB.

There doesn't seem to be any documentation or talk about this.

Anyone have any ideas?

Answer 1

Here are two useful links on the topic:

Downloading a large file using curl

How to partially download a remote file with cURL?

Basically you may have two problems here:

• You are reading into memory first as as such exhausting PHPs memory allocation
• You need to chunk download the file to overcome certain restrictions in the HTTP protocol.

There are also file sytem limitations and what not so check your file system type as mentioned by @ajreal (ie FAT32 has a limit of 4GB, 99% chance your not using FAT but still it is an example).

As the OP found out it was do with the DB:

Turns out it was a database issue. File sizes were stored in a mysql database, the sizes were in bytes and max size for "int" column is 2147483648. Changing the column type to "bigint" fixed the issue.

Answer 2

Assuming your file system can handle files larger than 2GB you can try using copy

copy("http:://example.org/your_file","/tmp/your_file");

Also make sure you set an appropriate time limit (with set_time_limit(...) ).