简体繁体 English

PHP长卷发请求

[英]PHP Long Curl Request

原文 2017-07-08 13:55:26 4 1 php/ json/ curl

I'm doing a Curl request to get a JSON file from a provider and the returned JSON file is huge. 我正在执行Curl请求以从提供程序获取JSON文件，并且返回的JSON文件很大。

If I run it in a browser, the response is a 21Mb string. 如果我在浏览器中运行它，则响应为21Mb字符串。 When I'm running this request with Curl, It doesn't just take a lot of time but it also ends with a Fatal error: Allowed memory size of 268435456 bytes exhausted error. 当我使用Curl运行此请求时，它不仅花费很多时间，而且还以Fatal error: Allowed memory size of 268435456 bytes exhausted结尾Fatal error: Allowed memory size of 268435456 bytes exhausted错误。

One way to solve it would be to add ini_set('memory_limit', '512M'); 解决该问题的一种方法是添加ini_set('memory_limit', '512M'); to my code (I already did it with 256 but it's still not enough...). 到我的代码（我已经用256做到了，但这还不够...）。 I was wondering if there is another way to get that data faster? 我想知道是否还有另一种方法可以更快地获取数据？

I know it's not a memory leak, it's just that the response is huge. 我知道这不是内存泄漏，只是响应很大。 I don't want to increase the memory limit, I know it's gonna work but I'd like to know if it's possible to get that data faster, even if I have to save it into a file to read it after. 我不想增加内存限制，我知道它可以工作，但是我想知道是否有可能更快地获取该数据，即使我必须将其保存到文件中以供以后读取。

Thanks! 谢谢！

1 个解决方案

well, JSON is UTF-8, and most UTF-8 data compresses very, very well. 好的，JSON是UTF-8，大多数UTF-8数据压缩得非常好。

now the first and easiest optimization, is to enable curl's built-in compression transfer, by doing CURLOPT_ENCODING=>'' , by setting it to emptystring, you tell curl to offer all the compression encodings that curl was compiled with (which usually includes gzip and deflate ) - and gzip in particular, when applied to jsons, can easily compress it down by a factor of 6-7, that should be a huge speedup in retrieving the json, provided that your libcurl is compiled with gzip support, and the target server supports gzip compressed trasnfer-encoding. 现在，第一个也是最简单的优化是通过执行CURLOPT_ENCODING=>''来启用curl的内置压缩传递，通过将其设置为emptystring，您可以告诉curl提供编译curl时使用的所有压缩编码（通常包括gzip和deflate ）-特别是gzip，当将其应用于json时，可以轻松将其压缩6-7倍，如果您的libcurl是使用gzip支持编译的，则gjson检索json的速度将会大大提高。目标服务器支持gzip压缩的trasnfer编码。

now, if you want to optimize even harder, the next step is to figure out what is the strongest compression supported by the target server, and implement that compression manually. 现在，如果您想更加优化，下一步就是找出目标服务器支持的最强压缩，然后手动实施该压缩。 lets say for example, that the strongest compression supported by your target server, is LZMA2 , in the xz format. 例如，假设目标服务器支持的最强压缩是xz格式的LZMA2。 then you'd manually set the header CURLOPT_HTTPHEADER=>array('Accept-Encoding: xz') , and after curl has downloaded it, decompress it manually, for example with https://github.com/payden/php-xz 然后您可以手动设置标头CURLOPT_HTTPHEADER=>array('Accept-Encoding: xz') ，然后在curl下载后，手动进行解压缩，例如使用https://github.com/payden/php-xz

xz, while not commonly supported in most web servers, offers even better compression than gzip, and can compress JSON's utf-8 up to a factor of 10 in some tests xz虽然在大多数Web服务器中通常不受支持，但提供的压缩比gzip更好，并且在某些测试中可以将JSON的utf-8压缩到10倍。

but it probably isn't xz, you should figure out the target server's strongest supported compression, whatever it is, and stick to it. 但是它可能不是xz，您应该弄清楚目标服务器支持的最强压缩，无论压缩如何，并坚持下去。