简体繁体 English

如何以较少的处理时间处理大文件

[英]How to handle large files with small process times

原文 2012-10-12 08:02:55 6 3 php/ javascript/ html

On my webservice users are allowed to create forms and let theire friends or co-workers create data with them. 在我的Web服务上，允许用户创建表单，并让其朋友或同事与他们一起创建数据。 The collected data can be downloaded as a zip file stream. 收集的数据可以作为zip文件流下载。 Sometimes users have huge amounts of data (up to 2gb) and the server simply kills the php process for obvious reasons. 有时，用户拥有大量数据（最大2gb），并且服务器出于明显的原因简单地终止了php进程。 Is it somehow possible to create such a file on client side without flash,java (btw java doesn't work anyway for most of my users) etc. ? 是否可以在没有Flash，Java的情况下在客户端创建这样的文件（对于大多数用户，BTW Java仍然无法正常工作）等？

3 个解决方案

Increase your script timeout and memory usage. 增加脚本超时和内存使用率。

use set_time_limit function Docs 使用set_time_limit函数文档

And use ini_set for memory_limit parameter. 并使用ini_set作为memory_limit参数。

And one more solution is to give the clients file parts. 另一种解决方案是为客户端提供文件部分。 ie give them a limit for downloading the number of records. 即给他们一个限制，以下载记录的数量。 ie 1-1000, 1001-2000 etc 即1-1000、1001-2000等

If you have control over the web server process I suggest you explore x-send-file as a solution to this. 如果您可以控制Web服务器进程，建议您探索x-send-file作为对此的解决方案。

See this SO question 看到这个问题

In essence it will end the php process and send the file via the http server. 本质上，它将结束php进程并通过http服务器发送文件。 This way time limits aren't an issue and you don't have a php instance hanging around. 这样，时间限制就不成问题，并且您没有php实例徘徊。

Create a worker shell that keeps running in the background in a loop and checks for new data. 创建一个工作程序外壳，使其在后台循环运行，并检查是否有新数据。 If it finds new, unprocessed data have it prepare the download in the background. 如果发现未处理的新数据，则在后台准备下载。 When the data is ready for download flag it as "ready" and inform the user (by email, polling via ajax for an status update, however you like) that his data was processed and is ready for download. 当数据准备好下载时，将其标记为“就绪”，并通知用户（通过电子邮件，通过ajax轮询以更新状态，但是您愿意），他的数据已被处理并且可以下载。

You can use nice to limit the CPU power used for that shell to avoid that it consumes all the available processing power and your site becomes slow. 您可以使用nice来限制用于该shell的CPU能力，以避免它消耗所有可用的处理能力，并且您的站点变慢。

That's exactly how I handle audio and video processing in one of my projects and it works fine. 这正是我在一个项目中处理音频和视频处理的方式，并且效果很好。