[英]How to avoid Maximum execution time of 60 seconds exceeded Laravel without change php.ini max_execution_time
I have a problem with my code where I will export to CSV which has more than 100,000 data, here I have used chunk
2 times, the first is the getData
variable, where this variable takes the AssetRepository
function in another class, and the other is my use when foreach
, if I load 1000 data using a limit
, the data can be exported.我的代码有问题,我将导出到包含超过 100,000 条数据的 CSV,这里我使用了 2 次
chunk
,第一个是getData
变量,该变量采用另一个类中的AssetRepository
函数,另一个是我在foreach
时使用,如果我使用limit
加载 1000 个数据,则可以导出数据。 Is it possible to load the data without changing max_execute_time
on php.ini
and only using chunk
?是否可以在不更改
php.ini
max_execute_time
并且仅使用chunk
情况下加载数据? if you can, how can I optimize my code ?如果可以,我该如何优化我的代码?
in this case, I'm using PostgreSQL.在这种情况下,我使用的是 PostgreSQL。
here is the code for AssetRepository.php
这是
AssetRepository.php
的代码
class AssetRepository
{
public $query = null;
private $trashed = false;
public static function query()
{
$repo = new AssetRepository();
$repo->query = DB::table('assets as a')
->select(
DB::raw("distinct a.id"),
"a.id",
"a.duration as duration",
DB::raw("COALESCE( NULLIF(a.qr_code,'') , 'QR Code Not Set' ) as qr_code"),
"a.material_no",
DB::raw("COALESCE( NULLIF(a.serial_no,'') , 'Serial No Not Set' ) as serial_no"),
"a.sbu_id",
"a.pop_id",
"a.building_id",
"a.type_id",
"asset_to_sid.cust_id",
"a.category_id",
"a.brand_id",
"a.model_id",
"a.id as id",
"b.name as model",
"b2.name as brand",
"p.name as pop",
"p2.name as sbu",
"q.name as building",
"a.updated_at",
"a.created_at",
"a.deleted_at",
'a.eos',
'a.eol',
"s.name as sts",
"c.name as category",
"a.app_code",
"a.name",
"a.status_approval as status_approval",
"a.approval_notes",
"a.approval_activities",
"a.habis_masa_garansi as habis_masa_garansi",
"permission_approval.action as action_approval",
DB::raw("CONCAT(u.first_name, ' ', u.last_name) as username"),
DB::raw("CONCAT(u2.first_name, ' ', u2.last_name) as username2"),
DB::raw("CONCAT(u3.first_name, ' ', u3.last_name) as approved_by"),
DB::raw("CASE WHEN q2.name is null THEN 'Not Set' ELSE q2.name END as room"),
DB::raw("CASE WHEN cast(a.installation_year as text) is null THEN 'Not Set' ELSE cast(a.installation_year as text) END as installation_year"),
DB::raw("CASE WHEN cast(b.mpls_hierarchy as text) is null THEN 'Not Set' ELSE cast(b.mpls_hierarchy as text) END as mpls_hierarchy"),
DB::raw("CASE WHEN cast(a2.name as text) is null THEN 'Not Set' ELSE cast(a2.name as text) END as rack"),
DB::raw("CASE WHEN cast(a.remark1 as text) is null THEN 'No Data' ELSE cast(a.remark1 as text) END as remark1"),
DB::raw("CASE WHEN cast(a.remark2 as text) is null THEN 'No Data' ELSE cast(a.remark2 as text) END as remark2"),
DB::raw("CASE WHEN cast(a.remark3 as text) is null THEN 'No Data' ELSE cast(a.remark3 as text) END as remark3"),
DB::raw("CASE WHEN cast(a.remark4 as text) is null THEN 'No Data' ELSE cast(a.remark4 as text) END as remark4"),
DB::raw("CASE WHEN cast(a.remark5 as text) is null THEN 'No Data' ELSE cast(a.remark5 as text) END as remark5"),
DB::raw("CASE WHEN cast(a.desc as text) is null THEN 'No Data' ELSE cast(a.desc as text) END as notes"),
DB::raw("CASE WHEN a.c_status = 1 THEN 'Complete' ELSE 'Not Complete' END AS complete"),
DB::raw("CASE WHEN a.c_status = 1 THEN 'btn-primary' ELSE 'btn-warning' END AS btn"),
DB::raw("CASE WHEN p.offline_sts = 1 THEN 'Offline' ELSE 'Online' END AS offline_sts"),
DB::raw("CASE WHEN p.offline_sts = 1 THEN 'btn-default' ELSE 'btn-info' END AS offline_btn"),
DB::raw("CASE WHEN p.offline_sts = 1 THEN 'disabled' ELSE 'enabled' END AS disableds")
)
->leftJoin('assets as a2', 'a.rack', '=', 'a2.id')
->join('kategoris as c', 'a.asset_category', '=', 'c.id')
->join('users as u', 'a.updated_by', 'u.id')
->join('users as u2', 'a.created_by', 'u2.id')
->leftJoin('users as u3', 'a.role_approval', 'u3.id')
->join('sbus as p', 'p.id', '=', 'a.pop_id')
->join('sbus as p2', 'p2.id', '=', 'a.sbu_id')
->leftJoin('pops as q', 'a.building_id', '=', 'q.id')
->leftJoin('pops as q2', 'a.room_id', '=', 'q2.id')
->leftJoin('brands as b', 'a.model_id', '=', 'b.id')
->leftJoin('permission_approval', 'a.permission_approval_id', '=', 'permission_approval.id')
->leftJoin('asset_to_sid', 'a.id', '=', 'asset_to_sid.asset_id')
->join('brands as b2', 'a.brand_id', '=', 'b2.id')
->join('statuses as s', 's.id', '=', 'a.status')
->leftJoin('statuses as ss', 'p.type', '=', 'ss.id')
->orderBy('a.updated_at', 'desc');
return $repo;
}
public function getQuery()
{
return $this->query ?? self::query();
}
public function get()
{
if (!$this->trashed) {
return $this->getQuery()->whereNull('a.deleted_at')->get();
}
return $this->getQuery()->get();
}
}
dan ini untuk export pada AssetController.php
dan ini untuk 导出 pada
AssetController.php
public function exportAll(Request $request)
{
$data = AssetRepository::query(); //From AssetRepository Function
$headers = array(
'Content-Type' => 'text/csv',
'Cache-Control' => 'must-revalidate, post-check=0, pre-check=0',
'Content-Disposition' => 'attachment; filename=export.csv',
'Expires' => '0',
'Pragma' => 'public',
);
$response = new StreamedResponse(function () use ($data) {
$handle = fopen('php://output', 'w');
$getData = $data->get();
$remark = Remark::all(['id','label','type']);
$remarkAsset = RemarkAsset::all(['asset_id','value','remark_id']);
$getHeader = array_keys((array)$getData[0]);
$newArray = array();
$setHeader = array();
foreach ($getHeader as $header) {
$setHeader[$header] = $header;
}
$remarkHeader = []; //result
foreach ($remark as $headerRemark) {
$remarkHeader[] = array(
'id' => $headerRemark['id'],
'label' => $headerRemark['label'],
'type' => $headerRemark['type']
);
$setHeader[$headerRemark['type']] = $headerRemark['type'];
}
$remarkAssets = [];
foreach ($remarkAsset as $assetRemark) {
$remarkAssets[] = (array)array(
'asset_id' => $assetRemark['asset_id'],
'value' => $assetRemark['value'],
'remark_id' => $assetRemark['remark_id']
);
}
array_push($newArray, (object)$setHeader);
// $coountData = count($getData) / 4;
$chunk = collect($getData);
$chunk->chunk(500);
foreach ($chunk as $data) {
$theKey=array_keys(array_combine(array_keys($remarkAssets), array_column($remarkAssets, 'asset_id')),$data->id);
foreach ($remarkHeader as $head) {
$countKey = count($theKey);
if ($countKey > 0) {
$valueRemark = '';
foreach ($theKey as $key) {
if ($remarkAssets[$key]['remark_id'] == $head['id']) {
$valueRemark = $remarkAssets[$key]['value'];
}
}
$data = (array)$data;
$data[$head['type']] = $valueRemark;
$data = (object)$data;
} else {
$data = (array)$data;
$data[$head['type']] = '';
$data = (object)$data;
}
}
array_push($newArray, $data);
}
$chunkArray = collect($newArray);
$chunkArray->chunk(500);
foreach ($chunkArray as $datas) {
if (is_object($datas))
$datas = (array)$datas;
fputcsv($handle, $datas);
}
fclose($handle);
}, 200, $headers);
return $response->send();
}
if necessary ignore AssetController.php
it is the query used in my code如有必要,请忽略
AssetController.php
它是我的代码中使用的查询
It's better to do these long running tasks asynchronously.最好异步执行这些长时间运行的任务。 In Laravel you can use queues for that.
在 Laravel 中,您可以为此使用队列。 As queues run on CLI you could configure a different
max_execution_time
for that.当队列在 CLI 上运行时,您可以为此配置不同的
max_execution_time
。 If you would prefer to keep the execution time the same, then you should try splitting up the task you're performing into multiple parts.如果您希望保持相同的执行时间,那么您应该尝试将您正在执行的任务分成多个部分。 If each of those parts does not exceed 1 minute, then you're good to go.
如果每个部分都不超过 1 分钟,那么您就可以开始了。
You can call set_time_limit(0)
to remove the time limit from the rest of the execution, or you can call set_time_limit(n)
in each iteration of a loop (for example) to reset the timer for n more seconds.您可以调用
set_time_limit(0)
以从执行的其余部分中删除时间限制,或者您可以在循环的每次迭代中调用set_time_limit(n)
(例如)将计时器重置 n 秒。
https://www.php.net/manual/en/function.set-time-limit.php https://www.php.net/manual/en/function.set-time-limit.php
ini_set('max_execution_time', 180); //3 minutes
Maybe this query returns so many elements PHP spends most of the time just wrapping a Collection
object around them.也许这个查询返回了很多元素,PHP 大部分时间只是在它们周围包装一个
Collection
对象。 If you want to see how much time is spent on the query itself, you could run it directly on your PostgreSQL Server
, the console ( php artisan tinker
) or use DB::listen
in your code如果您想查看查询本身花费了多少时间,您可以直接在您的
PostgreSQL Server
、控制台( php artisan tinker
)或在您的代码中使用DB::listen
运行它
public function exportAll(Request $request)
{
// PHP >= 7.4.0
DB::listen(fn($query) => dump($query->sql, $query->bindings, $query->time));
// PHP < 7.4.0
DB::listen(function ($query) { dump($query->sql, $query->bindings, $query->time); });
...
}
If the Collection
wrapping is the issue, try using a LazyCollection
.如果
Collection
包装是问题所在,请尝试使用LazyCollection
。 It's available since Laravel 6.0
.它从
Laravel 6.0
可用。 You use it by calling $data->cursor()
instead of $data->get()
.您可以通过调用
$data->cursor()
而不是$data->get()
来使用它。
A LazyCollection
is basically an object you can iterate over and use some Collection
methods on. LazyCollection
基本上是一个可以迭代并使用一些Collection
方法的对象。 They allow you to work with the data without the overhead of building a big Collection
for X amount of rows.它们允许您处理数据,而无需为 X 行构建大型
Collection
的开销。
I'll repost your exportAll
function with some changes I think will positively impact performance.我将重新发布您的
exportAll
函数,并进行一些我认为会对性能产生积极影响的更改。
public function exportAll(Request $request)
{
$data = AssetRepository::query(); //From AssetRepository Function
$headers = array(
'Content-Type' => 'text/csv',
'Cache-Control' => 'must-revalidate, post-check=0, pre-check=0',
'Content-Disposition' => 'attachment; filename=export.csv',
'Expires' => '0',
'Pragma' => 'public',
);
$response = new StreamedResponse(function () use ($data) {
$handle = fopen('php://output', 'w');
/**
* Use a LazyCollection instead
* $getData = $data->get();
*/
$getData = $data->cursor();
$remark = Remark::all(['id','label','type']);
$remarkAsset = RemarkAsset::all(['asset_id','value','remark_id']);
/**
* Since we are using a LazyCollection,
* we can't treat $getData as an array directly.
*
* $getHeader = array_keys((array)$getData[0]);
*/
$getHeader = array_keys((array)$getData->get(0));
$newArray = array();
/**
* This can be achieved with array_combine
*
* $setHeader = array();
*
* foreach ($getHeader as $header) {
* $setHeader[$header] = $header;
* }
*/
$setHeader = array_combine($getHeader, $getHeader);
/**
* $remarkHeader is unnecesary. You can just call $remark->toArray() instead.
* Also, what you're trying to do with the following foreach can be done with
* a combination of array_merge and array_combine
*
* $remarkHeader = []; //result
*
* foreach ($remark as $headerRemark) {
* $remarkHeader[] = array(
* 'id' => $headerRemark['id'],
* 'label' => $headerRemark['label'],
* 'type' => $headerRemark['type']
* );
*
* $setHeader[$headerRemark['type']] = $headerRemark['type'];
* }
*/
$setHeader = array_merge(
$setHeader,
array_combine(
$remark->pluck('type')->toArray(),
$remark->pluck('type')->toArray()
)
);
/**
* Again, $remarkAssets is unnecessary. All you're doing with this loop
* is the same as calling $remarkAsset->toArray()
*
* $remarkAssets = [];
* foreach ($remarkAsset as $assetRemark) {
* $remarkAssets[] = (array)array(
* 'asset_id' => $assetRemark['asset_id'],
* 'value' => $assetRemark['value'],
* 'remark_id' => $assetRemark['remark_id']
* );
* }
*/
array_push($newArray, (object)$setHeader);
// $coountData = count($getData) / 4;
/**
* $getData is already a Collection. Here, you're telling PHP to rebuild it
* for no reason. For large collections, this adds a lot of overhead.
* You can already call the chunk method on $getData anyways.
* You could do $chunk = $getData->chunk(500) for example.
* It's not even necessary to make a new variable for it since you won't use
* $chunk again after this.
*
* $chunk = collect($getData);
* $chunk->chunk(500);
*
* Also, according to the docs you're not using chunk properly.
* https://laravel.com/docs/6.x/collections#method-chunk
* You're supposed to loop twice because the chunk method doesn't alter the collection.
* If you run
* $chunk->chunk(500)
* foreach($chunk as $data) { ... }
* You're still looping over the entire Collection.
* Since your code is not made to work with chunks, I'll leave it like that
*
* foreach ($chunk as $data) {
*/
foreach ($getData as $data) {
/**
* This seems to return an array of the keys of $remarkAssets
* where 'asset_id' is equal to $data->id.
* You can achieve this through Collection methods on $remarkAsset instead.
*
* $theKey = array_keys(
* array_combine(
* array_keys($remarkAssets),
* array_column($remarkAssets, 'asset_id')
* ),
* $data->id
* );
*
* Since there is no real need to return an array, I'll leave $theKey as a collection.
*/
$theKey = $remarkAsset->where('asset_id', $data->id)->keys();
/**
* Since $remarkHeader doesn't exist in this context, we use $remark instead
*
* foreach ($remarkHeader as $head) {
*
* Since $theKey is a collection, the count is obtained
* through the count() Collection method. Also, since you don't
* ever use $countKey again, you could inline it instead.
*
* $countKey = count($theKey);
*
* if ($countKey > 0) {
*/
foreach ($remark as $head) {
if ($theKey->count() > 0) {
$valueRemark = '';
foreach ($theKey as $key) {
/**
* Since $remark is a collection and $head an object
* the following if statement needs to be rewritten
*
* if ($remarkAssets[$key]['remark_id'] == $head['id']) {
* $valueRemark = $remarkAssets[$key]['value'];
* }
*/
if ($remark->get($key)->remark_id == $head->id) {
$valueRemark = $remark->get($key)->value;
}
}
/**
* $data being a stdClass, you can just set the property instead of
* going through the trouble of casting it as an array, setting a value
* and then re-casting it as an object.
*
* $data = (array)$data;
* $data[$head['type']] = $valueRemark;
* $data = (object)$data;
* } else {
* $data = (array)$data;
* $data[$head['type']] = '';
* $data = (object)$data;
*/
$data->{$head['type']} = $valueRemark;
} else {
$data->{$head['type']} = '';
}
}
array_push($newArray, $data);
}
$chunkArray = collect($newArray);
/**
* As explained earlier, your use of chunk() doesn't do anything.
* We can then safely remove this line.
*
* $chunkArray->chunk(500);
*/
foreach ($chunkArray as $datas) {
if (is_object($datas))
$datas = (array)$datas;
fputcsv($handle, $datas);
}
fclose($handle);
}, 200, $headers);
return $response->send();
}
public function exportAll(Request $request)
{
$data = AssetRepository::query(); //From AssetRepository Function
$headers = array(
'Content-Type' => 'text/csv',
'Cache-Control' => 'must-revalidate, post-check=0, pre-check=0',
'Content-Disposition' => 'attachment; filename=export.csv',
'Expires' => '0',
'Pragma' => 'public',
);
$response = new StreamedResponse(function () use ($data) {
$handle = fopen('php://output', 'w');
$getData = $data->cursor();
$remark = Remark::all(['id','label','type']);
$remarkAsset = RemarkAsset::all(['asset_id','value','remark_id']);
$getHeader = array_keys((array)$getData->get(0));
$newArray = array();
$setHeader = array_combine($getHeader, $getHeader);
$setHeader = array_merge(
$setHeader,
array_combine(
$remark->pluck('type')->toArray(),
$remark->pluck('type')->toArray()
)
);
array_push($newArray, (object)$setHeader);
foreach ($getData as $data) {
$theKey = $remarkAsset->where('asset_id', $data->id)->keys();
foreach ($remark as $head) {
if ($theKey->count() > 0) {
$valueRemark = '';
foreach ($theKey as $key) {
if ($remark->get($key)->remark_id == $head->id) {
$valueRemark = $remark->get($key)->value;
}
}
$data->{$head['type']} = $valueRemark;
} else {
$data->{$head['type']} = '';
}
}
array_push($newArray, $data);
}
$chunkArray = collect($newArray);
foreach ($chunkArray as $datas) {
if (is_object($datas))
$datas = (array)$datas;
fputcsv($handle, $datas);
}
fclose($handle);
}, 200, $headers);
return $response->send();
}
You could also use Lazy Collections for Remark an RemarkAsset models like so您也可以像这样使用 Lazy Collections for Remark an RemarkAsset 模型
$remark = Remark::select('id','label','type')->cursor();
$remarkAsset = RemarkAsset::select('asset_id','value','remark_id')->cursor();
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.