I have a phrase counter function as part of a class that I would like to adapt to run very large data sets
<?php
private static function do_phrase_count($words, $multidim, $count, $pcount){
if($multidim === false){
$words = array($words);
}
$tally = array();
$arraycount = 0;
foreach($words as $wordgroup){
$max = count($wordgrounp) - $pcount;
for($x = 0; $x < $max; $x++){
$cutoff = $x + $pcount;
$spacekey = false;
$phrase = '';
$z = 0;
for($y = $x; $y < $cutoff; $y++){
if($spacekey) $phrase .= ' ';
else $spacekey = true;
$phrase .= $wordgroup[$y + $z];
$z++;
}
if(isset($tally[$phrase])){
$tally[$phrase]++;
$arraycount++;
}
else $tally[$phrase] = 1;
if($arraycount > 99999){
arsort($tally);
$tally = array_slice($tally, 0, 50000);
$arraycount = 49999;
}
}
}
arsort($tally);
$out = array_slice($tally, 0, $count);
return $out;
}
With every iteration, array_key_exists gets slower, so at a certain point I need to decrease the size of the tally array.
I was considering using a limit (100K) to stop the script from adding new array elements to $tally, or even using a percentage of total words, but after I stop adding new elements to the array I lose the ability to track trends that may pop up. (If I'm analysing data from a whole year, by the time I got to June, I wouldn't be able to see "summer time" as a trend).
Anyone have a solution as to how to limit my tally array to keep the script zinging without losing the ability to track trends?
UPDATE: I changed the script per your suggestions. Thank you for helping. I also figured out a solution to cut down on the size of the array.
In addition to Cheery's answer, remove count($var)
from the for loop. With each iteration, you are unnecessarily re-calculating the size of $var.
$groupsize = count($wordgroup) - $pcount;
for($x = 0; $x < $groupsize; $x++){
//...
Quote: God kills a kitten every time you call count() inside a loop.
if(isset($tally[$phrase]))
$tally[$phrase]++;
else
$tally[$phrase] = 1;
It should be faster than array_key_exists
ps: test sample
function genRandomString($length) {
$characters = '0123456789abcdefghijklmnopqrstuvwxyz';
$string = '';
for ($p = 0; $p < $length; $p++) {
$string .= $characters[mt_rand(0, strlen($characters)-1)];
}
return $string;
}
function microtime_float()
{
list($usec, $sec) = explode(" ", microtime());
return ((float)$usec + (float)$sec);
}
$len = 1000000; $str = genRandomString($len);
$tally = array();
$stamp1 = microtime_float();
for($i=0; $i<$len; $i++)
{
if(array_key_exists($str[$i], $tally))
$tally[$str[$i]]++;
else
$tally[$str[$i]] = 1;
}
echo microtime_float() - $stamp1 . '<br />';
$tally = array(); $stamp1 = microtime_float();
for($i = 0; $i<$len; $i++)
{
if(isset($tally[$str[$i]]))
$tally[$str[$i]]++;
else
$tally[$str[$i]] = 1;
}
echo microtime_float() - $stamp1 . '<br />';
Result:
0.80751395225525
0.44111108779907
Of course, the amount of keys is limited here.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.