从字符串中删除重复项

Question

Basically I have a script, It checks a page for proxies and adds it to a string then checks it for duplicates and outputs them.

The only issue is the page being checked easily has 20k+ proxies on it so doing this makes it take about 3-4 minutes straight and most times will give me a bytes exhausted error or else max_execution error.

Is there any way to check for duplicates and just output them any quicker or easier?

<?php

ini_set('memory_limit', '-1');

set_time_limit(1000);

//Curl Setup;
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'x');
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);        
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);

//Execute Curl;
$page = curl_exec($ch);

//Regex For Matching Proxies;
preg_match_all('/(\d){1,3}\.(\d){1,3}\.(\d){1,3}\.(\d){1,3}:(\d){1,5}/ism',$page,$output);

//Foreach Proxy Found, Output it;
foreach($output[0] as $op){ $proxies .= $op."\n"; }

//This doesnt work
implode('\n',array_unique(explode('\n', $proxies)));

//Output each proxy
echo $proxies;

?>

Oh, Also, When it does get the proxies with no errors and puts them into a textarea using AJAX it makes serious lag issues. You wont be able to click on anything thats how much lag there is. Not sure if this whole issue is more to do with AJAX but not sure.

Answer 1

Store them in an array and than use array_unique.

$proxies = array ();
for($i =0, $max = count ($output[0]];$i <$max; $i++) {
  $proxies[] = $output [0][$i];
  // unset to reduce memory usage. Unsure if it'll actually help but
  unset($output [0][$i]);
}
echo implode("\n", array_unique  ($proxies));

Or use the values as keys to an associative array. Not sure if this would be any faster though

$proxies = array ();
foreach($output[0] as $op){
  $proxies[$op] = null;
}
echo implode("\n", array_keys  ($proxies));

Answer 2

According to this ( http://php.net/manual/en/function.array-unique.php#70786 i don't have test it) a possible way for what you want would be something like the following code:

//Populate the array maybe different for your needs
$proxies = array();
foreach($output[0] as $op) {
    $proxies[] = $op;
}
$unique_proxies = array_keys(array_flip($proxies)); 

***Request:

It would be nice if you could make a benchmark using your data and tell us the results.