PHP remove duplicates in a large array

Question

I was removing duplicates from an array, i have used array_unique($array) ,

here an example for the array i tested on: [7 6 4 3 3 4 9] .

'the initial order of the values must be kept' 

Here's my Function:

 function rmrepeat ($array) {
    $array2 = array_unique($array); 
    return $array2; 
 } 
 // Input Array 
 $a=array(7, 6, 4, 3, 3, 4, 9); 
 $newArray = rmrepeat($a);

Then i got 2 main errors:

• when i tested it on an array with 299 elements it didnt work.

• second problem is that it didnt return the last element of the array

What i have tried:

• i tried to search on google on who to remove duplicates in large arrays but didnt find the result i'm looking for
• and i have tried the function end($array) then array_push function to add the last element to my function, but it didnt work too, and i guess it isnt even the right solution. and i didnt receive any clear error to share with you.

what can the problem be?

thanks in advance.

Answer 1

try this
$a=array(7,6,4,3,3,4,9); 
$newArray = array_unique($a);
print_r($newArray);

or this one

$a=array(7,6,4,3,3,4,9);
$count = count($a);
$c = [];
for($i=0;$i<count($a);$i++){
  if(!in_array($a[$i],$c)){
     array_push($c,$a[$i]);
  }
}
 print_r($c);

Answer 2

Your code works fine. So I'm 100% sure that you wrongly thought that the last value is omitted. In fact it is removed because it's repeated and you asked all duplicate values to be removed by running array_unique() . For a simple test run this code and check if you see those problems again.

function rmrepeat ($array) {
    $array2 = array_unique($array); 
    return $array2; 
} 

$a = [];
for($i=0;$i<1000;$i++){
    array_push($a, $i);
}

echo "Before removing duplicates: ", (string)count($a), PHP_EOL;

$a = rmrepeat($a);

echo "After removing duplicates: ", (string)count($a), PHP_EOL;

here there would not be any duplicated value so there will not be any change in the array length. if you change array_push($a, $i); with array_push($a, rand(100, 200)); then many indices would be deleted and in most cases last item will be deleted too.

Answer 3

In order to keep the last item when using array_unique you need to reverse the array first to make the last item first.

$arr = ["a" => 7, "B"=>6, "C"=>4, "D"=>3, "E"=>3, "F"=>4, "G"=>9];
var_dump($arr);

$arr = array_reverse($arr); // reverse to remove first occurence
$arr = array_reverse(array_unique($arr)); // array_unique and reverse back again

var_dump($arr);

Now it will keep "E" => 3 and remove "D", and "F" remains but "C" is removed.
https://3v4l.org/7P4IP

---

If order means keys then you can add true as second parameter to array_revers to keep the keys.

$arr = [1, 2, 5, 2, 9];
var_dump($arr);

$arr = array_reverse($arr, true);
$arr = array_reverse(array_unique($arr), true);

var_dump($arr);

returns

array(4) {
  [0] => int(1)
  // Notice key 1 is missing with value 2
  [2] => int(5)
  [3] => int(2)
  [4] => int(9)
}

https://3v4l.org/DkETK

Answer 4

Another solution that could help, which I don't recommend is to override the default memory capacity. You can do this using this:

ini_set('memory_limit', '-1')

Further Reading: http://php.net/memory_limit