Fastest way to compute & feed an arbitrary color centroid in an image to PHP

Question

I'm looking for the fastest way to compute a directional vector based on an arbitrary color in an image (a Rpi camera, but a JPEG file for testing is OK for now), aka tracking a colored ball project. Please note that the resulting vector (or centroid coordinates, whatever) needs to be passed to PHP for the program execution , so the solution I'm looking for needs to end with PHP, but can be anything before, given it can be implemented on both Windows and Linux.

Consider an input JPEG image:

Here are 2 example directional vectors I'm after, obtained based on a 1) teal color input and 2) purple color input. Obviously, only 1 vector will ever be asked at a time, I put 2 to demonstrate multiple examples into 1 image, but it's always gonna be only 1 vector at a time. Note that the resulting vectors ("v") are standardized to -1.0 (bottom/left) to +1.0 (bottom/right) so that zero is the middle of the picture.

Here are the various solutions I've implemented/tested so far and how much time the whole process takes, based on a 960x640 JPEG picture, but the implemented solution will be tied to a Rpi camera input, I do not have the camera yet so I use a JPEG image until the camera arrives from China.

1) 2700ms : Use GD2 that is bundled with PHP, for loop over each pixels, push pixels matching ~10% RGB values in XY arrays, average the XY arrays, compute/normalize directional vector from XY arrays.

$arr_matching_pixels = array('arr_x' => array(), 'arr_y' => array());
for($y = 0; $y < $h - 1; $y++){
    for($x = 0; $x < $w - 1; $x++){
        $arr_pixel = imagecolorsforindex($img, imagecolorat($img, $x, $y));
        if(abs($arr_pixel['red'] - $arr_seek_color['red']) < 30){
            if(abs($arr_pixel['green'] - $arr_seek_color['green']) < 30){
                if(abs($arr_pixel['blue'] - $arr_seek_color['blue']) < 30){
                    array_push($arr_matching_pixels['arr_x'], $x);
                    array_push($arr_matching_pixels['arr_y'], $y);
                }
            }
        }
    }
}
// Compute centroid of color... etc...

2) 700ms : Same as #1 except begin by resizing the canvas by 50% (acceptable loss) using imagecreatefromjpeg('_test_cam_img.jpg');

3) 560ms : Same as #2 except use ImageMagick with a pixel iterator loop to read the pixels

$imagick = new Imagick(realpath($o_img));
$arr_matching_pixels = array('arr_x' => array(), 'arr_y' => array());
$arr_pixel = array();
$iterator = $imagick->getPixelIterator();
foreach($iterator as $y => $pixels){
    foreach($pixels as $x => $pixel){
        $arr_pixel = $pixel->getColor();
        if(abs($arr_pixel['r'] - $arr_seek_color['red']) < 30){
            if(abs($arr_pixel['g'] - $arr_seek_color['green']) < 30){
                if(abs($arr_pixel['b'] - $arr_seek_color['blue']) < 30){
                    array_push($arr_matching_pixels['arr_x'], $x);
                    array_push($arr_matching_pixels['arr_y'], $y);
                }
            }
        }
    }
}
// Compute centroid of color... etc...

4) 340ms : Call the system's ImageMagick binary via the exec() function, pass it the image location, the chroma/color key, a resize by 50% param, a 10% fuzz param, and the sparse-color: modifier to extract a textual (CSV-like) list representation of desired pixels, then use PHP to loop over each line, explode commas and push all pixels in XY arrays, average the XY arrays, compute/normalize directional vector from XY arrays. I noted that calling exec() proves to be quite slower than executing the same command directly from the Windows command line.

$imagick = new Imagick(realpath($o_img));
$out = exec('"E:\Users\Ben\Roaming Apps\imagemagick-6.9.3\convert" E:\wamp64\www\test_cam_img.jpg -resize 50% -fuzz 10% +transparent rgb(' . $arr_seek_color['red'] . ',' . $arr_seek_color['green'] . ',' . $arr_seek_color['blue'] . ') sparse-color:');
$arr_lines = explode(' ', $out);
$arr_matching_pixels = array('arr_x' => array(), 'arr_y' => array());
foreach($arr_lines as $str_line){
    $arr_xy_coords = explode(',', $str_line);
    array_push($arr_matching_pixels['arr_x'], $arr_xy_coords[0]);
    array_push($arr_matching_pixels['arr_y'], $arr_xy_coords[1]);
}
// Compute centroid of color... etc...

5) 32ms : PHP creates an "in" text file containing the image path and the chroma/color key and begins looping until it reads an "out" text file. A python+OpenCV script already/always runs a (stoppable) infinite loop constantly looking for an "in" text file and when it exists, it read it, explodes the values, makes a 1-bit mask using the HSV values ~10% (cv2.inRange) from the "in" file, then makes an array using cv2.findNonZero(mask) and computes the array mean value and writes it to an "out" text file that PHP immediately reads, containing the directional vector value. This is by far, the fastest way I have found, but it is awkward, because it implies that the python script will have to be programmed in a CRONJOB and monitored/relaunched in a single instance if it crashes.

file_put_contents('_avg_color_coords_in.txt', $o_img . "\n" . $arr_seek_color['h'] . ',' . $arr_seek_color['s'] . ',' . $arr_seek_color['l']);

$starttime = time();
while((time() - $starttime) < 5){ // Max 5 seconds (exaggerated)
    if(file_exists('_avg_color_coords_out.txt')){
        $dir_vector = (float) file_get_contents('_avg_color_coords_out.txt');
        if(!@unlink('_avg_color_coords_out.txt')){
            sleep(1);
            unlink('_avg_color_coords_out.txt');
        }
        break;
    }
    usleep(2000);
}
// $dir_vector ("v", the centroid of the color) is already computed by Python


// ---------- PYTHON SCRIPT ----------
import math
import cv2
import numpy as np
import os
import time

#cap = cv2.VideoCapture(0)

#while (1):
#    _, frame = cap.read()
if(os.path.exists('_avg_color_coords_stop.txt')):
    exit()
while not os.path.exists('_avg_color_coords_in.txt'):
    time.sleep(0.002)
f = open('_avg_color_coords_in.txt', 'r')
imgsrc = f.readline().rstrip('\n')
rgbcol = [int(x) for x in f.readline().rstrip('\n').split(',')]
frame = cv2.imread(imgsrc)
h, w = frame.shape[:2]

hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
hfacl = rgbcol[0] / 360 * 180 * 0.95
hfach = rgbcol[0] / 360 * 180 * 1.05
sfacl = rgbcol[1] / 100 * 255 * 0.9
sfach = rgbcol[1] / 100 * 255 * 1.1
vfacl = rgbcol[2] / 100 * 255 * 0.9
vfach = rgbcol[2] / 100 * 255 * 1.1
lower_color = np.array([hfacl, sfacl, vfacl]) # 0..180, 0..255, 0..255 not percentage!
upper_color = np.array([hfach, sfach, vfach]) # 0..180, 0..255, 0..255 not percentage!
mask = cv2.inRange(hsv, lower_color, upper_color)
#cv2.imshow('mask', mask)

points = cv2.findNonZero(mask)
if(points.any()):
    avg = np.mean(points, axis=0)
else:
    avg = [0,0]
#print(avg)

v = -math.atan(((w * 0.5) - avg[0][0]) / (h - avg[0][1])) / (3.1415 * 0.5);
f2 = open('_avg_color_coords_out.txt', 'w+')
f2.write("%s" % str(v))

#    k = cv2.waitKey(5) & 0xff
#    if k == 27:
#        break

#cv2.destroyAllWindows()
#cap.release()

f2.close()
f.close()
os.remove('_avg_color_coords_in.txt')

6) 38ms : Same as #5 except begin by resizing the canvas by 50% (acceptable loss) which doesn't seem to speed up things at all, and even seems counterproductive a little bit.

Is there a faster way or is this optimal? This will run every second on a 900mhz Rpi, so it needs to be quick. I think 30ms on a 900mhz CPU will be around 150-200ms (not tested yet, waiting for the camera to ship)

Answer 1

I had a quick go in php-vips :

#!/usr/bin/env php
<?php

require __DIR__ . '/vendor/autoload.php';

use Jcupitt\Vips;

$image = Vips\Image::newFromFile($argv[1], ['access' => 'sequential']);

# Target colour in RGB.
$target = [50, 10, 100];

# Select pixels where all bands are less than 10 away from the target.
# (and render it to memory ... we'll be reusing this mask image).
# The mask image will have one band with 0 for false and 255 for true.
$mask = $image->subtract($target)->abs()->less(10)->bandand()->copyMemory();

# The number of set pixels in the mask.
$n_set = $mask->avg() * $mask->width * $mask->height / 255;

# Handy for debugging: uncomment to write the mask image for inspection.
# $mask->writeToFile("x.png");

# Make a two-band image where band 0 is x coordinates and band 1 is y
# coordinates.
$coords = Vips\Image::xyz($mask->width, $mask->height);

# Make an indexed histogram: sum $coords at each position.
$pos = $coords->hist_find_indexed($mask);

# fetch the sum of the 255 value (true) pixels
[$x_sum, $y_sum] = $pos->getpoint(255, 0);

echo("x = " . $x_sum / $n_set . "\n");
echo("y = " . $y_sum / $n_set . "\n");

I can run it like this:

$ time ./locate-rgb.php ~/pics/x.jpg
x = 483.375
y = 487.75
real    0m0.079s
user    0m0.085s
sys 0m0.022s

So about 80ms on this modest laptop. That includes PHP startup and shutdown, and decompressing the JPG image.

That's only going to work in very constrained lighting and camera setups, but perhaps that's OK? It would be easy to make the ball detection fancier, but of course it would slow it down a bit.

Answer 2

As I said in the comments, it is far from clear what you are trying to do, or why it has to be done in PHP.

It is unclear why you say you are using JPEG, but you give us a PNG but are presumably using video if speed is important to you. The difference between using a PNG and a JPEG and video is probably more than the time required for the processing. The artefacts and loss of colour information resulting from using a JPEG probably make a significant difference.

If you said "I want to acquire video and track a ball at the highest possible frame rate" and expanded on what the constraints are and what you are really doing and how everything is connected and how often things need doing, you would get on a lot better than complaining about the time to generate some ill-defined "vector" by iterating over pixels in PHP which is always going to be a disaster.

Eric suggested you use MMAL for JPEG processing yet your final system probably has no JPEGs involved.

Pending a decent overview of what you actually want to achieve, I guess you actually want to know the centroid of the ball but again you only talk about passing lists of X,Y coordinates around.

Rather than passing massive lists of coordinates around, you can let ImageMagick calculate the moments which will give you the centroid of the ball:

convert garden.jpg \
   -fuzz 10% -fill white -opaque "srgb(98,40,245)" \
   +fuzz     -fill black +opaque white             \
   -colorspace gray -verbose -moments info:

Sample Output

  Format: JPEG (Joint Photographic Experts Group JFIF format)
  Mime type: image/jpeg
  Class: DirectClass

  ...
  ...
  Channel moments:
    Gray:
      Centroid: 897.842,585.948                         <<< CENTROID OF WHITE AREA
      Ellipse Semi-Major/Minor axis: 14.9801,14.3506
      Ellipse angle: 2.17802
      Ellipse eccentricity: 0.205004
      Ellipse intensity: 252.977 (0.992066)
      I1: 0.000629708 (0.160576)
      I2: 7.30093e-10 (4.74743e-05)
      ...
      ...

Note also that, if you explained what you were doing and showed your code, you would also get a better answer. For example, here is the timing for calculating moments in colour

time convert garden.jpg -fill white -fuzz 10% -opaque "srgb(98,40,245)" +fuzz -fill black +opaque white -verbose -moments info: > /dev/null

real    0m0.401s
user    0m4.161s
sys     0m0.050s

but going to greyscale shaves 30% off the time:

time convert garden.jpg -fill white -fuzz 10% -opaque "srgb(98,40,245)" +fuzz -fill black +opaque white -colorspace gray -verbose -moments info: > /dev/null

real    0m0.327s
user    0m3.383s
sys     0m0.058s

And scaling the image by 50% makes an even greater difference, shaving a further 65% off the time:

time convert garden.jpg -scale 50% -fill white -fuzz 10% -opaque "srgb(98,40,245)" +fuzz -fill black +opaque white -colorspace gray -verbose -moments info: > /dev/null

real    0m0.100s
user    0m0.433s
sys     0m0.011s

And that is exactly why you need to be specific and show your code. You may be using resize rather than scale - I don't know.

If Python and OpenCV are the fastest (although you haven't shown your code, so it is not exactly clear what you are doing) you could develop on that idea and improve the speed by exchanging images and data through Redis , but again, it is still unclear what you are really doing. You may currently be needlessly bashing the filesystem waiting for files to appear when you could simply be doing a blocking POP of HSL data and results off Redis. But I can't be sure. If I saw your code, I might suggest acquiring onto a RAMdisk, but again, I can't currently say.