How do I analyze video stream on iOS?

Question

For example, there are QR scanners which scan video stream in real time and get QR codes info. I would like to check the light source from the video, if it is on or off, it is quite powerful so it is no problem.

I will probably take a video stream as input, maybe make images of it and analyze images or stream in real time for presence of light source (maybe number of pixels of certain color on the image?)

How do I approach this problem? Maybe there is some source of library?

Answer 1

It sounds like you are asking for information about several discreet steps. There are a multitude of ways to do each of them and if you get stuck on any individual step it would be a good idea to post a question about it individually.

1: Get video Frame

Like chaitanya.varanasi said, AVFoundation Framework is the best way of getting access to an video frame on IOS. If you want something less flexible and quicker try looking at open CV's video capture . The goal of this step is to get access to a pixel buffer from the camera. If you have trouble with this, ask about it specifically.

2: Put pixel buffer into OpenCV

This part is really easy. If you get it from openCV's video capture you are already done. If you get it from an AVFoundation you will need to put it into openCV like this

//Buffer is of type CVImageBufferRef, which is what AVFoundation should be giving you
//I assume it is BGRA or RGBA formatted, if it isn't, change CV_8UC4 to the appropriate format

CVPixelBufferLockBaseAddress( Buffer, 0 );

int bufferWidth = CVPixelBufferGetWidth(Buffer);
int bufferHeight = CVPixelBufferGetHeight(Buffer);

unsigned char *pixel = (unsigned char *)CVPixelBufferGetBaseAddress(Buffer);
cv::Mat image = cv::Mat(bufferHeight,bufferWidth,CV_8UC4,pixel); //put buffer in open cv, no memory copied

//Process image Here

//End processing
CVPixelBufferUnlockBaseAddress( pixelBuffer, 0 );

note I am assuming you plan to do this in OpenCV since you used its tag. Also I assume you can get the OpenCV framework to link to your project. If that is an issue, ask a specific question about it.

3: Process Image

This part is by far the most open ended. All you have said about your problem is that you are trying to detect a strong light source. One very quick and easy way of doing that would be to detect the mean pixel value in a greyscale image. If you get the image in colour you can convert with cvtColor . Then just call Avg on it to get the mean value. Hopefully you can tell if the light is on by how that value fluctuates.

chaitanya.varanasi suggested another option, you should check it out too.

openCV is a very large library that can do a wide wide variety of things. Without knowing more about your problem I don't know what else to tell you.

Answer 2

Look at the AVFoundation Framework from Apple.

Hope it helps!

You can try this method: start by getting all images to an AVCaptureVideoDataOutput . From the method: captureOutput:didOutputSampleBuffer:fromConnection ,you can sample/calculate every pixel. Source: answer

Also, you can take a look at this SO question where they check if a pixel is black. If its such a powerful light source, you can take the inverse of the pixel and then determine using a set threshold for black.

Answer 3

The above sample code only provides access to the pixel values stored in the buffer; you cannot run any other commands but those that change those values on a pixel-by-pixel basis:

for ( uint32_t y = 0; y < height; y++ )
{
    for ( uint32_t x = 0; x < width; x++ )
    {
        bgraImage.at<cv::Vec<uint8_t,4> >(y,x)[1] = 0;
    }
}

This—to use your example—will not work with the code you provided:

    cv::Mat bgraImage = cv::Mat( (int)height, (int)extendedWidth, CV_8UC4, base );
cv::Mat grey = bgraImage.clone();
cv::cvtColor(grey, grey, 44);