简体繁体 English

Android中的多线程实时摄像头处理

[英]Multithreaded real time camera processing in Android

原文 2014-03-31 07:38:08 4 1 java/ android/ multithreading/ video-processing

The goal is to get real-time image processing working on the live preview of an Android device. 目标是在Android设备的实时预览中实现实时图像处理。 This has been tackled here on SO many times, and has some mentions on the web (see references at the end of the question), however, I haven't found a single suggestion on a proper multithreaded framework to accomplish this. 这已经在这里解决了很多次，并且在Web上有一些提及（请参阅问题末尾的参考资料），但是，我还没有找到关于实现此目标的适当多线程框架的任何建议。

To my understanding, to make proper use of multithreading (and devices with more than one physical core) one would implement the following: 据我了解，要正确使用多线程（以及具有多个物理核心的设备），应实现以下目标：

The frame handler is called on the UI thread ( onCameraPreview ) and enqueues the frame data to a shared queue Q . 在UI线程（ onCameraPreview ）上调用帧处理程序，并将帧数据排队到共享队列Q 。
A thread pool of n threads awaits on Q . n线程的线程池在Q上等待。 Each thread dequeus a frame whenever available, does the "heavy" image processing and posts a message to the UI thread with the result. 每个线程都会在可能的情况下出帧一个帧，进行“大量”图像处理，并将消息与结果一起发送到UI线程。

My main question is whether the above is the right direction, and how exactly to accomplish it on Android. 我的主要问题是上述方法是否正确，以及如何在Android上准确实现。 Also, what would be the required modifications if the image processing had to be done in serial - ie no more than 1 frame being processed at any given time but still have the processing done on a thread separate than the main one. 同样，如果必须串行执行图像处理，即在任何给定时间处理的帧数不超过一帧，但仍然在与主线程分开的线程上进行处理，将需要进行哪些修改。

Update 1: 更新1：
I ended up implementing this as an AsyncTask that recieves the raw frame data as input and uses the onPostExecute callback to update the UI. 我最终将其实现为一个AsyncTask ，它接收原始帧数据作为输入，并使用onPostExecute回调来更新UI。 However, this happens about 10 times per second which makes me wonder whether it is not generating too much overhead. 但是，这种情况大约每秒发生10次，这使我怀疑它是否不会产生太多开销。 This is why, as far as I'm concerned, this question is still open to get a validation this is indeed the most efficient method. 这就是为什么就我而言，这个问题仍未得到验证，这确实是最有效的方法。 Also, it is still unclear to me how one would expand this approach to multiple worker threads on multiple cores. 同样，我仍然不清楚如何将这种方法扩展到多个内核上的多个工作线程。

And links: 和链接：
http://ibuzzlog.blogspot.co.il/2012/08/how-to-do-real-time-image-processing-in.html http://ibuzzlog.blogspot.co.il/2012/08/how-to-do-real-time-image-processing-in.html

1 个解决方案

Android documentation states that the call given to "onPreviewFrame()" with a byte[] array cannot be put in a queue or something because the array is reused after each call. Android文档指出，使用byte []数组对“ onPreviewFrame（）”进行的调用无法放入队列或其他内容，因为该数组在每次调用后都会被重用。 The call has to be returned as soon as possible to attain better framerates. 必须尽快返回该呼叫以获得更好的帧率。

So here is what you can do (I done it and got 29 FPS) 这就是您可以做的（我做到了，得到了29 FPS）

Pass the array to a Native c or c++ library through JNI call 通过JNI调用将数组传递给本机c或c ++库
Implement a queue in your C program and malloc a node to store this array 在您的C程序中实现一个队列，并分配一个节点来存储此数组
Memory copy the array and return the call. 内存复制数组并返回调用。
And Return the call 并回电
En queue the node for processing and de queue nodes to process with some encoders such as FFMPEG 将节点排入队列进行处理，将节点排入队列以使用某些编码器（例如FFMPEG）进行处理
Run this Queuing process and Encoding process parallel so that after encoding a frame you can clear that from memory to prevent memory overflow. 并行运行此排队过程和编码过程，以便在对帧进行编码后，可以从内存中清除该帧，以防止内存溢出。
And remember to choose the optimal resolution so that your encoder matches the speed of input framerate so that there is no memory overflow due to queue. 并记住选择最佳分辨率，以使编码器与输入帧速率的速度相匹配，从而不会因队列而导致内存溢出。