[英]How-to convert an iOS camera image to greyscale using the Accelerate Framework?
It seems like this should be simpler than I'm finding it to be.看起来这应该比我发现的更简单。
I have an AVFoundation
frame coming back in the standard delegate method:我有一个
AVFoundation
框架在标准委托方法中返回:
- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection *)connection
where I would like to convert the frame to greyscale using the Accelerate.Framework
.我想使用
Accelerate.Framework
将帧转换为灰度。
There is a family of conversion methods in the framework, including vImageConvert_RGBA8888toPlanar8()
, which looks like it might be what I would like to see, however, I can't find any examples of how to use them!框架中有一系列转换方法,包括
vImageConvert_RGBA8888toPlanar8()
,这看起来可能是我想看到的,但是,我找不到任何如何使用它们的示例!
So far, I have the code:到目前为止,我有代码:
- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection *)connection
{
@autoreleasepool {
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
/*Lock the image buffer*/
CVPixelBufferLockBaseAddress(imageBuffer,0);
/*Get information about the image*/
uint8_t *baseAddress = (uint8_t *)CVPixelBufferGetBaseAddress(imageBuffer);
size_t width = CVPixelBufferGetWidth(imageBuffer);
size_t height = CVPixelBufferGetHeight(imageBuffer);
size_t stride = CVPixelBufferGetBytesPerRow(imageBuffer);
// vImage In
Pixel_8 *bitmap = (Pixel_8 *)malloc(width * height * sizeof(Pixel_8));
const vImage_Buffer inImage = { bitmap, height, width, stride };
//How can I take this inImage and convert it to greyscale?????
//vImageConvert_RGBA8888toPlanar8()??? Is the correct starting format here??
}
}
So I have two questions: (1) In the code above, is RBGA8888
the correct starting format?所以我有两个问题: (1)在上面的代码中,
RBGA8888
是正确的起始格式吗? (2) How can I actually make the Accelerate.Framework
call to convert to greyscale? (2) 我怎样才能真正使
Accelerate.Framework
调用转换为灰度?
There is an easier option here. 这里有一个更简单的选择。 If you change the camera acquire format to YUV, then you already have a greyscale frame that you can use as you like.
如果您将相机采集格式更改为YUV,那么您已经拥有了可以随意使用的灰度帧。 When setting up your data output, use something like:
设置数据输出时,请使用以下内容:
dataOutput.videoSettings = @{ (id)kCVPixelBufferPixelFormatTypeKey : @(kCVPixelFormatType_420YpCbCr8BiPlanarFullRange) };
You can then access the Y plane in your capture callback using: 然后,您可以使用以下方法访问捕获回调中的Y平面:
CVPixelBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CVPixelBufferLockBaseAddress(pixelBuffer, kCVPixelBufferLock_ReadOnly);
uint8_t *yPlane = CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 0);
... do stuff with your greyscale camera image ...
CVPixelBufferUnlockBaseAddress(pixelBuffer);
The vImage method is to use vImageMatrixMultiply_Planar8
and a 1x3 matrix. vImage方法是使用
vImageMatrixMultiply_Planar8
和1x3矩阵。 vImageConvert_RGBA8888toPlanar8
is the function you use to convert a RGBA8888 buffer into 4 planar buffers. vImageConvert_RGBA8888toPlanar8
是用于将RGBA8888缓冲区转换为4个平面缓冲区的函数。 These are used by vImageMatrixMultiply_Planar8
. 这些由
vImageMatrixMultiply_Planar8
。 vImageMatrixMultiply_ARGB8888
will do it too in one pass, but your gray channel will be interleaved with three other channels in the result. vImageMatrixMultiply_ARGB8888
也会一次性完成,但灰色通道将与结果中的其他三个通道交错。 vImageConvert_RGBA8888toPlanar8
itself doesn't do any math. vImageConvert_RGBA8888toPlanar8
本身不做任何数学运算。 All it does is separate your interleaved image into separate image planes. 它所做的就是将交错图像分离成单独的图像平面。
If you need to adjust the gamma as well, then probably vImageConvert_AnyToAny()
is the easy choice. 如果你还需要调整伽玛,那么可能
vImageConvert_AnyToAny()
是最简单的选择。 It will do the fully color managed conversion from your RGB format to a grayscale colorspace. 它将完成从RGB格式到灰度色彩空间的全色管理转换。 See vImage_Utilities.h.
请参见vImage_Utilities.h。
I like Tarks answer better though. 我喜欢Tarks更好的回答。 It just leaves you in a position of having to color manage the Luminance manually (if you care).
它只是让你处于必须手动着色管理亮度的位置(如果你在意)。
This method is meant to illustrate getting Accelerate's vImage
use in converting BGR images to grayscale. 此方法旨在说明在将BGR图像转换为灰度时使用Accelerate的
vImage
。 Your image may very well be in RGBA format and you'll need to adjust the matrix accordingly, but the camera outputs BGRA so I'm using it here. 您的图像很可能是RGBA格式,您需要相应地调整矩阵,但相机会输出BGRA,所以我在这里使用它。 The values in the matrix are the same values used in OpenCV for cvtColor , there are other values you might play with like luminosity .
矩阵中的值与OpenCV中用于cvtColor的值相同,您可以使用其他值,如发光度 。 I assume you malloc the appropriate amount of memory for the result.
我假设你为结果提供了适当的内存量。 In the case of grayscale it is only 1-channel or 1/4 the memory used for BGRA.
在灰度的情况下,它仅用于BGRA的1通道或1/4的存储器。 If anyone finds issues with this code please leave a comment.
如果有人发现此代码存在问题,请发表评论。
Converting to grayscale in this way may NOT be the fastest. 以这种方式转换为灰度可能不是最快的。 You should check the performance of any method in your environment.
您应该检查环境中任何方法的性能。 Brad Larson's GPUImage might be faster, or even OpenCV's
cvtColor
. Brad Larson的GPUImage可能更快,甚至OpenCV的
cvtColor
。 In any case you will want to remove the calls to malloc and free for the intermediate buffers and manage them for the app lifecycle. 在任何情况下,您都希望删除对malloc的调用,并为中间缓冲区释放,并为应用程序生命周期管理它们。 Otherwise, the function call will be dominated by the malloc and free.
否则,函数调用将由malloc和free控制。 Apple's docs recommend reusing the whole vImage_Buffer when possible.
Apple的文档建议尽可能重用整个vImage_Buffer。
You can also read about solving the same problem with NEON intrinsics . 您还可以阅读有关使用NEON内在函数解决相同问题的信息。
Finally, the fastest method is not converting at all. 最后,最快的方法根本就没有转换。 If you're getting image data from the device camera the device camera is natively in the
kCVPixelFormatType_420YpCbCr8BiPlanarFullRange
format. 如果您从设备摄像头获取图像数据,则设备摄像头本身采用
kCVPixelFormatType_420YpCbCr8BiPlanarFullRange
格式。 Meaning, grabbing the first plane's data (Y-Channel, luma) is the fastest way to get grayscale. 意思是,抓住第一架飞机的数据(Y-Channel,luma)是获得灰度的最快方法。
- (void)convertBGRAFrame:(const CLPBasicVideoFrame &)bgraFrame toGrayscale:(CLPBasicVideoFrame &)grayscaleFrame
{
vImage_Buffer bgraImageBuffer = {
.width = bgraFrame.width,
.height = bgraFrame.height,
.rowBytes = bgraFrame.bytesPerRow,
.data = bgraFrame.rawPixelData
};
void *intermediateBuffer = malloc(bgraFrame.totalBytes);
vImage_Buffer intermediateImageBuffer = {
.width = bgraFrame.width,
.height = bgraFrame.height,
.rowBytes = bgraFrame.bytesPerRow,
.data = intermediateBuffer
};
int32_t divisor = 256;
// int16_t a = (int16_t)roundf(1.0f * divisor);
int16_t r = (int16_t)roundf(0.299f * divisor);
int16_t g = (int16_t)roundf(0.587f * divisor);
int16_t b = (int16_t)roundf(0.114f * divisor);
const int16_t bgrToGray[4 * 4] = { b, 0, 0, 0,
g, 0, 0, 0,
r, 0, 0, 0,
0, 0, 0, 0 };
vImage_Error error;
error = vImageMatrixMultiply_ARGB8888(&bgraImageBuffer, &intermediateImageBuffer, bgrToGray, divisor, NULL, NULL, kvImageNoFlags);
if (error != kvImageNoError) {
NSLog(@"%s, vImage error %zd", __PRETTY_FUNCTION__, error);
}
vImage_Buffer grayscaleImageBuffer = {
.width = grayscaleFrame.width,
.height = grayscaleFrame.height,
.rowBytes = grayscaleFrame.bytesPerRow,
.data = grayscaleFrame.rawPixelData
};
void *scratchBuffer = malloc(grayscaleFrame.totalBytes);
vImage_Buffer scratchImageBuffer = {
.width = grayscaleFrame.width,
.height = grayscaleFrame.height,
.rowBytes = grayscaleFrame.bytesPerRow,
.data = scratchBuffer
};
error = vImageConvert_ARGB8888toPlanar8(&intermediateImageBuffer, &grayscaleImageBuffer, &scratchImageBuffer, &scratchImageBuffer, &scratchImageBuffer, kvImageNoFlags);
if (error != kvImageNoError) {
NSLog(@"%s, vImage error %zd", __PRETTY_FUNCTION__, error);
}
free(intermediateBuffer);
free(scratchBuffer);
}
typedef struct
{
size_t width;
size_t height;
size_t bytesPerRow;
size_t totalBytes;
unsigned long pixelFormat;
void *rawPixelData;
} CLPBasicVideoFrame;
I got through the grayscale conversion, but was having trouble with the quality when I found this book on the web called Instant OpenCV for iOS . 我完成了灰度转换,但是当我在网络上发现这本名为Instant OpenCV for iOS的书时,我遇到了质量问题。 I personally picked up a copy and it has a number of gems, although the code is bit of a mess.
我个人拿了一份副本,它有很多宝石,虽然代码有点混乱。 On the bright-side it is a very reasonably priced eBook.
从好的方面来看,这是一部价格非常合理的电子书。
I'm very curious about that matrix. 我对这个矩阵非常好奇。 I toyed around with it for hours trying to figure out what the arrangement should be.
我玩弄了好几个小时试图弄清楚应该是什么样的安排。 I would have thought the values should be on the diagonal, but the Instant OpenCV guys put it as above.
我本以为这些值应该在对角线上,但是Instant OpenCV的人把它放在上面。
if you need to use BGRA vide streams - you can use this excellent conversion here 如果你需要使用BGRA视频流 - 你可以在这里使用这个出色的转换
This is the function you'll need to take: 这是您需要采取的功能:
void neon_convert (uint8_t * __restrict dest, uint8_t * __restrict src, int numPixels)
{
int i;
uint8x8_t rfac = vdup_n_u8 (77);
uint8x8_t gfac = vdup_n_u8 (151);
uint8x8_t bfac = vdup_n_u8 (28);
int n = numPixels / 8;
// Convert per eight pixels
for (i=0; i < n; ++i)
{
uint16x8_t temp;
uint8x8x4_t rgb = vld4_u8 (src);
uint8x8_t result;
temp = vmull_u8 (rgb.val[0], bfac);
temp = vmlal_u8 (temp,rgb.val[1], gfac);
temp = vmlal_u8 (temp,rgb.val[2], rfac);
result = vshrn_n_u16 (temp, 8);
vst1_u8 (dest, result);
src += 8*4;
dest += 8;
}
}
more optimisations (using assembly) are in the link 更多优化(使用程序集)在链接中
(1) My experience with the iOS camera framework has been with images in the kCMPixelFormat_32BGRA
format, which is compatible with the ARGB8888 family of functions. (1) 我使用 iOS 相机框架的经验是
kCMPixelFormat_32BGRA
格式的图像,它与 ARGB8888 系列函数兼容。 (It may be possible to use other formats as well.) (也可以使用其他格式。)
(2) The simplest way to convert from BGR to grayscale on iOS is to use vImageMatrixMultiply_ARGB8888ToPlanar8()
: https://developer.apple.com/documentation/accelerate/1546979-vimagematrixmultiply_argb8888top (2) 在 iOS 上从 BGR 转换为灰度的最简单方法是使用
vImageMatrixMultiply_ARGB8888ToPlanar8()
: https : //developer.apple.com/documentation/accelerate/1546979-vimagematrixmultiply_argb8888top
Here is a fairly complete example written in Swift.这是一个用 Swift 编写的相当完整的示例。 I'm assuming the Objective-C code would be similar.
我假设 Objective-C 代码是类似的。
guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
// TODO: report error
return
}
// Lock the image buffer
if (kCVReturnSuccess != CVPixelBufferLockBaseAddress(imageBuffer, CVPixelBufferLockFlags.readOnly)) {
// TODO: report error
return
}
defer {
CVPixelBufferUnlockBaseAddress(imageBuffer, CVPixelBufferLockFlags.readOnly)
}
// Create input vImage_Buffer
let baseAddress = CVPixelBufferGetBaseAddress(imageBuffer)
let width = CVPixelBufferGetWidth(imageBuffer)
let height = CVPixelBufferGetHeight(imageBuffer)
let stride = CVPixelBufferGetBytesPerRow(imageBuffer)
var inImage = vImage_Buffer(data: baseAddress, height: UInt(height), width: UInt(width), rowBytes: stride)
// Create output vImage_Buffer
let bitmap = malloc(width * height)
var outImage = vImage_Buffer(data: bitmap, height: UInt(height), width: UInt(width), rowBytes: width)
defer {
// Make sure to free unless the caller is responsible for this
free(bitmap)
}
// Arbitrary divisor to scale coefficients to integer values
let divisor: Int32 = 0x1000
let fDivisor = Float(divisor)
// Rec.709 coefficients
var coefficientsMatrix = [
Int16(0.0722 * fDivisor), // blue
Int16(0.7152 * fDivisor), // green
Int16(0.2126 * fDivisor), // red
0 // alpha
]
// Convert to greyscale
if (kvImageNoError != vImageMatrixMultiply_ARGB8888ToPlanar8(
&inImage, &outImage, &coefficientsMatrix, divisor, nil, 0, vImage_Flags(kvImageNoFlags))) {
// TODO: report error
return
}
The code above was inspired by a tutorial from Apple on grayscale conversion, which can be found at the following link.上面的代码灵感来自 Apple 的灰度转换教程,可在以下链接中找到。 It also includes conversion to a
CGImage
if that is needed.如果需要,它还包括转换为
CGImage
。 Note that they assume RGB order instead of BGR, and they only provide a 3 coefficients instead of 4 (mistake?) https://developer.apple.com/documentation/accelerate/vimage/converting_color_images_to_grayscale请注意,他们假设 RGB 顺序而不是 BGR,并且他们只提供 3 个系数而不是 4 个(错误?) https://developer.apple.com/documentation/accelerate/vimage/converting_color_images_to_grayscale
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.