[英]Save depth images from TrueDepth camera
我正在嘗試從iPhoneX TrueDepth相機保存深度圖像。 使用AVCamPhotoFilter示例代碼,我能夠在手機屏幕上實時查看轉換為灰度格式的深度。 我無法弄清楚如何以原始(16位或更多)格式保存深度圖像序列。
我有depthData
這是一個實例AVDepthData
。 它的一個成員的是depthDataMap
這是一個實例CVPixelBuffer
和圖像格式類型kCVPixelFormatType_DisparityFloat16
。 有沒有辦法將其保存到手機轉移以進行離線操作?
“原始”深度/差異地圖沒有標准的視頻格式,這可能與AVCapture沒有真正提供記錄它的方法有關。
你有幾個值得調查的選擇:
將深度貼圖轉換為灰度貼圖(您可以使用AVCamPhotoFilter示例代碼中的代碼執行此操作),然后將這些貼圖傳遞到AVAssetWriter
以生成灰度視頻。 根據您選擇的視頻格式和灰度轉換方法,您為讀取視頻而編寫的其他軟件可能能夠從灰度幀中以足夠的精度恢復深度/差異信息。
只要你有CVPixelBuffer
,你就可以自己獲取數據並隨心所欲地做任何事情。 使用CVPixelBufferLockBaseAddress
(帶有readOnly
標志)確保在讀取內容時不會更改內容,然后將指針CVPixelBufferGetBaseAddress
提供的數據復制到您想要的任何位置。 (使用其他像素緩沖區函數來查看要復制的字節數,並在完成后解鎖緩沖區。)
但請注意:如果您花費太多時間從緩沖區復制或以其他方式保留它們,它們將不會被釋放,因為新的緩沖區從捕獲系統進入,並且您的捕獲會話將掛起。 (總而言之,如果沒有測試設備是否具有內存和I / O帶寬以進行大量錄制,那么目前還不清楚。)
您可以使用Compression庫創建包含原始CVPixelBuffer數據的zip文件。 這個解決方案幾乎沒有問題。
inflateInit2(&strm, -15);
使它工作。 這是我的代碼(我將其限制在250幀,因為它將其保存在RAM中,但如果需要更多幀,您可以刷新到磁盤):
// DepthCapture.swift
// AVCamPhotoFilter
//
// Created by Eyal Fink on 07/04/2018.
// Copyright © 2018 Resonai. All rights reserved.
//
// Capture the depth pixelBuffer into a compress file.
// This is very hacky and there are lots of TODOs but instead we need to replace
// it with a much better compression (video compression)....
import AVFoundation
import Foundation
import Compression
class DepthCapture {
let kErrorDomain = "DepthCapture"
let maxNumberOfFrame = 250
lazy var bufferSize = 640 * 480 * 2 * maxNumberOfFrame // maxNumberOfFrame frames
var dstBuffer: UnsafeMutablePointer<UInt8>?
var frameCount: Int64 = 0
var outputURL: URL?
var compresserPtr: UnsafeMutablePointer<compression_stream>?
var file: FileHandle?
// All operations handling the compresser oobjects are done on the
// porcessingQ so they will happen sequentially
var processingQ = DispatchQueue(label: "compression",
qos: .userInteractive)
func reset() {
frameCount = 0
outputURL = nil
if self.compresserPtr != nil {
//free(compresserPtr!.pointee.dst_ptr)
compression_stream_destroy(self.compresserPtr!)
self.compresserPtr = nil
}
if self.file != nil {
self.file!.closeFile()
self.file = nil
}
}
func prepareForRecording() {
reset()
// Create the output zip file, remove old one if exists
let documentsPath = NSSearchPathForDirectoriesInDomains(.documentDirectory, .userDomainMask, true)[0] as NSString
self.outputURL = URL(fileURLWithPath: documentsPath.appendingPathComponent("Depth"))
FileManager.default.createFile(atPath: self.outputURL!.path, contents: nil, attributes: nil)
self.file = FileHandle(forUpdatingAtPath: self.outputURL!.path)
if self.file == nil {
NSLog("Cannot create file at: \(self.outputURL!.path)")
return
}
// Init the compression object
compresserPtr = UnsafeMutablePointer<compression_stream>.allocate(capacity: 1)
compression_stream_init(compresserPtr!, COMPRESSION_STREAM_ENCODE, COMPRESSION_ZLIB)
dstBuffer = UnsafeMutablePointer<UInt8>.allocate(capacity: bufferSize)
compresserPtr!.pointee.dst_ptr = dstBuffer!
//defer { free(bufferPtr) }
compresserPtr!.pointee.dst_size = bufferSize
}
func flush() {
//let data = Data(bytesNoCopy: compresserPtr!.pointee.dst_ptr, count: bufferSize, deallocator: .none)
let nBytes = bufferSize - compresserPtr!.pointee.dst_size
print("Writing \(nBytes)")
let data = Data(bytesNoCopy: dstBuffer!, count: nBytes, deallocator: .none)
self.file?.write(data)
}
func startRecording() throws {
processingQ.async {
self.prepareForRecording()
}
}
func addPixelBuffers(pixelBuffer: CVPixelBuffer) {
processingQ.async {
if self.frameCount >= self.maxNumberOfFrame {
// TODO now!! flush when needed!!!
print("MAXED OUT")
return
}
CVPixelBufferLockBaseAddress(pixelBuffer, .readOnly)
let add : UnsafeMutableRawPointer = CVPixelBufferGetBaseAddress(pixelBuffer)!
self.compresserPtr!.pointee.src_ptr = UnsafePointer<UInt8>(add.assumingMemoryBound(to: UInt8.self))
let height = CVPixelBufferGetHeight(pixelBuffer)
self.compresserPtr!.pointee.src_size = CVPixelBufferGetBytesPerRow(pixelBuffer) * height
let flags = Int32(0)
let compression_status = compression_stream_process(self.compresserPtr!, flags)
if compression_status != COMPRESSION_STATUS_OK {
NSLog("Buffer compression retured: \(compression_status)")
return
}
if self.compresserPtr!.pointee.src_size != 0 {
NSLog("Compression lib didn't eat all data: \(compression_status)")
return
}
CVPixelBufferUnlockBaseAddress(pixelBuffer, .readOnly)
// TODO(eyal): flush when needed!!!
self.frameCount += 1
print("handled \(self.frameCount) buffers")
}
}
func finishRecording(success: @escaping ((URL) -> Void)) throws {
processingQ.async {
let flags = Int32(COMPRESSION_STREAM_FINALIZE.rawValue)
self.compresserPtr!.pointee.src_size = 0
//compresserPtr!.pointee.src_ptr = UnsafePointer<UInt8>(0)
let compression_status = compression_stream_process(self.compresserPtr!, flags)
if compression_status != COMPRESSION_STATUS_END {
NSLog("ERROR: Finish failed. compression retured: \(compression_status)")
return
}
self.flush()
DispatchQueue.main.sync {
success(self.outputURL!)
}
self.reset()
}
}
}
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.