In a MTKView How to get pixel information ( RGB and Alpha values ) of a given point in 3D (x,y,z)?

Question

Ok we have draw several things in a MTKView We can move and turn around them using Metal, MetalKit functions but we are unable to get pixel information of a given point in 3D (x,y,z). We searched several hours and could not find any solution for that.

Answer 1

This seems like a classic case of trying to consult your view (in the Model-View-Controller paradigm sense of the term) for information held by the model. Consult your model directly.

A 3D rendering technology like Metal is all about flattening 3D information to 2D and efficiently throwing away information not relevant to that 2D representation. Neither Metal nor the MTKView has any information about points in 3D by the end of the render.

Also note, your model probably doesn't have information for arbitrary 3D points. Most likely all of your 3D models (in the other sense of the term) are shells. They have no interiors, just surfaces. So, unless a 3D point falls exactly on the surface and not at all inside nor outside of the model, it has no color.

Answer 2

impossible to get color from 3D model space, but possible to get color from 2D view space.

func getColor(x: CGFloat, y: CGFloat) -> UIColor? {

    if let curDrawable = self.currentDrawable {
        var pixel: [CUnsignedChar] = [0, 0, 0, 0]  // bgra

        let textureScale = CGFloat(curDrawable.texture.width) / self.bounds.width
        let bytesPerRow = curDrawable.texture.width * 4
        let y = self.bounds.height - y
        curDrawable.texture.getBytes(&pixel, bytesPerRow: bytesPerRow, from: MTLRegionMake2D(Int(x * textureScale), Int(y * textureScale), 1, 1), mipmapLevel: 0)
        let red: CGFloat   = CGFloat(pixel[2]) / 255.0
        let green: CGFloat = CGFloat(pixel[1]) / 255.0
        let blue: CGFloat  = CGFloat(pixel[0]) / 255.0
        let alpha: CGFloat = CGFloat(pixel[3]) / 255.0
        let color = UIColor(red:red, green: green, blue:blue, alpha:alpha)
        return color
    }
    return nil 
}