如何将 rgb 值关联到透视投影中的像素位置？

Question

I am trying to associate rgb values to pixel coordinates after having done a perspective projection. The equation for the perspective projection is:

where x , y , are the pixel locations of the point, X , Y , and Z are locations of points in the camera frame, and the other parameters denote the intrinsic camera parameters. Given a point cloud containing the point locations and rgb values, I would like to associate rgb values to pixel locations according to the perspective projection.

The following code should create the correct image:

import matplotlib.pyplot as plt
import open3d as o3d
import numpy as np
cx = 325.5;
cy = 253.5;
fx = 518.0;
fy = 519.0;
K = np.array([[fx, 0, cx], [0, fy, cy], [0, 0, 1]])
pcd = o3d.io.read_point_cloud('freiburg.pcd', remove_nan_points=True)
points = np.array(pcd.points)
colors = np.array(pcd.colors)
projection = (K @ points.T).T
normalization = projection / projection[:, [2]] #last elemet must be 1
pixel_coordinates = normalization.astype(int)
img = np.zeros((480, 640, 3))
#how can I fill the img appropriately? The matrix pixel coordinates should
# inform about where to place the color intensities.

for position, intensity in zip(pixel_coordinates, colors):
    row, column = position[0], position[1]
    #img[row, column, :] = intensity # returns with error 
    img[column, row, :] = intensity # gives a strange picture.

The point cloud can be read here . I expect to be able to associate the rgb values in the last loop:

for position, intensity in zip(pixel_coordinates, colors):
    row, column = position[0], position[1]
    #img[row, column, :] = intensity # returns with error 
    img[column, row, :] = intensity # gives a strange picture.

Strangely, if the second-to-last line is not commented, the program returns and IndexError while attempting to write a rgb values outside the range of available columns. The last line in the loop runs however without problems. The generated picture and the correct picture can be seen below:

How can I modify the code above to obtain the correct image?

Answer 1

A couple of issues:

• You are ignoring the nonlinear distortion in the projection. Are the images you are comparing to undistorted? If they are, are you sure your projection matrix K is the one associated to the undistorted image?
• Projecting the 3D points will inevitably produce a point cloud on the image plane, not a continuous image. To produce an image somewhat natural you likely need to interpolate nearby samples in the 2D point cloud. Your choice of interpolation filter determines the quality of the result. For example, you could first make an image of rgb buckets, a similar image of weights, project the 3d points, place their rgb values in the closest bucket (the one obtained by rounding the projection x,y coords), with a weight equal to the reciprocal of the distance of the projection from the bucket's center (ie the reciprocal of the euclidean norm of the rounding residuals). You then first compute the output pixel values as weighted averages at each bucket and then, if there are any unfilled bucket, you fill them by (say) bilinear interpolation of the filled neighbors. The last step will fill 1-pixel holes surrounded by already filled values. For larger holes you will need to choose some kind of infill procedure.