How to generate a 3D point cloud from depth image and color image acquired from Matlab

Question

I have 2 set data acquired from Kinect 1- depth image with size 480 640 (uint16) from a scene 2- color image with same size (480 640*3 single) from same scene.

How can I merge these data together to generate a colored 3D point clouds with PLY format in Matlab?

Unfortunately I don't have an access to Kinect anymore and I should use only these data.

Answer 1

I've never tried to do that in matlab, but i think that this is what you are looking for:

http://es.mathworks.com/help/vision/ref/pcfromkinect.html

The tool is inside the Computer Vision System Toolbox™.

Answer 2

Nice question. You should use this tutorial from Burrus - basically you need to use your depth information to convert the color/depth centroids into the 3rd dimension. Be aware that Kinect v1 depth and color streams are slightly mismatched, so take that into account as well.

The tutorial can be found here: http://nicolas.burrus.name/index.php/Research/KinectCalibration You can also use the work of this author: Khoshelham, K., & Elberink, SO (2012) - Accuracy and resolution of Kinect depth data for indoor mapping applications

The matlab code should be something like:

% All formulas and values from:
% Khoshelham, K., & Elberink, S. O. (2012).
% Accuracy and resolution of Kinect depth data for indoor mapping applications.
% Sensors (Basel, Switzerland), 12(2), 1437–54. doi:10.3390/s120201437

load('janFrameThousand.mat')
pc=zeros([size(D) 3]);
W=size(D,2);
H=size(D,1);
f=5.453;
for indWidth = 1:W
    for indHeight= 1:H
        % copy z value
        pc(indHeight,indWidth,3)=D(indHeight,indWidth);
        % calc x value
        pc(indHeight,indWidth,1)=-(pc(indHeight,indWidth,3)/f)*...
            ((indWidth-W/2)*0.0093+0.063);
        % calc y value
        pc(indHeight,indWidth,2)=-(pc(indHeight,indWidth,3)/f)*...
            ((indHeight-H/2)*0.0093+0.039);
    end
end
X=pc(:,:,1);
% X=X(:);
Y=pc(:,:,2);
% Y=Y(:);
Z=-pc(:,:,3);
Z(Z==0)=NaN;

Surface=surf(X,Y,Z,'edgecolor','none','facecolor','interp');
lighting gouraud
camlight
% colormap(repmat(winter,20,1))
axis image
axis vis3d
xlabel('X axis')
ylabel('Y axis')
zlabel('Z axis')

Answer 3

Here is the MATLAB OOP code to convert RGB-D images to point clouds, provided the camera's intrinsic parameters:

% Class definition
classdef rgbd2pointcloud < handle
    
    % Public properties
    properties
        color
        depth
        json_filename
        cam_intrinsics
    end
    % Public methods
    methods
        % Constructor
        function obj = rgbd2pointcloud(color, depth, cam_intrinsics, json_filename)
            if isstring(color)
                obj.color = imread(color);
            else
                obj.color = color;
            end
            if isstring(depth)
                obj.depth = double(imread(depth));
            else
                obj.depth = double(depth);
            end
            obj.json_filename = json_filename;
            obj.cam_intrinsics = cam_intrinsics;
        end
        % Get the XYZ and RGB data
        function [xyz, rgb] = xyz_rgb(obj)
            cam_params = get_camera_intrinsics(obj);
    
            xx = 1 : cam_params.width;
            yy = 1 : cam_params.height;
            [xv, yv] = meshgrid(xx, yy);
                  
            u = reshape(xv', 1, []);
            v = reshape(yv', 1, []);
    
            z = reshape(obj.depth', 1, []) / cam_params.depth_scale;
            x = (u - cam_params.cx) .* z / cam_params.fx;
            y = (v - cam_params.cy) .* z / cam_params.fy;
            
            xyz = [x; y; z]';
            rgb = reshape(pagetranspose(obj.color), [], 3);
        end 
        % Write to the point cloud
        function write2file(obj, xyz, rgb, file_name)
            ptCloud = pointCloud(xyz, Color = rgb);
            pcwrite(ptCloud,file_name, 'Encoding', 'compressed')
        end
    end
    % Private methods
    methods (Access = private)
        % JSON reader
        function cam_params = readJSON(obj)
            fid = fopen(obj.json_filename); 
            raw = fread(fid,inf); 
            str = char(raw'); 
            fclose(fid); 
            
            cam_params = jsondecode(str);
        end
        % Get the camera intrinsics
        function cam_params = get_camera_intrinsics(obj)
            if ~isempty(obj.json_filename)
                cam_params = readJSON;
            else
                cam_params = obj.cam_intrinsics;
            end
        end        
    end
end

Below is the code usage:

%% Inputs
% Read the RGB and D images
color = imread("redwood_847.png");
depth = imread("redwood_847d.png");

% Json filename
json_filename = [];

% Output point cloud filename
file_name = 'output.pcd';

% Camera intrinsics
camera_intrinsic.cx              = 319.5;
camera_intrinsic.cy              = 239.5;
camera_intrinsic.fx              = 525;
camera_intrinsic.fy              = 525;
camera_intrinsic.depth_scale     = 1000; % Depth scale (constant) to convert mm to m vice-versa
camera_intrinsic.width           = 640;
camera_intrinsic.height          = 480;

%% Object callback    
obj = rgbd2pointcloud(color, depth, camera_intrinsic, json_filename);
[xyz, rgb] = obj.xyz_rgb();
obj.write2file(xyz, rgb, file_name)

%% Display the point cloud
figure;
pcshow('output.pcd', 'VerticalAxis', 'x', 'VerticalAxisDir', 'down')
xlabel("X")
ylabel("Y")
zlabel("Z")

I have also uploaded this code to the MATLAB File Exchange