OpenGL/GLM - Switching coordinate system direction

Question

I am converting some old software to support OpenGL. DirectX and OpenGL have different coordinate systems (OpenGL is right, DirectX is left). I know that in the old fixed pipeline functionality, I would use:

glScalef(1.0f, 1.0f, -1.0f);

This time around, I am working with GLM and shaders and need a compatible solution. I have tried multiplying my camera matrix by a scaling vector with no luck.

Here is my camera set up:

// Calculate the direction, right and up vectors
 direction = glm::vec3(cos(anglePitch) * sin(angleYaw), sin(anglePitch), cos(anglePitch) * cos(angleYaw));
 right = glm::vec3(sin(angleYaw - 3.14f/2.0f), 0, cos(angleYaw - 3.14f/2.0f));
 up = glm::cross(right, direction);

 // Update our camera matrix, projection matrix and combine them into my view matrix
 cameraMatrix = glm::lookAt(position, position+direction, up);
 projectionMatrix = glm::perspective(50.0f, 4.0f / 3.0f, 0.1f, 1000.f);
 viewMatrix = projectionMatrix * cameraMatrix;

I have tried a number of things including reversing the vectors and reversing the z coordinate in the shader. I have also tried multiplying by the inverse of the various matrices and vectors and multiplying the camera matrix by a scaling vector.

Answer 1

Don't think about the handedness that much. It's true, they use different conventions, but you can just choose not to use them and it boils down to almost the same thing in both APIs. My advice is to use the exact same matrices and setups in both APIs except for these two things:

All you should need to do to port from DX to GL is:

1. Reverse the cull-face winding - DX culls counter-clockwise by default, while that's what GL keeps.
2. Adjust for the different depth range: DX uses a depth range of 0(near) to 1(far), while GL uses a signed range from -1 for near and 1 for far. You can just do this as "a last step" in the projection matrix.

DX9 also has issues with pixel-coordinate offsets, but that's something else entirely and it's no longer an issue with DX10 onward.

From what you describe, the winding is probably your problem, since you are using the GLM functions to generate matrices that should be alright for OpenGL.