How to solve camera rotation along the X-axis?

Question

I have an Opengl ray-tracer, which is capable of loading obj files and visualize it in a ray-tracing way. My application loads the obj file with assimp and then sends all of the triangle faces (including the vertices and their indices) to the fragment shader by using shader storage buffer. After that, the fragment shader calculates the color of the pixel by tracing the path of light rays. The light rays are coming from the position of camera ( posCamera ) and going throught the pixels of a virtual canvas. The basic structure is about to render the results to this quad shaped canvas.

At the moment, I am working on the camera. The camera is always looking at the 0,0,0 point ( viewPoint ) and can be rotated around the y-axis by this function below:

void setCamera(float param) {
    connect = posCamera - viewPoint;

    posCamera = glm::vec3(connect.x * cos(param) + connect.z * sin(param), connect.y,
                          -connect.x * sin(param) + connect.z * cos(param)) + viewPoint;

    canvasX = cross(upVector, connect) * getLength(connect) * tanf(fieldOfview / 2);
    canvasY = cross(connect, canvasX) * getLength(connect) * tanf(fieldOfview / 2);

    setCam(posCamera,viewPoint, canvasY, fieldOfview);

I just want to rotate the scene around the x-axis as well. To solve this, I am created a new function:

void rotateAroundX(float param) {
    connect = posCamera - viewPoint;
    posCamera = glm::vec3(posCamera.x,
                          connect.y * cos(param) + connect.z * sin(param),
                          -connect.y * sin(param) + connect.z * cos(param))+viewPoint;
    canvasX = cross(upVector, connect) * getLength(connect) * tanf(fieldOfview / 2);
    canvasY = cross(connect, canvasX) * getLength(connect) * tanf(fieldOfview / 2);

    setCam(posCamera,viewPoint, canvasY, fieldOfview);
}

It is working, but unfortunately there is some distortion connected to the ratio. What is more, sometimes the camera just circulating around a point. Here is a video connected to the problem: link . You can see that the horizontal rotation doing fine.

Here is my vertex shader:

#version 460 core
layout(location = 0) in vec2 normQuadCoord;

uniform vec3 viewPoint;
uniform vec3 canvasX, canvasY;
out vec3 pixel;

void main()
{
    pixel = canvasX * normQuadCoord.x + canvasY * normQuadCoord.y + viewPoint;
    gl_Position = vec4(normQuadCoord, 0, 1);
}

and here is a link to the full source code: source

Beside the camera rotation, I don't know, if a free flying camera is possible with this 'rendering to quad' concept. Any help is appreciated!

---

Update 1

I tried Rabbid76's advice and modified the source. I made some refactorization as well. The method, which updates the rectangle canvas dimensions can be seen below:

void Init::updateCanvasSizes() {
    connect = camera.getPosCamera() - camera.getViewPoint();

    float aspect = (float)SCR_W_H.first/(float)SCR_W_H.second;
    float length = tanf(camera.getFieldOfview() / 2);

    canvasX = glm::normalize(glm::cross(camera.upVector, connect)) / length / aspect;
    canvasY = glm::normalize(glm::cross(connect, canvasX)) / length;

}

The method, which handles the camera rotation around the X-axis are modified to this:

void Init::rotateCamAroundX(float param) {
    camera.posCamera = glm::vec3(camera.posCamera.x,
               (camera.posCamera.y - camera.viewPoint.y) * cos(param) + (camera.posCamera.z - camera.viewPoint.z) * sin(param) + camera.viewPoint.y,
               -(camera.posCamera.y - camera.viewPoint.y) * sin(param) + (camera.posCamera.z - camera.viewPoint.z) * cos(param) + camera.viewPoint.z);

    updateCanvasSizes();
}

The field of view have been modified from 45 degrees to 45 * (float)M_PI / 180 and as a result the ratio of the model became a bit better. Unfortunately, during the camera rotation around X-axis, scene stretching and distortion can be still found. When the camera reaches a certain degree, it also turns back to look at the camera.viewPoint . I don't know how to "turn it off".

The results can be seen in this video: link . Firstly, the rotation around Y-axis, and then the rotation around X-axis can be seen.

Answer 1

You have to respect the aspect ratio of the viewport.

The aspect ratio is taken into account by canvasX and canvasY when calculating the point for the direction of the ray.

p = pixel = canvasX * normQuadCoord.x + canvasY * normQuadCoord.y + viewPoint;

If the viewport is rectangular, the magnitude of the vectors must be different. canvasX is the vector from the center to the right and canvasY is the vector from the center to the top of the viewport. In order to compensate for the rectangularity of the view and to achieve a correct projection, the ratio of the vector lengths must be equal to the reciprocal aspect ratio:

 width     | canvasY |
-------- = -----------
 height    | canvasX |

Therefore the length of canvasX has to be scaled by the reciprocal aspect ratio. The aspect ratio is the quotient of the width and height of the view( (float)SCR_WIDTH/(float)SCR_HEIGHT ):

float aspect = (float)SCR_WIDTH/(float)SCR_HEIGHT;
float length = tanf(fieldOfview / 2);

canvasX = glm::normalize(glm::cross(upVector, connect)) / length / aspect;
canvasY = glm::normalize(glm::cross(connect, canvasX)) / length;