C++ - Ray Casting in with DirectX11

Question

I'm having lots of troubles with my ray casting calculations and can't quite figure out where the problem is. I'm using DirectX's Math library to create vectors and the such.

Here is my code so far:

auto normalizedDeviceCoords = glm::vec2((float(a_frontend->GetMousePos().x) / float(a_frontend->GetWidth())) * 2.f - 1.f, 
                                        -((float(a_frontend->GetMousePos().y) / float(a_frontend->GetHeight())) * 2.f - 1.f));
auto mouseOrigin = DirectX::XMVectorSet(normalizedDeviceCoords.x, normalizedDeviceCoords.y, 0.0f, 1.0f);
auto mouseEnd = DirectX::XMVectorSet(normalizedDeviceCoords.x, normalizedDeviceCoords.y, 1.0f, 1.0f);

auto viewproj = DirectX::XMMatrixMultiply(a_frontend->GetViewMatrix(), a_frontend->GetProjMatrix());
auto determinant = DirectX::XMMatrixDeterminant(a_camera.ViewProj());
auto inverseviewproj = DirectX::XMMatrixInverse(&determinant, a_camera.ViewProj());
auto rayOrigin = DirectX::XMVector4Transform(mouseOrigin, inverseviewproj);
auto rayEnd = DirectX::XMVector4Transform(mouseEnd, inverseviewproj);
auto raySubtraction = DirectX::XMVectorSubtract(rayEnd, rayOrigin);
auto rayDirection = DirectX::XMVector3Normalize(raySubtraction);

auto planeNormal = DirectX::XMVectorSet(0.f, 1.f, 0.f, 0.f);
auto pointOnPlane = DirectX::XMVectorSet(0.f, -0.1f, 0.f, 0.f);

DirectX::XMFLOAT3 denominator;
DirectX::XMStoreFloat3(&denominator, DirectX::XMVector3Dot(planeNormal, rayDirection));
if (fabs(denominator.x) <= 0.0001f)
{
    return;
}

auto pointMinusRay = DirectX::XMVectorSubtract(pointOnPlane, rayOrigin);

DirectX::XMFLOAT3 t;
auto almostT = DirectX::XMVector3Dot(pointMinusRay, planeNormal);
DirectX::XMStoreFloat3(&t, DirectX::XMVectorScale(almostT, 1.f / denominator.x));

if (t.x < 0)
{
    return;
}

auto rayDirectionLength = DirectX::XMVector3Length(rayDirection);
auto rayPoint = DirectX::XMVectorAdd(rayOrigin, DirectX::XMVectorScale(rayDirection, t.x));

I have checked that the normalized device coordinates are correct, the ray direction's length is also one, and the ray direction appears to be correct (hard to tell by just the numbers).

Any info will be useful. I've already looked at some information here on stack overflow and on other forums also and haven't gotten anywhere.

Answer 1

My team mate was able to solve this problem. He just started using the DirectX Math function Vector3Unproject. This will take the mouse coordinates and turn them into world coordinates, using both the origin and the end (so taking the mouse coordinates with az of 0 and then of 1). Then subtracting those from one another to get a vector that points somewhere in the world. We were then able to take that direction and transform it according to the camera's current y position divided by the negative of the direction vector's y. Here is all the code that we have now:

DirectX::XMVECTOR mouseNear = DirectX::XMVectorSet((float)a_mousePos.x, (float)a_mousePos.y, 0.0f, 0.0f);
DirectX::XMVECTOR mouseFar = DirectX::XMVectorSet((float)a_mousePos.x, (float)a_mousePos.y, 1.0f, 0.0f);
DirectX::XMVECTOR unprojectedNear = DirectX::XMVector3Unproject(mouseNear, 0, 0, a_width, a_height, a_nearZ, a_farZ,
                                                                a_projection, a_view, DirectX::XMMatrixIdentity());
DirectX::XMVECTOR unprojectedFar = DirectX::XMVector3Unproject(mouseFar, 0, 0, a_width, a_height, a_nearZ, a_farZ,
                                                                a_projection, a_view, DirectX::XMMatrixIdentity());
DirectX::XMVECTOR result = DirectX::XMVector3Normalize(DirectX::XMVectorSubtract(unprojectedFar, unprojectedNear));
DirectX::XMFLOAT3 direction;
DirectX::XMStoreFloat3(&direction, result);
return direction;

//Get the distance to the ground.
DirectX::XMFLOAT3 cameraPosition = a_camera.GetPosition();

//Get the point on the ground.
cameraPosition.x += direction.x * (cameraPosition.y / -direction.y);
cameraPosition.z += direction.z * (cameraPosition.y / -direction.y);

Hopefully this may prove helpful for someone in the future.