使用 2d bbox 和 plate bbox 预测汽车的表面

Question

I'm trying to solve an interesting problem w/o using GPU intensive model in inference time. (No Deep Learning)

Input: 2D Image which contains car(s) in it, with accurate bboxes, and also a bbox of the plate's car. (We also know that the cameras are located just a bit above the cars)

Output: Surface of the car prediction (the bottom side of a cuboid in 3d bbox)

Approach 1: I'm trying to leverage the fact that I have some prior knowledge except the 2d bbox of the car, but also the 2d bbox of the plate, which can give me the orientation of the car, I thought about taking an angle between the center bbox of the car and the center bbox of the 2d plate to understand what is the direction the car is facing at.

After I know the direction the car is facing to, I also can roughly know where should be one of the edges of the surface because of the fact that the 3d bbox is bounded by the 2d bbox (thus the surface is also bounded), and the fact that the 2d bbox of the plate is a few pixels far from the surface, so one of the edges of the surface can be estimated.

But, the problem here is determining the lateral edges, how 'long' should they be. I'm not quite sure how to estimate the lateral sides of the bottom surface, but I think it can be somehow inferred by the size of the 2d bbox of the car (which again, should bound that surface). Maybe I'll be able to solve it after finding the edge of the surface, and then exploring ways to infer the lateral edges of that surface.

Approach 2: Annotating the data with 3d bboxes with a pre-trained model, and trying to predict the 3d bbox from a 2d bbox (and probably some more priors like 2d bbox of the plate), but I'm not using a deep model to do it, but a simple NN with a few layers to predict the 3d bbox. (trained in a supervised manner)

Answer 1

Using Deep learning-based object detection methods is tend to achieve a really high detection accuracy. Deep neural network is a trend to improve the accuracy of bounding box, designing a reasonable regression loss function is also an important way. So, if you are considering accuracy as an important factor on the project you may need to consider using deep learning.

But if the accuracy doesn't matter that much and you really prefer not to use deep learning then you can use other simple ways.

The conventional 2D object detection yields 4 degrees of freedom (DoF) axis-aligned bounding boxes with center (x, y) and 2D size (w, h), the 3D bounding boxes in autonomous driving context generally have 7 DoF: 3D physical size (w, h, l), 3D center location (x, y, z) and yaw. Note that roll and pitch are normally assumed to be zero. Now the question is, how do we recover a 7-DoF object from a 4-DoF one? You can find a solution and approach explanation based on this research , but it is a little bit complex since it came from a research.

In your 2nd Approach:

" Annotating the data with 3d bboxes with a pre-trained model "

You can try that, then putting all the work for the 3D bbox creation during inference. This is too specific and very complex problem to answer directly, even more without deep learning. But I hope my answer can help a bit.

Here is another approach I can share just in case you want to consider:

You can also train your own model that has different classes for each direction of the car. It actually may take you a lot of time to prepare the dataset for it. Using that model, you can easily detect car direction. By that you may able to let a specific function to create a 3D bbox based on that car-direction detected. Although I cannot recommend this approach if you do not prefer making your own annotated dataset since it really takes a lot of time.

You can use OpenCV for creating the 3D bbox by getting the specific values you'll need from the 2D bbox.

But do take note that it will not provide you the best accuracy. It's still the best way to use Deep Learning instead for better accuracy. You can find a lot of implementation of this in the net.