How to get where 2D texture labels were mapped onto a 3D object

Question

I have texture file that is applied to my complex 3D object (.OBJ file), that texture is filled with varying labels. I want to create additional objects for those unique labels at the location they end up on the 3D object, so the labels can be interactive.

I found a similar post but it uses a few deprecated methods namely Geometry: Three.js - calculate 3D coordinates from UV coordinates

I'm unsure how to translate it to the new version of ThreeJS, or if there's a better way to achieve my goal.

To be more specific about the exact 3D object, it's of human muscle anatomy. And the texture is the labels and color shading for the muscles.

I'm rather new to ThreeJS and 3D modeling so I may be overlooking something obvious.

Answer 1

I don't believe you can make points on your texture interactive, however, have you considered using CSS2DRenderer?

Examples of use below:
css2d_label
molecules

I would recommend creating an element(button) for each label within the DOM or simply refer to the element. Create a new CSS2DObject for each label/button, set the position of each CSS2DObject in relation to your 3D object and add each CSS2DObject to your scene.

const labelOne = document.getElementById('labelOne');
const labelTwo = document.getElementById('labelTwo');

const labelRenderer = new CSS2DRenderer();
labelRenderer.setSize(window.innerWidth, window.innerHeight);
document.body.appendChild(labelRenderer.domElement);

const labelObjOne = new CSS2DObject(labelOne.nativeElement);
labelObjOne.position.set(0, 0, 0);
scene.add(labelObjOne);

const labelObjTwo = new CSS2DObject(labelTwo.nativeElement);
labelObjTwo.position.set(0, 1, 0);
scene.add(labelObjTwo);

With this approach, you may want to remove all current labels from your texture for a nicer finish.

Answer 2

So, after quite a bit of research I found the solution that I was looking for.

The first key step to take was realizing that in my case with the.obj file, everything is stored in parallel UVs, vertices (our faces basically), and normals (I believe this is the case with any ThreeJS object). The only problem was that the UVs and faces are simple 1-dimensional arrays. You have to do a bit more work to organize everything so it's in triplets and you can index it by a face index. Where each index returns all vertices, UVs, and normal that make up one face. As each face is made up of 3 vertices, 3 normals, and 3 UVs. So, once you preprocess it in a way where a face index will return all the relevant data most of the work is done. A rather helpful link to understand the above mentioned is: https://r105.threejsfundamentals.org/threejs/lessons/threejs-custom-buffergeometry.html

The next step (I moved to Python because I found it easier and it's a one-time task) you need to grab all the pixels that make up a single label in your texture. In my case, my labels were all a unique color so I can easily just search for that in a picture/texture and save it accordingly, in my case I used a dictionary where the label was the key and the value was the pixels that made up the label.

The third key realization is that UVs make a triangle, Each face is a triangle. with a texture triangle on it (more or less), But the nice thing is. UVs are in 2D space while faces are in 3D space which is more complicated to deal with, But since we pre-processed our UVs, faces, and normals by a face index. we know already where each UVS space belongs in 3D space without having to map it ourselves.

The next step is to break down the UV triangles*. First you want to convert them to pixel coordinates because we need integers not floats for this to work. Then for each UVs triangle you want to find all points that exist within that triangle (including the points that make up its perimeter). Then you want to store all those points in a dictionary/hash map where each tuple (x, y) will be a key whose value is the index of the face it used to belong to. This takes a while, relatively speaking but increases the speed of the following search A LOT. You only have to run this once, so if you save it to a file, it becomes even faster running it in the future.

Some code I found to help me break down triangles is this: Determining all discrete points inside a triangle and Get all points of a straight line in python

Then we simply need to loop over our label pixels as tuples (x, y) and check the dictionary to find what face they belong to. Due to some rounding errors, there will be some missed pixels that aren't found in the dictionary, but it's not too noticeable and in my use case works great.

Saving those faces you can then import it back to JavaScript. Then you simply create a new Buffer Geometry object, add all the vertices that make up the faces of the label(s) you want and then create the mesh (don't forget to set the color or you won't be able to see it). You'll also want to set the mesh x,y position as well as the scale to match your original object. And boom you have a 3D object over where your label is.

*You can conversely loop through each pixel label and see if it exists within a UVs triangle but that runs in O(n^2) and becomes nearly unfeasible if you have a big texture or complicated object with many vertices. It is more accurate though. But my solution runs in O(n) with a slight hit to accuracy.

In the future I will try and add some minimal working code to my answer here but right now it needs to be cleaned up quite a bit!