Showing highlighted silhouette of mesh ONLY when it is occluded

Question

so I'm trying to create an online game using Babylon.js but have run into a problem thats got me a little stumped so hoping someone here would be willing to help me out. Please bear with me on this one, i'm a complete newbie with babylon as i've only every worked with THREE.js. Right now my game consists of a scene compromising of multiple meshes with multiple users represented as avatars (created from basic circle geometry for the moment) loaded into an environment. What I want to do is highlight the outline of these avatars ONLY when they are occluded by any other object, meaning that when they are not occluded they look normal with no highlight but when behind an object their highlighted silhouette can be seen by others (including yourself as you can see your own avatar). This is very akin to effects used in many other video games (see example below).

Example of Effect

Thus far, based on some googling and forum browsing ( Babylonjs outline through walls & https://forum.babylonjs.com/t/highlight-through-objects/8002/4 ) I've figured out how to highlight the outline of objects using Babylon.HighlighLayer and I know that i can render objects above others via RenderingGroups but I can't seem to figure out how to use them in conjunction to create the effect I want. The best i've managed to do is get the highlighted avatar render above everything but I need just the silhouette not the entire mesh. I'm also constrained by the fact that my scene has many meshes in it that are loaded dynamically and i'm also trying to keep things as optimal as possible. Can't afford to use very computationally expensive procedures.

Anybody know of the best way to approach this? Would greatly appreciate any advice or assistance you can provide.Thanks!

Answer 1

So I asked the same question on the babylon forums which helped me to find a solution. All credit goes to the guy's that helped me out over there but just in case someone else comes across this question seeking an answer, here is a link to that forum question https://forum.babylonjs.com/t/showing-highlighted-silhouette-of-mesh-only-when-it-is-occluded/27783/7

Edit: Ok thought i'd include the two possible solutions here properly as well as their babylon playgrounds. All credit goes to roland & evgeni_popov who came up with these solutions on the forum linked above.

The first solution is easier to implement but slightly less performant than the second solution.

Clone Solution: https://playground.babylonjs.com/#JXYGLT%235

// roland@babylonjs.xyz, 2022

const createScene = function () {
const scene = new BABYLON.Scene(engine);

const camera = new BABYLON.ArcRotateCamera('camera', -Math.PI / 2, Math.PI / 2, 20, new BABYLON.Vector3(0, 0, 0), scene)
camera.attachControl(canvas, true);

const light = new BABYLON.HemisphericLight("light", new BABYLON.Vector3(0, 1, 0), scene);
light.intensity = 0.7;

const wall = BABYLON.MeshBuilder.CreateBox('wall', { width: 5, height: 5, depth: 0.5 }, scene)
wall.position.y = 1
wall.position.z = -2

const sphere = BABYLON.MeshBuilder.CreateSphere('sphere', { diameter: 2, segments: 32 }, scene)
sphere.position.y = 1

const sphereClone = sphere.clone('sphereClone')
sphereClone.setEnabled(false)

const matc = new BABYLON.StandardMaterial("matc", scene);
matc.depthFunction = BABYLON.Constants.ALWAYS;
matc.disableColorWrite = true;
matc.disableDepthWrite = true;

sphereClone.material = matc;

sphere.occlusionQueryAlgorithmType = BABYLON.AbstractMesh.OCCLUSION_ALGORITHM_TYPE_ACCURATE
sphere.occlusionType = BABYLON.AbstractMesh.OCCLUSION_TYPE_STRICT

const hl = new BABYLON.HighlightLayer('hl1', scene, { camera: camera })
hl.addMesh(sphereClone, BABYLON.Color3.Green())
hl.addExcludedMesh(wall);

let t = 0;

scene.onBeforeRenderObservable.add(() => {
    sphere.position.x = 10 * Math.cos(t);
    sphere.position.z = 100 + 104 * Math.sin(t);

    if (sphere.isOccluded) {
        sphereClone.setEnabled(true)
        sphereClone.position.copyFrom(sphere.position);
    } else {
        sphereClone.setEnabled(false)
    }

    t += 0.03;
})

return scene;
};

This second solution is slightly more performant than above as you don't need a clone but involves overriding the AbstactMesh._checkOcclusionQuery function which is the function that updates the isOccluded property for meshes such that the mesh is always rendered even when occluded. There's no overhead if you are using the occlusion queries only for the purpose of drawing silhouettes however If you are also using them to avoid drawing occluded meshes then there's an overhead because the meshes will be drawn even if they are occluded. In which case your probably best of going with the first solution

Non-Clone solution: https://playground.babylonjs.com/#JXYGLT#14

// roland@babylonjs.xyz, 2022

const createScene = function () {
const scene = new BABYLON.Scene(engine);

const camera = new BABYLON.ArcRotateCamera('camera', -Math.PI / 2, Math.PI / 2, 20, new BABYLON.Vector3(0, 0, 0), scene)
camera.attachControl(canvas, true);

const light = new BABYLON.HemisphericLight("light", new BABYLON.Vector3(0, 1, 0), scene);
light.intensity = 0.7;

const wall = BABYLON.MeshBuilder.CreateBox('wall', { width: 5, height: 5, depth: 0.5 }, scene)
wall.position.y = 1
wall.position.z = -2

const sphere = BABYLON.MeshBuilder.CreateSphere('sphere', { diameter: 2, segments: 32 }, scene)
sphere.position.y = 1

sphere.occlusionQueryAlgorithmType = BABYLON.AbstractMesh.OCCLUSION_ALGORITHM_TYPE_ACCURATE
sphere.occlusionType = BABYLON.AbstractMesh.OCCLUSION_TYPE_STRICT

const mats = new BABYLON.StandardMaterial("mats", scene);
sphere.material = mats;

const hl = new BABYLON.HighlightLayer('hl1', scene, { camera: camera })
hl.addExcludedMesh(wall);

let t = 0;

const cur = BABYLON.AbstractMesh.prototype._checkOcclusionQuery;

scene.onDisposeObservable.add(() => {
    BABYLON.AbstractMesh.prototype._checkOcclusionQuery = cur;
});

BABYLON.AbstractMesh.prototype._checkOcclusionQuery = function() {
    cur.apply(this);
    return false;
}

scene.onBeforeRenderObservable.add(() => {
    sphere.position.x = 10 * Math.cos(t);
    sphere.position.z = 100 + 104 * Math.sin(t);

    if (sphere.isOccluded) {
        hl.addMesh(sphere, BABYLON.Color3.Green())
        mats.depthFunction = BABYLON.Constants.ALWAYS;
        mats.disableColorWrite = true;
    } else {
        hl.removeMesh(sphere);
        mats.depthFunction = BABYLON.Constants.LESS;
        mats.disableColorWrite = false;
    }

    t += 0.03;
})

return scene;
};