Avatar Pupettering with ThreeJS, ReadyPlayerMe, Kalidokit and MediaPipe

Question

I try to animate a ReadyPlayer Me avatar using ThreeJS and Kalidokit (or something else) with MediaPipe Hollisitc Pose . Here is a working JSFiddle:

https://jsfiddle.net/nxg5bp2h/1/

It almost works, but the code for Pose Estimation is all messed-up. I use the one inspired from the scripts code of Kalidokit but arms seems reversed and the legs going up...

let  riggedPose = Kalidokit.Pose.solve(poselm3D, poselm,{runtime:'mediapipe',video: 
$('#mediapipe-video')[0]})
if (!riggedPose) { return; }

this.rigRotation('Hips', riggedPose.Hips.rotation, 0.7)
this.rigPosition('Hips', {
    x: -riggedPose.Hips.position.x, // Reverse direction
    y:  riggedPose.Hips.position.y + 1, // Add a bit of height
    z: -riggedPose.Hips.position.z // Reverse direction
}, 1, 0.07)

this.rigRotation('Chest', riggedPose.Spine, 0.25, 0.3)
this.rigRotation('Spine', riggedPose.Spine, 0.45, 0.3)

this.rigRotation('RightUpperArm', riggedPose.RightUpperArm, 1, 0.3)
this.rigRotation('RightLowerArm', riggedPose.RightLowerArm, 1, 0.3)
this.rigRotation('LeftUpperArm',  riggedPose.LeftUpperArm,  1, 0.3)
this.rigRotation('LeftLowerArm',  riggedPose.LeftLowerArm,  1, 0.3)

this.rigRotation('LeftUpperLeg',  riggedPose.LeftUpperLeg,  1, 0.3)
this.rigRotation('LeftLowerLeg',  riggedPose.LeftLowerLeg,  1, 0.3)
this.rigRotation('RightUpperLeg', riggedPose.RightUpperLeg, 1, 0.3)
this.rigRotation('RightLowerLeg', riggedPose.RightLowerLeg, 1, 0.3)

If someone can help me fix it, spot the error or to do the math? I'm a little bit lost to understand where/why rotation are incorrect. May be all the RPM models are rigged another way?!

Thanks !

Answer 1

I found that if you swap the y and z rotations then you get a better result... it's still a bit unstable but it's at least in the right proximity.

let euler = new this.THREE.Euler(rotation.x * dampener, rotation.z * dampener, rotation.y * dampener)