Knowledge Hub

Jenny: Building a 3D AI Avatar with Real-Time Lip Sync

Engineering articleBy Naveed Sohail Gung2025-08-01AIThree.jsBlenderNode.js

Behind the scenes of Jenny — a 3D AI avatar chat app with Blender-authored visuals, text-to-speech, real-time lip sync, and conversational AI powered by a Node.js backend.

Article summary

Behind the scenes of Jenny — a 3D AI avatar chat app with Blender-authored visuals, text-to-speech, real-time lip sync, and conversational AI powered by a Node.js backend.

What this article covers

The Concept Text-based chatbots feel impersonal. I wanted conversations to feel human — a 3D avatar that looks at you, moves its lips in sync with speech, and conveys emotion through facial expressions. That's Jenny .

What this article covers (2)

Avatar Pipeline Jenny's 3D model was created in Blender with blend shapes (morph targets) for: Visemes — mouth shapes for each phoneme group (A, E, I, O, U, consonants). Emotions — happy, sad, surprised, thinking — mapped to expression weights. Idle animations — subtle breathing, blinking, and micro head movements. The model is exported as a .glb with embedded animations and loaded into a React Three Fiber scene.

What this article covers (3)

Real-Time Lip Sync The Node.js backend takes the AI response text, sends it to a TTS engine, and returns an audio buffer along with a phoneme timeline. The frontend steps through the timeline, blending morph targets to match the speech in real time.

Topics and technologies

  • AI
    Browse related articles in the knowledge hub.
  • Three.js
    Browse related articles in the knowledge hub.
  • Blender
    Browse related articles in the knowledge hub.
  • Node.js
    Browse related articles in the knowledge hub.
  • MongoDB
    Browse related articles in the knowledge hub.

Related navigation