Jenny: Building a 3D AI Avatar with Real-Time Lip Sync
Behind the scenes of Jenny — a 3D AI avatar chat app with Blender-authored visuals, text-to-speech, real-time lip sync, and conversational AI powered by a Node.js backend.
Article summary
Behind the scenes of Jenny — a 3D AI avatar chat app with Blender-authored visuals, text-to-speech, real-time lip sync, and conversational AI powered by a Node.js backend.
What this article covers
The Concept Text-based chatbots feel impersonal. I wanted conversations to feel human — a 3D avatar that looks at you, moves its lips in sync with speech, and conveys emotion through facial expressions. That's Jenny .
What this article covers (2)
Avatar Pipeline Jenny's 3D model was created in Blender with blend shapes (morph targets) for: Visemes — mouth shapes for each phoneme group (A, E, I, O, U, consonants). Emotions — happy, sad, surprised, thinking — mapped to expression weights. Idle animations — subtle breathing, blinking, and micro head movements. The model is exported as a .glb with embedded animations and loaded into a React Three Fiber scene.
What this article covers (3)
Real-Time Lip Sync The Node.js backend takes the AI response text, sends it to a TTS engine, and returns an audio buffer along with a phoneme timeline. The frontend steps through the timeline, blending morph targets to match the speech in real time.
Topics and technologies
Related navigation
-
Browse all blog posts
Full engineering archive on the knowledge hub.
-
View delivery case studies
Architecture, implementation, and outcome breakdowns.