Jenny — 3D AI Avatar Chat

Problem context

Text-based chatbots lack personality. Video avatars are expensive to render server-side. The challenge was building a 3D avatar that runs entirely in the browser, syncs lip movement to speech audio with sub-50ms accuracy, and conveys emotion through facial expressions — all while maintaining 60 FPS on consumer hardware.

Implementation approach

Created a detailed 3D avatar in Blender with blend shapes (morph targets) for visemes (mouth shapes for phoneme groups), emotions (happy, sad, surprised, thinking), and idle animations (breathing, blinking, head micro-movements).

Implementation approach (2)

Built a Node.js backend that takes AI response text, sends it to a TTS engine, and returns an audio buffer alongside a phoneme timeline with precise timestamps.

Outcome and delivery impact

Jenny achieves audio-visual sync within 30ms — imperceptible to users. The exponential smoothing on morph target weights produces natural mouth movement that reviewers described as "uncannily lifelike."

Outcome and delivery impact (2)

The 3D model with full animations and morph targets loads in 1.8 seconds. Mobile performance stays above 30 FPS even with real-time lip sync active, thanks to LOD switching and efficient morph target batching.

Reported metrics

• Lip Sync Accuracy
  Reported value: < 30ms
• Model Load Time
  Reported value: 1.8s
• Blended Emotions
  Reported value: 7
• Mobile Performance
  Reported value: 30+ FPS

Technologies involved

• React
  Open the technology archive to view related project work.
• Three.js
  Open the technology archive to view related project work.
• Blender
  Open the technology archive to view related project work.
• Node.js
  Open the technology archive to view related project work.
• MongoDB
  Open the technology archive to view related project work.
• Web Audio API
  Open the technology archive to view related project work.
• TTS
  Open the technology archive to view related project work.