Dong Zhang
Sophia-in-Audition project teaser

Sophia-in-Audition

Sophia-in-Audition: Virtual Production with a Robot Performer

We present Sophia-in-Audition (SiA), a new frontier in virtual production, by employing the humanoid robot Sophia within an UltraStage environment composed of a controllable lighting dome coupled with multiple cameras.

Robot Performer

Sophia-in-Audition employs the humanoid robot Sophia within UltraStage, a controllable lighting dome coupled with multiple cameras. The SiA dataset records her facial expressions, costumes, and performances under side, butterfly, top, and other cinematic lighting setups.

Sophia actuator regions and facial expression samples
SiA dataset of costumes, expressions, and cinematic lighting setups

Identity Augmentation

A dual-track video generation method separates appearance and motion. Selected keyframes are processed with a diffusion model under textual guidance, while EBSynth preserves the motion, expressions, lighting, and temporal continuity of Sophia's original performance.

Identity augmentation pipeline separating appearance and motion
Identity augmentation results across generated video sequences

Cinematic Lighting

UltraStage reproduces cinematic lighting references around the performer. The resulting sequences combine generated identities with immersive and dynamic illumination, replicate iconic film performances, and extend the scene through simulated lightning and virtual camera movement.

Generated identities under lighting reconstructed from film references
Dynamic lighting results across Sophia and generated identities
Film performance reference and replicated robot performance
Sophia and a generated identity under simulated lightning
Generated identity along a virtual camera movement path

Audience Study

User studies reported positive reception toward Sophia's performances. Dynamic clips were rated highly for natural lighting and for their alignment with professional film standards.