Explore how creative technologist Robert Lester collaborated with NYC's Technodramatists to create a groundbreaking live AR theater experience. Discover the custom real-time facial motion capture system and interactive 3D content built in Unreal Engine for this innovative adaptation of Shakespeare's Comedy of Errors. Learn how the traditional Italian masked theater of Commedia dell'arte influenced the relationship between AR projections and live performance.
Actor Claire Tyers fearlessly portrayed more than 6 different characters using Augmented Reality to capture and project multiple masks to expand on what is possible with live theater technology. Facial animation data was captured with a latest generation iPhone Xs. Claire was able to control character switching with a Nintendo Switch controller installed on the performance rig that was reprogrammed to communicate with our Unreal Engine software.
A short breakdown video showcasing the different masks that were used and some of the considerations that went into our interactive development.
Claire Tyers expressive performance was amplified with face-tracking and live animation with Unreal Engine and custom made CG masks.
Facial motion capture was executed with a custom app installed on an iPhone Xs. Our software took advantage of new functionality introduced by Epic Games ARKit integration, and their incredibly helpful Face AR Sample Project, which is freely available on Epic’s site. Many thanks to Epic Games for the work that went into this integration and execution.
Overview of the Unreal Engine project layout (It’s a pretty simple layout actually!).
Each mask was calibrated via custom blueprint logic that was used to tweak & amplify the mask performance to the actor’s specific expressive facial qualities.
We tracked more than 50 facial data points to drive animated blendshapes in UE4. A combination of gyroscope and tracking data were used to recreate head and neck rotation values.
Claire Tyers gave more than 10 public performances of the 1st 4 scenes from the play. The system was nimble enough that under her control she could control the pacing and flow of every performance, allowing for improvisation whims and breaking the 4th wall.