Experience Type: Virtual Production for Still Image (Collaborative Project)
Platform: Still Image Format
Game Engine: Unreal Engine 5.2.1
Additional Tools: Blender 3.6
The above prototype is the culmination of a collaborative effort between technicians from the Photography and Immersive Technology departments at the Edinburgh College of Art. The team comprised of Chris Ambrose and Shareen Sorour, who were responsible for building and staging the physical set and Pradyumna Panikker for developing the virtual 3D environment and compositing the video footage with the virtual backdrop in Unreal Engine.
With the growing interest in Virtual Production across universities, the intent of the collaboration was to develop a simple but efficient workflow that students and staff can adopt to build setups using affordable equipment while being presentable.
Unreal Engine was used to develop the virtual environment and blend real-world footage with a virtual backdrop.
IDEATION & PRE-PRODUCTION
"Finding common ground among different visual media."
The collaboration began as a discussion on emerging technologies and how creatives in Immersive Media and Lens-based Technology can collaborate to develop accessible setups and workflows for use in the teaching space.
"Using an alternate, cost-effective method for development."
Due to the unavailability of an LED wall and the necessary supporting hardware and technology to run a high-end ICVFX setup, a cheaper alternative was adopted with real-time green screen compositing. Using a Cam-Link device, the technology feeds the camera's video signal in real-time into Unreal Engine to be composited with the virtual backdrop.
The above visual depicts a still from a preliminary test session of the real-time green screen composting workflow using Unreal Engine.
Following the successful implementation of the technology, the next phase involved finalizing the general design and aesthetic of the scene to demonstrate the workflow's feasibility in creating presentable compositions.
"Selecting a suitable reference to develop the composition."
Following a quick discussion on the general atmosphere of the composition, a still from the Stanley Kubrick classic Eyes Wide Shut was chosen as the primary reference.
The above still from Eyes Wide Shut by Stanley Kubrick was chosen for its atmosphere, featuring contrasting lighting temperatures between the foreground and background.
The final composition featured a similar aesthetic with certain creative liberty taken to make the setup unique and presentable with the available technology and materials.
DEVELOPMENT
"Using Blender to prototype the scene's lighting."
The developmental phase began with the prototyping of the setup's lighting design using Blender. The software's real-time lighting visualization features allowed us to iterate the position and brightness of the light sources, giving us a clear idea of how physical lights and virtual lights were to be staged to get the desired effect.
The above visual depicts a low-fidelity recreation of the reference. Blender provided a blueprint to recreate the lighting and atmosphere in Unreal Engine.
"The final stage and lighting setup was an improvisational and iterative process."
Using the reference and the prototype created in Blender, the physical and virtual set were developed simultaneously with the intent of effectively blending the physical set with the virtual environment through rapid iteration.
The above visual is a timelapse depicting the various stages of recreating the foreground's stage lighting. Simultaneously, the virtual background lighting and environment was developed, mirroring the setup of the foreground.
The following timelapse depicts the final composited footage as seen from Unreal Engine. Once the virtual lighting reached a presentable level of quality, further refinements to mesh and material detailing were made in Unreal Engine.
The above visual depicts the set with a screen displaying the composited shot in Unreal Engine.
Video data from the camera was streamed in real-time as a 2D media plate in Unreal Engine. The media plate was then rendered on a 2D plane matching the aspect ratio of the footage following which, the footage was strategically positioned in the virtual environment to match the perspective and scale of the physical scene with the virtual backdrop.
PEER RECEPTION AND FURTHER DEVELOPMENT
The prototype was presented to upper management of Information, Technical and Technology Services and members of the CoStar (Convergent Screen Technologies and Performance in Realtime) project as a proof of concept to integrate Virtual Production practices and workflows into the teaching curriculum.
The prototype successfully kickstarted the introduction of a voluntary project involving students across various filmmaking, sound design and 3D design courses across the University of Edinburgh in which participants are required to produce and deliver a short film using real-time green screen compositing in Unreal Engine.
REFERENCES
Kubrick, S. (1999) Eyes Wide Shut. Warner Bros., United States.