Magic Leap collaborating with NVIDIA to advance digital twins for enterprise

In Augmented Reality News 

April 12, 2023 – Magic Leap has announced that it is collaborating with NVIDIA to pioneer immersive, photorealistic, ray-traced digital twins in NVIDIA’s Omniverse platform, in order to create new efficiencies and value in engineering, architecture, retail, and other industries.

Magic Leap began its collaboration with NVIDIA to find a way to enable off-board computing and streaming of 1:1 scale 3D digital models to its Magic Leap 2 augmented reality (AR) headset. According to Magic Leap, the collaboration allows enterprise users to render and stream immersive, full-scale digital twins from the NVIDIA Omniverse platform to Magic Leap 2 without compromised visual quality and includes an extension integrating common enterprise modeling programs into the UI.

Technical advancements that the companies have worked on include: 

• Ray tracing and Deep Learning Super Sampling (DLSS) that enables content to be rendered with highly detailed, photorealistic quality;
• Magic Leap Dynamic Dimming technology, which eliminates 99.7% of ambient light from the display to make AR usable in a greater variety of bright lighting conditions;
• Accurate environmental lighting and reflections across digital content;
• Depth-based reprojection capability that eliminates latency and anchors digital content persistently within a user’s environment.

To help make design and 3D modeling more efficient, and to empower enterprises and professionals who create, design, and collaborate to render physically accurate, photorealistic digital twins on the Magic Leap 2 device, Magic Leap created an Omniverse extension that renders content to the headset from the most common enterprise modeling tools, such as Autodesk Maya, without the need for conversions or optimizations. Content is stored in Omniverse Nucleus as USD files. In Nucleus, users can live edit and collaborate on the content.

By simply clicking “View on Magic Leap”, an image is rendered on the server and streamed to Magic Leap 2, where it is instantly displayed for the user in the correct position, orientation, and scale. Omniverse’s core XR extension will automatically render the high-quality stereo images and submit them to the industry-standard OpenXR API.

The server-side OpenXR remote rendering runtime then leverages NVIDIA GPU acceleration to composite and encode the stereo images. These images are then transmitted over the network where they are received, decoded, and displayed on Magic Leap 2.

Magic Leap 2 is also constantly sending head poses, eye tracking, controller state, and other input data back to the server where it is surfaced to Omniverse through OpenXR. On the back end, other users can continue to live edit the USD content with changes instantly rendered and streamed to Magic Leap 2.

Magic Leap stated that by leveraging a number of Magic Leap 2 features and NVIDIA RTX GPUs, the solution helps to mitigate common challenges, including:

1. Reprojection mitigates latency and locks content in place – Magic Leap 2 can tolerate more than 100 milliseconds of end-to-end latency while maintaining an immersive experience for the user;
2. NVIDIA RTX ray tracing and DLSS enable photorealistic quality;
3. Dynamic Dimming technology makes content opaque and adds behaviorally-realistic shadows;
4. Accurate environmental lighting and reflections ensure immersion. The company added that it is actively researching ways to make this functionality even more realistic through dynamic lighting estimation.

Magic Leap noted that it will continue its collaboration with NVIDIA, and will build upon its advancements in digital twin technology to deliver greater value to enterprise users. With more immersive and accessible digital twins, teams can accelerate design, deepen collaboration, and transform design reviews and other experiences.

Magic Leap’s integration with Omniverse is currently in early access, with an expected general availability release in the first half of this year. To find out more about working with Magic Leap to deploy digital twins across enterprise and business, click here.

Image credit: Magic Leap