NVIDIA Audio2Face is an impressive tool running on Omniverse, the company’s open real-time simulation platform. The technology is capable of applying animations that match surprisingly well to the Digital Mark 3D character model, which is included with an audio file. Whether you’re animating characters in games, movies, real-time digital assistants, or just for fun, the tech is able to animate a 3D character’s animation to suit any voiceover track. The technology does this automatically, works well in most languages, and lets you apply more details.

To do this, NVIDIA has implemented a deep neural network that maps facial animations to audio in real time. When you first start the program, it may take some time to build the Tensor RT Engine, which optimizes the neural network for your hardware. From there, you can see and act on changes in real time as long as your hardware can handle it.

As an added bonus, the company’s educational website features loads of videos for NVIDIA on Demand, Omniverse, and Audio2Face. The first video on Audio2Face, even detailed information on how to export the Process to other tools like Unreal Engine 4 is available here.

The success of the process also seems to depend on the quality of the sound used. Charlie Chaplin’s unusually famous speech in the movie The Great Dictator is featured in a video, and the harekedollarser doesn’t look as clear as it does in the more lively, modern recordings. However, contrary to the sound quality, we can often say that this vehicle will get better over time.

Like it? Share with your friends!

Michael Lewis


Your email address will not be published. Required fields are marked *