Nvidia Releases Open-Source Voice AI Model PersonaPlex-7B

Artificial intelligence researchers have long sought to create voice assistants that can engage in natural-sounding conversations, but the current state of technology often falls short. The biggest annoyance is waiting for a machine to finish speaking before responding, which disrupts the flow of conversation and makes interactions feel stilted. Nvidia has released PersonaPlex-7B, an open-source voice model designed to address this issue by allowing simultaneous listening and speaking.

Nvidia’s PersonaPlex-7B uses a dual-stream design, where one stream continuously processes incoming audio from the user while another generates the agent’s speech and text in real-time. This approach eliminates the need for handoffs between separate systems, reducing latency and enabling full-duplex communication like phone calls or walkie-talkies.

The model is built on top of Moshi’s dual-stream Transformer architecture, which was developed by Kyutai, a French lab. Nvidia fine-tuned and extended this architecture to create PersonaPlex-7B, resulting in a 90.8% turn-taking rate and interruption response latency as low as 240 milliseconds.

PersonaPlex-7B is not the first full-duplex speech AI model, but it’s one of the few that have been made available for public use. The company reports impressive benchmarking results, including a time-to-first-token of roughly 170 milliseconds and a turn-taking rate of 90.8%. These numbers indicate that PersonaPlex-7B can respond quickly to interruptions and adapt to changing conversation dynamics.

The model’s architecture allows it to be fine-tuned for specific applications, such as customer service or tutoring. Before a conversation starts, PersonaPlex takes two prompts: one sets vocal characteristics and speaking style based on audio tokens, while the other defines the persona itself, its role, background, and scenario. This enables the same underlying model to be used in different contexts without retraining.

Nvidia’s decision to release PersonaPlex-7B as an open-source project has significant implications for the industry. By making a working full-duplex speech AI available for free, Nvidia is setting a public benchmark that other companies must answer. The company’s move also commoditizes the layer just below its actual product, encouraging more developers to build on top of it and potentially leading to increased demand for Nvidia GPUs.

PersonaPlex-7B has been released under an MIT license, with weights available through Hugging Face and code hosted on GitHub. This allows anyone to use and modify the model, which is a significant departure from the typical approach of keeping full-duplex speech AI behind API paywalls or demo videos.

Nvidia’s investment in Reflection AI, a startup that developed the 501-billion-parameter Beam model, has been reported to be worth $800 million. The company also provides chip supply deals worth billions through 2029. This partnership highlights Nvidia’s commitment to advancing voice AI technology and its willingness to invest in companies working on related projects.

Nvidia’s move into open-source voice AI is not without precedent. Chip and software companies have long used this strategy to create a foundation for their products, encouraging more developers to build on top of it and increasing demand for their hardware. By releasing PersonaPlex-7B as an open-source project, Nvidia is following in the footsteps of these companies.

Nvidia’s release of PersonaPlex-7B marks a significant milestone in voice AI research, offering a working full-duplex speech model that can be used by developers and researchers worldwide. The company’s decision to make this technology available for free sets a new standard for the industry, challenging other companies to create similar models with comparable performance.