NVIDIA Nemotron 3.5 Transcribe and Granary v2: A Multilingual Speech Model with Advanced Features
NVIDIA has released an updated version of its multilingual speech model, Nemotron 3.5 Transcribe, which combines transcription, speaker attribution, translation, and audio question answering into a single checkpoint. This open-weight model is now available on Hugging Face and ModelScope for developers to use in their applications.
The new model can transcribe 40 languages, translate 31 of them to or from English, and attribute speakers in 13 languages. It also handles recordings up to an hour long and labels up to eight speakers. This makes it a versatile tool for various industries such as healthcare, education, and customer service.
Nemotron 3.5 Transcribe has achieved top leaderboard results on several benchmarks, including the Hugging Face Open ASR Leaderboard, where it ranks first among open-weight models in English transcription accuracy. It also tops two speaker-attribution benchmarks outright, demonstrating its ability to accurately identify speakers in multilingual conversations.
The model’s performance is impressive, with an average word error rate (WER) of 5.33% on the Hugging Face Open ASR Leaderboard for English transcription. This means that it can transcribe speech with a high degree of accuracy, even in complex and noisy environments.
Nemotron 3.5 Transcribe also excels in speaker attribution, posting the lowest error rate on David AI’s 21-language diarization leaderboard. It ranks first in VoiceArena’s 14-system comparison for English full-session benchmarking, demonstrating its ability to accurately identify speakers even when there is overlap between them.
The model’s architecture combines a speech encoder with a hybrid Mamba-Transformer mixture-of-experts language model, allowing it to process and understand complex linguistic structures. This enables Nemotron 3.5 Transcribe to handle a wide range of languages and dialects, making it a valuable tool for multilingual applications.
Nemotron 3.5 Transcribe is designed to be flexible and adaptable, allowing developers to fine-tune the model to their specific needs using the NeMo Speech fine-tuning tools. This includes adapting recognition to domain vocabulary and transcript conventions, ensuring that the model can accurately transcribe speech in various contexts.
The Granary v2 data pipeline plays a crucial role in training Nemotron 3.5 Transcribe, providing aligned transcript variants, task labels, and inspectable quality metadata for multilingual speech data. This allows developers to create high-quality training datasets tailored to their specific needs.
Granary v2 is an open data curation pipeline that prepares multilingual speech used in Nemotron 3.5 Transcribe’s training. It produces three transcript forms for each recording, including verbatim and clean styles, which can be requested from the model with a prompt. This enables developers to customize the output of the model to suit their specific requirements.
The Granary v2 pipeline is organized into four reusable stages: Audio Preparation (Metadata), Transcription (ASR), Task Label Generation (Text), and High Fidelity (HiFi). Each stage processes raw or long-form audio, tracks speakers across the full recording, and tags acoustic events and language. This ensures that the training data is accurate and reliable.
Developers can run the NeMo Curator pipeline with Ray and NVIDIA Dynamo backends on their own recordings or authorized public corpora, or build on the Granary v2 public manifests. The manifests include the three transcript forms as tn_raw, itn_raw, and itn_clean, along with labels described above, entity hints, derived instruction pairs, provenance, and a per-record quality flag.
Nemotron 3.5 Transcribe can be fine-tuned using the NeMo Speech medical ASR case study, which demonstrates its ability to adapt recognition to domain vocabulary and transcript conventions. The case study shows that fine-tuning cuts held-out WER from 22.44% for the best-prompt baseline to 12.17%, a 45.8% relative reduction.
The adapted model’s performance is impressive, with a word error rate (WER) of 12.17% on the Eka Care’s Indian medical speech benchmark. This means that it can accurately transcribe complex and nuanced linguistic structures in various contexts, making it a valuable tool for multilingual applications.
Nemotron 3.5 Transcribe has been tested on several benchmarks, including the Hugging Face Open ASR Leaderboard, David AI DAI-ASR-I18N Leaderboard, and VoiceArena Diarization-Bench. Its performance is impressive across all these benchmarks, demonstrating its ability to accurately transcribe speech in various languages and contexts.
Nemotron 3.5 Transcribe can be used for a wide range of applications, including customer service, healthcare, education, and more. It can also be fine-tuned using the NeMo Speech fine-tuning tools to adapt recognition to domain vocabulary and transcript conventions, ensuring that it accurately transcribes speech in various contexts.
Nemotron 3.5 Transcribe is available on Hugging Face and ModelScope for developers to use in their applications. Developers can try Nemotron 3.5 Transcribe in the Hugging Face demo or on ModelScope Studios, then download the checkpoint from Hugging Face or ModelScope and serve it with the vLLM deployment guide.
Developers can also adapt Nemotron 3.5 Transcribe to their specific needs using the Granary v2 manifests on Hugging Face or ModelScope and the medical ASR case study. Alternatively, they can let a coding agent drive the workflow with the fine-tuning agent skill.
Discussion
No messages yet. Be the first to write.