@AIatMeta: Today we're sharing the next m...

Today we're sharing the next milestone in our Seamless Communication research — a new family of AI translation models that preserve expression and deliver near-real time streaming translations.

More on this new work ➡️ https://t.co/KNZCEEPk9v

More on the individual models 🧵 https://t.co/ZzCIR4GBe2

SeamlessM4T v2 is a foundational multilingual & multitask model for both speech & text. It's the successor to SeamlessM4T, demonstrating performance improvements across ASR, speech-to-speech, speech-to-text & text-to-speech tasks.

Evaluated across all tasks and languages through a collection of automatic metrics, it significantly outperformed previous state-of-the-art models.

SeamlessExpressive enables the transfer of tones, emotional expression and vocal styles in speech translation. It incorporates an expressivity encoder and expressive unit-to-speech generator conditioned on source speech to deliver translations that maintain the unique nuances of the original speaker.

You can try a demo of SeamlessExpressive now using your own voice as an input ➡️ https://t.co/04ChruNiF3

SeamlessStreaming is a new model that enables streaming speech-to-speech and speech-to-text translations with <2 seconds of latency. To deliver stronger results and adapt to differences in language structures, it intelligently decides when it has enough context to output the next translated segment. It does this through a learned read/write policy which enables stronger performance across many different language pairs.

Combining the strength of these three models, we're also introducing Seamless, a unified model that merges the quality and multilinguality of SeamlessM4T v2, the low latency of SeamlessStreaming and the expression preservation of Seamless expressive into one unified system. https://t.co/93PnlK7J3j

In addition to these models, we're sharing new datasets, watermarking research & papers on this work ➡️ https://t.co/GoJXI7ILV5

We believe open science is crucial to AI translation that benefits everyone & we look forward to continuing this work with the research community.