Introducing ColQwen-Omni, a 3B omnimodal retriever that extends the ColPali concept of multimodal retrieval with late interaction to audio chunks and short videos, with no performance degradation on visual document retrieval wrt our best models! (1/N)
2
The model is trained solely on visual document retrieval. Based on the Qwen-Omni backbone, it is able to generalize its newly learned document embedding capabilities to embed audios and short videos without ever having seen those during contrastive training! (2/N)
1:07
3
We can thus do "Retrieval in Audio Space". It only takes 10s to embed 30 minutes of audio, and we can find the most relevant chunks based on spoken content, but also ambient sounds, emotions, etc... (3/N)
4
This is orders of magnitude faster than indexing based on STT transcriptions, and better captures non-verbal cues / sounds. We wrote a blogpost with code examples, and even showcased a fun end-to-end AudioRAG demo! (4/N) huggingface.co/blog/manu/colq…
5
This is a v0, and as we specifically train to improve audio retrieval, performance will improve. Eventually, we aim to investigate cross-modality performance transfer and create a great all-around retriever with @antonio_loison @pteiletche + Quentin. (5/N)
6
Thanks to the Qwen team for a great backbone model, adapting it for retrieval was super quick to do but it's hard to find the time :/ For those interested @xueguang_ma also trained a single-vector version to check out ! (6/N) huggingface.co/vidore/colqwen…
7
@xueguang_ma 🔊 For those that hate clicking links, here's a quick video demo of end-to-end AudioRAG! Not prod-ready but very fun !
1:47
Save this thread — create a free accountSave this thread