Introducing ColQwen-Omni, a 3B omnimodal retriever that extends the...

@ManuelFaysse
Manuel Faysse@ManuelFaysse
42 views Jul 17, 2025 ~1 min read
Advertisement
1
Introducing ColQwen-Omni, a 3B omnimodal retriever that extends the ColPali concept of multimodal retrieval with late interaction to audio chunks and short videos, with no performance degradation on visual document retrieval wrt our best models! (1/N)
Media image
2
The model is trained solely on visual document retrieval. Based on the Qwen-Omni backbone, it is able to generalize its newly learned document embedding capabilities to embed audios and short videos without ever having seen those during contrastive training! (2/N)
1:07
3
We can thus do "Retrieval in Audio Space". It only takes 10s to embed 30 minutes of audio, and we can find the most relevant chunks based on spoken content, but also ambient sounds, emotions, etc... (3/N)
4
This is orders of magnitude faster than indexing based on STT transcriptions, and better captures non-verbal cues / sounds. We wrote a blogpost with code examples, and even showcased a fun end-to-end AudioRAG demo! (4/N)
huggingface.co/blog/manu/colq…
5
This is a v0, and as we specifically train to improve audio retrieval, performance will improve. Eventually, we aim to investigate cross-modality performance transfer and create a great all-around retriever with @antonio_loison @pteiletche + Quentin. (5/N)
6
Thanks to the Qwen team for a great backbone model, adapting it for retrieval was super quick to do but it's hard to find the time :/ For those interested @xueguang_ma also trained a single-vector version to check out ! (6/N)
huggingface.co/vidore/colqwen…
7
@xueguang_ma 🔊 For those that hate clicking links, here's a quick video demo of end-to-end AudioRAG! Not prod-ready but very fun !
1:47
Actions
What You Can Do
  • Export as PDF or Markdown
  • Batch Export to Notion
  • Bookmark & Highlight
  • Screenshot Tweet
Create Free Account

Includes 7-day Premium trial

Advertisement