Introducing Cambrian-S it’s a position, a dataset, a benchmark, and...
Saining Xie@sainingxie
47 views
Nov 10, 2025
~7 min read
Advertisement
1
Introducing Cambrian-S
it’s a position, a dataset, a benchmark, and a model
but above all, it represents our first steps toward exploring spatial supersensing in video. 🧶
it’s a position, a dataset, a benchmark, and a model
but above all, it represents our first steps toward exploring spatial supersensing in video. 🧶
3:30
2
last year, we built Cambrian-1, an open exploration of multimodal models for images. instead of just scaling up to Cambrian-2 or 3, we paused to ask:
- what does true multimodal intelligence mean?
- does the LLM paradigm even make sense for sensory modeling?
- and why is human sensing so effortless, so intuitive, yet so powerful?
something fundamental was missing
and you really can’t build superintelligence without first building supersensing.
so, what is supersensing?
supersensing in our context isn’t about fancy sensors or better cameras.
it’s about how a digital being truly experiences the world, absorbing endless streams of input and learning from them.
supersensing is part of intelligence
just like eyes are the part of the brain that touches the outside world.
you don't need sensing to solve coding and math.
but AI agents in the real world need sensory modeling.
or maybe as @karpathy said, that’s all it’ll ever need.
okay, enough philosophy.
to be more concrete, we think the following taxonomy makes sense and captures how things evolve from what we have now to what we actually need to build next.
0. (linguistic-only understanding): no sensory capabilities; reasoning confined to text and symbols. current MLLMs have progressed beyond this stage, yet still retain traces of its bias.
1. semantic perception: parsing pixels into objects, attributes, and relations. this corresponds to the strong multimodal "show and tell" capabilities present in MLLMs.
2. streaming event cognition: processing live, unbounded streams while proactively interpreting and responding to ongoing events. this aligns with current efforts to make MLLMs real-time assistants.
3. implicit 3D spatial cognition: understanding video as projections of a 3D world. agents must know what is present, where, how things relate, and how configurations change over time. today’s multimodal models remain VERY limited here.
4. predictive world modeling: the brain makes “unconscious inferences” by predicting latent world states based on prior expectations. Current multimodal systems don’t have an internal model that can anticipate future states, maintain persistent memory, or reason and plan.
to study all this, video is the ultimate medium --
it’s how humans experience the world every day,
a direct projection of our lived experience. [2/n]
- what does true multimodal intelligence mean?
- does the LLM paradigm even make sense for sensory modeling?
- and why is human sensing so effortless, so intuitive, yet so powerful?
something fundamental was missing
and you really can’t build superintelligence without first building supersensing.
so, what is supersensing?
supersensing in our context isn’t about fancy sensors or better cameras.
it’s about how a digital being truly experiences the world, absorbing endless streams of input and learning from them.
supersensing is part of intelligence
just like eyes are the part of the brain that touches the outside world.
you don't need sensing to solve coding and math.
but AI agents in the real world need sensory modeling.
or maybe as @karpathy said, that’s all it’ll ever need.
okay, enough philosophy.
to be more concrete, we think the following taxonomy makes sense and captures how things evolve from what we have now to what we actually need to build next.
0. (linguistic-only understanding): no sensory capabilities; reasoning confined to text and symbols. current MLLMs have progressed beyond this stage, yet still retain traces of its bias.
1. semantic perception: parsing pixels into objects, attributes, and relations. this corresponds to the strong multimodal "show and tell" capabilities present in MLLMs.
2. streaming event cognition: processing live, unbounded streams while proactively interpreting and responding to ongoing events. this aligns with current efforts to make MLLMs real-time assistants.
3. implicit 3D spatial cognition: understanding video as projections of a 3D world. agents must know what is present, where, how things relate, and how configurations change over time. today’s multimodal models remain VERY limited here.
4. predictive world modeling: the brain makes “unconscious inferences” by predicting latent world states based on prior expectations. Current multimodal systems don’t have an internal model that can anticipate future states, maintain persistent memory, or reason and plan.
to study all this, video is the ultimate medium --
it’s how humans experience the world every day,
a direct projection of our lived experience. [2/n]
3
the first problem is (always) benchmarking
I love multimodal LLMs
but without open and critical exploration,
we risk optimizing for the wrong things.
many video benchmarks don’t even need real sensing; as we show, a single frame or captions can score high.
companies build strong language models → benchmarks reward text recall →
models score higher by scaling up LLMs →
others follow
the loop goes on, until we realize the ghost we’ve summoned still trails behind a cat, or a squirrel .
the moravec paradox strikes again: what’s effortless for humans, sensing, is hardest for models.
this isn’t only about science.
real value comes from models that perceive and act in the real world:
from factories to farms to hospitals…
from robots to AI glasses that people are supposedly going to wear as personal assistants.
LLMs can’t do that alone.
superintelligence starts with supersensing. [3/n]
I love multimodal LLMs
but without open and critical exploration,
we risk optimizing for the wrong things.
many video benchmarks don’t even need real sensing; as we show, a single frame or captions can score high.
companies build strong language models → benchmarks reward text recall →
models score higher by scaling up LLMs →
others follow
the loop goes on, until we realize the ghost we’ve summoned still trails behind a cat, or a squirrel .
the moravec paradox strikes again: what’s effortless for humans, sensing, is hardest for models.
this isn’t only about science.
real value comes from models that perceive and act in the real world:
from factories to farms to hospitals…
from robots to AI glasses that people are supposedly going to wear as personal assistants.
LLMs can’t do that alone.
superintelligence starts with supersensing. [3/n]
4
one key part of supersensing is visual spatial intelligence.
we had vsi-bench before that tests spatial reasoning, but videos were short and tasks too easy.
so we made something dumber but tougher: vsi-super.
task 1: vsr -- long-horizon spatial recall. we edit videos with diffusion models to insert out-of-context but still fitting objects (not random NIAHs).
task 2: vsc -- continual counting under changing views and scenes ...models just need to count what they see.
one test checks memory for object order, the other tests counting
simple, streamable -- can’t get easier right?
yet top LLMs fail.
and no, it’s not just about long context.
why?
- videos can be arbitrarily long.
for humans, that’s easy. our senses handle "infinite tokens." it's a lossy process, but we can remember days or weeks of experience. models can’t.
- they lack true spatial cognition and generalization.
humans can keep counting forever; models can’t.
leading models like gemini 2.5, for example, only counts up to a few dozen.
then it stops, even with longer videos and more objects.
tells you a lot about its training distribution. [4/n]
we had vsi-bench before that tests spatial reasoning, but videos were short and tasks too easy.
so we made something dumber but tougher: vsi-super.
task 1: vsr -- long-horizon spatial recall. we edit videos with diffusion models to insert out-of-context but still fitting objects (not random NIAHs).
task 2: vsc -- continual counting under changing views and scenes ...models just need to count what they see.
one test checks memory for object order, the other tests counting
simple, streamable -- can’t get easier right?
yet top LLMs fail.
and no, it’s not just about long context.
why?
- videos can be arbitrarily long.
for humans, that’s easy. our senses handle "infinite tokens." it's a lossy process, but we can remember days or weeks of experience. models can’t.
- they lack true spatial cognition and generalization.
humans can keep counting forever; models can’t.
leading models like gemini 2.5, for example, only counts up to a few dozen.
then it stops, even with longer videos and more objects.
tells you a lot about its training distribution. [4/n]
5
you might ask -- isn’t this just a data or scaling problem?
partly, yes.
that’s why we’re building the new Cambrian-S video MLLM family.
we want to push the limits of the current paradigm.
we think data and scaling are essential for supersensing (just not sufficient.)
the core issue: there’s no real training data for spatial cognition.
so we built one called VSI-590K.
it includes 590,000 training samples, from first-person exploration of indoor envs with 3D annotations, videos from simulators, and youtube videos pseudo-labeled with vision tools like VGGT.
we explored post-training recipes, data mixtures, and a range of engineering details --
training models from 0.5B to 7B parameters.
the result shows strong spatial reasoning -- up to 30% better than our base MLLM. even the smallest models perform impressively.
we’ve open-sourced both the data and the models.
we think they’ll be useful for many tasks.
but let’s be clear -- this still doesn’t solve vsi-super.
and we’re increasingly convinced:
building multimodal models the LLM way isn’t the final path to supersensing. [5/n]
partly, yes.
that’s why we’re building the new Cambrian-S video MLLM family.
we want to push the limits of the current paradigm.
we think data and scaling are essential for supersensing (just not sufficient.)
the core issue: there’s no real training data for spatial cognition.
so we built one called VSI-590K.
it includes 590,000 training samples, from first-person exploration of indoor envs with 3D annotations, videos from simulators, and youtube videos pseudo-labeled with vision tools like VGGT.
we explored post-training recipes, data mixtures, and a range of engineering details --
training models from 0.5B to 7B parameters.
the result shows strong spatial reasoning -- up to 30% better than our base MLLM. even the smallest models perform impressively.
we’ve open-sourced both the data and the models.
we think they’ll be useful for many tasks.
but let’s be clear -- this still doesn’t solve vsi-super.
and we’re increasingly convinced:
building multimodal models the LLM way isn’t the final path to supersensing. [5/n]
6
looking ahead, we’re prototyping something new --
we call it predictive sensing.
our paper cited tons of work from cogsci and developmental psychology.
the more we read, the more amazed we became by human / animal sensing.
the human visual system is super high-bandwidth, yet insanely efficient.
each eye’s 6 million cone receptors can transmit ~1.6 Gbit/s,
yet the brain uses only about 10 bits/s to guide behavior.
most sensory data is filtered, compressed, and everything is autopiloted -- you don’t even notice.
how does our brain pull that off?
one leading theory:
your brain runs a predictive world model in the background for sensing,
constantly forecasting the future and comparing it to what actually happens.
- if the prediction error is low → it’s expected, you can ignore it.
- if it’s high → it’s a surprise, and your brain pays attention, updating memory.
we don't have anything comparable in LLMs right now.
to test this idea, we trained a latent frame prediction (LFP) head on top of Cambrian-S.
we estimate "surprise" during inference, and use it in two ways:
1️⃣ surprise-driven memory management -- compress or skip non-surprising frames, focus compute on surprising ones.
2️⃣ surprise-driven event segmentation -- use surprise spikes to detect event boundaries or scene changes.
by leveraging signals from this internal predictive model,
we’re already seeing promising gains on spatial cognition tasks.
it’s just a toy predictive world model -- but with this mechanism,
our small model outperforms gemini on vsi-super. [6/n]
we call it predictive sensing.
our paper cited tons of work from cogsci and developmental psychology.
the more we read, the more amazed we became by human / animal sensing.
the human visual system is super high-bandwidth, yet insanely efficient.
each eye’s 6 million cone receptors can transmit ~1.6 Gbit/s,
yet the brain uses only about 10 bits/s to guide behavior.
most sensory data is filtered, compressed, and everything is autopiloted -- you don’t even notice.
how does our brain pull that off?
one leading theory:
your brain runs a predictive world model in the background for sensing,
constantly forecasting the future and comparing it to what actually happens.
- if the prediction error is low → it’s expected, you can ignore it.
- if it’s high → it’s a surprise, and your brain pays attention, updating memory.
we don't have anything comparable in LLMs right now.
to test this idea, we trained a latent frame prediction (LFP) head on top of Cambrian-S.
we estimate "surprise" during inference, and use it in two ways:
1️⃣ surprise-driven memory management -- compress or skip non-surprising frames, focus compute on surprising ones.
2️⃣ surprise-driven event segmentation -- use surprise spikes to detect event boundaries or scene changes.
by leveraging signals from this internal predictive model,
we’re already seeing promising gains on spatial cognition tasks.
it’s just a toy predictive world model -- but with this mechanism,
our small model outperforms gemini on vsi-super. [6/n]
7
it’s a long paper, but trust me --
there are a lot of details we found genuinely interesting.
if you’re working on video multimodal models
it might be worth a read.
I don’t know if our approach is the right path --
but I do know the current paradigm is not enough,
and open science, open research is the only way forward.
and it’s not just this one paper.
we’re also co-releasing two related projects led by @_ellisbrown 👇
1️⃣ a study on multimodal benchmark design -- how to stress-test benchmarks and properly remove language bias.
2️⃣ a summary of our lessons from building simulators to collect spatial sensing videos (the same data we used for Cambrian-S). [7/n]
there are a lot of details we found genuinely interesting.
if you’re working on video multimodal models
it might be worth a read.
I don’t know if our approach is the right path --
but I do know the current paradigm is not enough,
and open science, open research is the only way forward.
and it’s not just this one paper.
we’re also co-releasing two related projects led by @_ellisbrown 👇
1️⃣ a study on multimodal benchmark design -- how to stress-test benchmarks and properly remove language bias.
2️⃣ a summary of our lessons from building simulators to collect spatial sensing videos (the same data we used for Cambrian-S). [7/n]
8
for all the details, resources, and projects, check out:
🔗 cambrian-mllm.github.io
we’re a small group of researchers, but we’ve been fortunate to have incredible support from thought leaders like @drfeifei and @ylecun.
(huge congrats to both on the Queen Elizabeth Prize!)
they’ve guided us throughout this project
helping us think differently, recalibrate our north stars,
and encourage us to explore new paths toward intelligence.
the project is led by @shushengyang who previously worked on Qwen models and is now fearlessly pushing into the next paradigm.
core contributions from @jihanyang13, @PinzhiHuang, @_ellisbrown, and many others (too many to tag but equally important ❤️).
special thanks to @googlecloud TRC program for enabling us with the compute + storage to dream big 🚀
we’re proud to give back with many open-source JAX and PyTorch XLA codebases.
finally, hope you enjoyed our little film for this project
a $0-budget student production led by @fred_lu_443,
but really, a love letter to NYC.
a city full of people, stories, and motion --
the spark behind our dream of supersensing intelligence.
not just to see the world,
but to feel it.
to understand it.
to help the people in it. 🤍 [n/n]
🔗 cambrian-mllm.github.io
we’re a small group of researchers, but we’ve been fortunate to have incredible support from thought leaders like @drfeifei and @ylecun.
(huge congrats to both on the Queen Elizabeth Prize!)
they’ve guided us throughout this project
helping us think differently, recalibrate our north stars,
and encourage us to explore new paths toward intelligence.
the project is led by @shushengyang who previously worked on Qwen models and is now fearlessly pushing into the next paradigm.
core contributions from @jihanyang13, @PinzhiHuang, @_ellisbrown, and many others (too many to tag but equally important ❤️).
special thanks to @googlecloud TRC program for enabling us with the compute + storage to dream big 🚀
we’re proud to give back with many open-source JAX and PyTorch XLA codebases.
finally, hope you enjoyed our little film for this project
a $0-budget student production led by @fred_lu_443,
but really, a love letter to NYC.
a city full of people, stories, and motion --
the spark behind our dream of supersensing intelligence.
not just to see the world,
but to feel it.
to understand it.
to help the people in it. 🤍 [n/n]