🕳️🐇Into the Rabbit Hull – Part II Continuing our interpretation of...

@thomas_fel_
Thomas Fel@thomas_fel_
6 views Oct 17, 2025 ~2 min read
Advertisement
1
🕳️🐇Into the Rabbit Hull – Part II

Continuing our interpretation of DINOv2, the second part of our study concerns the geometry of concepts and the synthesis of our findings toward a new representational phenomenology:
the Minkowski Representation Hypothesis
Media image
2
Under the Linear Rep. Hypothesis, we'd expect Dictionary to be quasi-orthogonality. Instead, training drives atoms from near-Grassmannian initialization to higher coherence.
Several concepts fire almost always the embedding is partly dense (!), contradicting pure sparse coding.
Media image
3
We found antipodal feature pairs (dᵢ ≈ − dⱼ): vertical vs horizontal lines, white vs black shirts, left vs right…

Also, co-activation statistics only moderately shape geometry: concepts that fire together aren't necessarily nearby—nor orthogonal when they don't.
Media image
4
Patch embeddings form smooth, connected surfaces tracing objects and boundaries -- even after removing positional components.

This may suggests interpolative geometry: tokens as mixtures between landmarks, shaped by clustering and spreading forces in the training objectives.
Media image
5
Can position explain this ?
We found that pos. information collapses: from high-rank to a near 2-dim sheet. Early layers encode precise location; later ones retain abstract axes.
This compression frees dimensions for features, and *position doesn't explain PCA map smoothness*
Media image
6
Taken together, the signs of partial density, local connectedness, and coherent dictionary atoms indicate that DINO’s representations are organized beyond linear sparsity alone.
Media image
7
Synthesizing these observations, we propose a refined view, motivated by Gärdenfors' theory and attention geometry.
Activations = multiple convex hulls simultaneously: a rabbit among animals, brown among colors, fluffy among textures.

The Minkowski Representation Hypothesis.
Media image
8
If this holds, three implications:
(i) Concepts = points (or regions), not directions
(ii) Probing is bounded: toward archetypes, not vectors
(iii) Can't recover generating hulls from sum: we should look deeper than just a single-layer activations to recover the true latents
Media image
9
That concludes this two-part descent into the Rabbit Hull.
Huge thanks to all collaborators who made this work possible — and again especially to @WangBinxu with whom this project was built, experiment after experiment.
🎮 kempnerinstitute.github.io/dinovision/
📄 arxiv.org/pdf/2510.08638
Actions
What You Can Do
  • Export as PDF or Markdown
  • Batch Export to Notion
  • Bookmark & Highlight
  • Screenshot Tweet
Create Free Account

Includes 7-day Premium trial

Advertisement