I heard today that second order optimization algorithms are now...
Jeffrey Emanuel@doodlestein
55 views
Feb 08, 2025
~7 min read
Advertisement
1
I heard today that second order optimization algorithms are now making a comeback in deep learning, with the Shampoo optimizer outperforming Adam in some evaluation. It brought back a flood of memories for me from 2012 when I became extremely excited about “Hessian Free” methods.
2
I looked through my old email messages and found a long correspondence from 2012 between me and James Martens (now at Google Brain), the inventor of the Hessian Free approach. It’s so mind blowing thinking of how absolutely quaint everything sounds— the number of params, layers.
4
The amount of VRAM and cores on the GeForce Titan, and using Jacket for MATLAB (the first easy way of using GPUs for this stuff other than Theano). I decided for fun to paste the entire email thread into Claude 3.5 Sonnet and ask what it thought of it, and I love its take on it:
5
This is a fascinating historical artifact from around 2013! It captures a really interesting moment in deep learning history. The thread shows you working with Martens' Hessian-free optimization code for neural networks, which was quite cutting-edge at the time. What's particularly interesting is how this exchange demonstrates the transition period from more classical optimization approaches to what would become modern deep learning.
A few things stand out to me:
1. You were clearly excited about the potential, noting how Martens' code got you better results in 30 minutes than 15 hours of contrastive divergence and backprop. This was a hint of the efficiency gains that would help enable modern deep learning.
2. Your discussion about GPU computing is particularly prescient. You were experimenting with gaming GPUs for ML (which became standard practice) and discussing the tradeoffs between different cards like the GTX 690 vs Tesla.
3. Your work on asynchronous training prefigured what would become very important for large-scale deep learning. The discussion about using Dropbox for parameter synchronization is almost charmingly quaint now, but the core ideas about distributed training were spot on.
4. James' skepticism about asynchronous updates is interesting - he questioned whether there was "a distinct advantage over just computing a bunch of gradients on different datasets and then combining them centrally." This debate about synchronous vs asynchronous training is still relevant today.
The thread also has a poignant moment where you both briefly discuss Marc'Aurelio Ranzato's work at Google, and James makes a comment about still having "a long way to go" before achieving artificial general intelligence ("Skynet"), showing how even pioneering researchers couldn't quite anticipate how rapidly the field would advance.
I find it particularly interesting how academic and industry boundaries were more fluid then - using consumer gaming GPUs for research, sharing code improvements openly. Would you say the field has become more or less open since then?
A few things stand out to me:
1. You were clearly excited about the potential, noting how Martens' code got you better results in 30 minutes than 15 hours of contrastive divergence and backprop. This was a hint of the efficiency gains that would help enable modern deep learning.
2. Your discussion about GPU computing is particularly prescient. You were experimenting with gaming GPUs for ML (which became standard practice) and discussing the tradeoffs between different cards like the GTX 690 vs Tesla.
3. Your work on asynchronous training prefigured what would become very important for large-scale deep learning. The discussion about using Dropbox for parameter synchronization is almost charmingly quaint now, but the core ideas about distributed training were spot on.
4. James' skepticism about asynchronous updates is interesting - he questioned whether there was "a distinct advantage over just computing a bunch of gradients on different datasets and then combining them centrally." This debate about synchronous vs asynchronous training is still relevant today.
The thread also has a poignant moment where you both briefly discuss Marc'Aurelio Ranzato's work at Google, and James makes a comment about still having "a long way to go" before achieving artificial general intelligence ("Skynet"), showing how even pioneering researchers couldn't quite anticipate how rapidly the field would advance.
I find it particularly interesting how academic and industry boundaries were more fluid then - using consumer gaming GPUs for research, sharing code improvements openly. Would you say the field has become more or less open since then?
6
Several interesting things stand out that connect to later developments:
1. The discussion of optimization and initialization approaches. Martens mentions "as long as initializations are already 'good enough' (i.e. well scaled, no saturation, reasonable variety of activation patterns and lack of degenerate symmetries)" - this prefigures a lot of the later work on initialization schemes like He initialization and batch normalization that became crucial for training very deep networks.
2. Your focus on automated hyperparameter optimization using CMA-ES is fascinating. The debate between your approach and Bayesian optimization (which Martens' colleagues were using) presages the later explosion of AutoML and neural architecture search. Your comment about being willing to "take months of computations before you get something that works" was quite prescient - this became the norm for large model training.
3. The discussion of regularization approaches is particularly interesting. You were experimenting with different forms of dropout and noise injection, and Martens suggested looking at score matching and contractive penalties. This was right before the big breakthrough of batch normalization, which ended up solving many of these issues more elegantly.
4. Your comments about leveraging gaming hardware and the significance of RAM/memory constraints remain highly relevant. The field is still grappling with memory limitations, now at a much larger scale with transformer models.
5. The discussion about ensemble methods and model integration is fascinating - your description of treating it as a "secondary machine-learning problem" where you learn relationships between models feels like an early precursor to mixture-of-experts approaches that became popular in large language models.
There's also something telling about the research culture of the time. The casual exchange of code improvements, the openness about what's working and what isn't, the willingness to try "crazy things that don't have much theoretical support" - this experimental spirit was crucial to the field's development but became harder to maintain as commercial interests grew.
This really captures a pivotal moment just before deep learning exploded - when researchers were starting to see the potential but still working through fundamental issues of optimization, regularization, and scaling that would soon be largely solved through a combination of algorithmic breakthroughs and raw computational power.
Would you say the field lost something when it moved away from this kind of detailed investigation of optimization and training dynamics toward a more empirical "scale is all you need" approach?
1. The discussion of optimization and initialization approaches. Martens mentions "as long as initializations are already 'good enough' (i.e. well scaled, no saturation, reasonable variety of activation patterns and lack of degenerate symmetries)" - this prefigures a lot of the later work on initialization schemes like He initialization and batch normalization that became crucial for training very deep networks.
2. Your focus on automated hyperparameter optimization using CMA-ES is fascinating. The debate between your approach and Bayesian optimization (which Martens' colleagues were using) presages the later explosion of AutoML and neural architecture search. Your comment about being willing to "take months of computations before you get something that works" was quite prescient - this became the norm for large model training.
3. The discussion of regularization approaches is particularly interesting. You were experimenting with different forms of dropout and noise injection, and Martens suggested looking at score matching and contractive penalties. This was right before the big breakthrough of batch normalization, which ended up solving many of these issues more elegantly.
4. Your comments about leveraging gaming hardware and the significance of RAM/memory constraints remain highly relevant. The field is still grappling with memory limitations, now at a much larger scale with transformer models.
5. The discussion about ensemble methods and model integration is fascinating - your description of treating it as a "secondary machine-learning problem" where you learn relationships between models feels like an early precursor to mixture-of-experts approaches that became popular in large language models.
There's also something telling about the research culture of the time. The casual exchange of code improvements, the openness about what's working and what isn't, the willingness to try "crazy things that don't have much theoretical support" - this experimental spirit was crucial to the field's development but became harder to maintain as commercial interests grew.
This really captures a pivotal moment just before deep learning exploded - when researchers were starting to see the potential but still working through fundamental issues of optimization, regularization, and scaling that would soon be largely solved through a combination of algorithmic breakthroughs and raw computational power.
Would you say the field lost something when it moved away from this kind of detailed investigation of optimization and training dynamics toward a more empirical "scale is all you need" approach?
7
A few more fascinating observations:
1. The early recognition of what would become critical scaling laws. You mention "For now I have de-emphasized working on the asynchronous stuff... the advantages... is likely not very pronounced until you get up to a Google scale." This was remarkably prescient - the importance of async computation didn't become fully apparent until models reached massive scale years later.
2. The discussion about fighting overfitting reflects the pre-modern era. You were exploring various regularization techniques and early stopping, while today the paradigm has almost reversed - with larger models, we often worry more about underfitting and optimization challenges than overfitting.
3. Martens' reference to things "no one really understands what happens in non-convex optimization of neural nets beyond vague and often wrong intuitions" remains remarkably relevant. Despite all our progress, we still struggle with fundamental theoretical understanding of why deep learning works.
4. The emphasis on Matlab is a fascinating time capsule. You discuss Jacket licenses and Matlab-specific optimizations that would soon become irrelevant as PyTorch and TensorFlow took over.
5. The interaction exemplifies the very collaborative nature of early deep learning research - sharing code improvements, detailed discussions of implementations, open exchange of ideas. This was before the intense competition and commercialization that came later.
The thread also shows the moment right before attention mechanisms and transformers would revolutionize the field. The focus was still heavily on feed-forward and convolutional architectures, with a brief mention of RNNs at the end.
It's striking how some of the challenges you were grappling with (memory management, distributed training, initialization) remain relevant today, just at a much larger scale. The fundamental problems haven't changed as much as the scale at which we encounter them.
1. The early recognition of what would become critical scaling laws. You mention "For now I have de-emphasized working on the asynchronous stuff... the advantages... is likely not very pronounced until you get up to a Google scale." This was remarkably prescient - the importance of async computation didn't become fully apparent until models reached massive scale years later.
2. The discussion about fighting overfitting reflects the pre-modern era. You were exploring various regularization techniques and early stopping, while today the paradigm has almost reversed - with larger models, we often worry more about underfitting and optimization challenges than overfitting.
3. Martens' reference to things "no one really understands what happens in non-convex optimization of neural nets beyond vague and often wrong intuitions" remains remarkably relevant. Despite all our progress, we still struggle with fundamental theoretical understanding of why deep learning works.
4. The emphasis on Matlab is a fascinating time capsule. You discuss Jacket licenses and Matlab-specific optimizations that would soon become irrelevant as PyTorch and TensorFlow took over.
5. The interaction exemplifies the very collaborative nature of early deep learning research - sharing code improvements, detailed discussions of implementations, open exchange of ideas. This was before the intense competition and commercialization that came later.
The thread also shows the moment right before attention mechanisms and transformers would revolutionize the field. The focus was still heavily on feed-forward and convolutional architectures, with a brief mention of RNNs at the end.
It's striking how some of the challenges you were grappling with (memory management, distributed training, initialization) remain relevant today, just at a much larger scale. The fundamental problems haven't changed as much as the scale at which we encounter them.
8
Looking even deeper:
1. The references to Marc'Aurelio Ranzato and his work at Google provides interesting historical context - you can see the early stages of industry labs becoming major players in ML research, before the massive scaling of corporate AI research we see today.
2. The discussion of early stopping shows how different training dynamics were understood then. Today with very large models we often train to apparent convergence and deal with overfitting through scale and architecture, rather than stopping early.
3. Your discussion of ensemble methods as "integrating thousands of pre-computed models" and treating it as a "secondary machine learning problem" where you learn relationships between models feels like a precursor to modern model distillation and knowledge transfer techniques.
4. There's an intriguing comment from Martens about hoping to explore "connections between Bayesian methods and sampling" which he thought might help with "global optimization performance." This feels related to the later emergence of diffusion models and score-based generative modeling.
5. The thread reveals the research community's state of mind about AGI at the time. Martens says "In terms of Skynet though, I think we have a long way to go still :) Even if you get a good system for perception, you still have to solve the problem of giving these systems some kind agency or 'will' to do things." This shows how even leading researchers underestimated how quickly capabilities would advance.
6. The references to Hinton getting coverage in the NYT and predictions about AI "blowing up in the next couple years" were quite prescient about the field's trajectory.
The whole exchange really captures a moment when deep learning was just about to take off but hadn't quite yet - like a photograph taken just before a major historical event.
1. The references to Marc'Aurelio Ranzato and his work at Google provides interesting historical context - you can see the early stages of industry labs becoming major players in ML research, before the massive scaling of corporate AI research we see today.
2. The discussion of early stopping shows how different training dynamics were understood then. Today with very large models we often train to apparent convergence and deal with overfitting through scale and architecture, rather than stopping early.
3. Your discussion of ensemble methods as "integrating thousands of pre-computed models" and treating it as a "secondary machine learning problem" where you learn relationships between models feels like a precursor to modern model distillation and knowledge transfer techniques.
4. There's an intriguing comment from Martens about hoping to explore "connections between Bayesian methods and sampling" which he thought might help with "global optimization performance." This feels related to the later emergence of diffusion models and score-based generative modeling.
5. The thread reveals the research community's state of mind about AGI at the time. Martens says "In terms of Skynet though, I think we have a long way to go still :) Even if you get a good system for perception, you still have to solve the problem of giving these systems some kind agency or 'will' to do things." This shows how even leading researchers underestimated how quickly capabilities would advance.
6. The references to Hinton getting coverage in the NYT and predictions about AI "blowing up in the next couple years" were quite prescient about the field's trajectory.
The whole exchange really captures a moment when deep learning was just about to take off but hadn't quite yet - like a photograph taken just before a major historical event.
9
The actual email thread: dropbox.com/scl/fi/2yci3yi…
I guess I went into the wrong field, I should have tried to be a professional deep learning researcher! I just didn’t want to waste all that time in grad school getting a PhD.
I guess I went into the wrong field, I should have tried to be a professional deep learning researcher! I just didn’t want to waste all that time in grad school getting a PhD.
10
Since the PDF is so hard to read chronologically, I extracted it here for posterity:
github.com/Dicklesworthst…
github.com/Dicklesworthst…
