@doodlestein: I heard today that second orde...

I heard today that second order optimization algorithms are now making a comeback in deep learning, with the Shampoo optimizer outperforming Adam in some evaluation. It brought back a flood of memories for me from 2012 when I became extremely excited about “Hessian Free” methods.

I looked through my old email messages and found a long correspondence from 2012 between me and James Martens (now at Google Brain), the inventor of the Hessian Free approach. It’s so mind blowing thinking of how absolutely quaint everything sounds— the number of params, layers.