In general there are two different kinds of methodology to produce progress in any science or engineering field... both are important and can lead to transformative progress.

There's the "Edison way" where you brute-force a large predefined design space and you keep what works, without necessarily understanding why it works. This is akin to biological evolution. Nearly all of deep learning was built this way (despite the fancy math in papers, which is there to look nice 99% of the time).

And there's the "Einstein way" where you think up big ideas in a top-down fashion and derive precise results from them, before testing those results. This requires imagination, curiosity, high-abstraction thinking. This is, I suppose, the mode of progress favored by mathematicians.

View Tweet

AI (not LLMs, but search based AI) will be superhuman at the first type relatively quickly.