AI Scientists Use Evolution to Design Better Neural Network Trainers

Researchers employed genetic algorithms to automatically design deep learning optimizers, discovering a new algorithm that outperforms the industry-standard Adam optimizer by 2.6% overall and 7.7% on specific benchmarks. The evolved algorithm makes surprising design choices that contradict conventional wisdom, suggesting evolution can unlock optimization strategies humans wouldn’t naturally discover.