Neural Ensemble Search (NES) introduces a so-far-untapped dimension of neural architecture search: finding networks that can be combined into a strong ensemble. We’ve showed that this approach yield better performance and uncertainty calibration than deep ensembles of the same fixed architecture. NES algorithms offer an automatic way of finding such architectures that optimize for ensemble performance and eventually yield more diverse ensembles, without ever explicitly defining diversity.
