Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars

Posted on December 5, 2023 by Rhea Sukthanker

Authors: Simon Schrodi, Danny Stoll, Binxin Ru, Rhea Sanjay Sukthanker, Thomas Brox, and Frank Hutter TL;DR We take a functional view of neural architecture search that allows us to construct highly expressive search spaces based on context-free grammars, and show that we can efficiently find well-performing architectures. NAS is great, but… The neural architecture plays […]

Rethinking Bias Mitigation: Fairer Architectures Make for Fairer Face Recognition

Posted on by Rhea Sukthanker

Deep learning is applied to a wide variety of socially-consequential domains, e.g., credit scoring, fraud detection, hiring decisions, criminal recidivism, loan repayment, and face recognition, with many of these applications impacting the lives of people more than ever — often in biased ways. Dozens of formal definitions of fairness have been proposed, and many algorithmic […]

Symbolic Explanations for Hyperparameter Optimization

Posted on August 31, 2023 by Sarah Segel

Authors: Sarah Segel, Helena Graf, Alexander Tornede, Bernd Bischl, and Marius Lindauer TL;DR We propose to apply symbolic regression in a hyperparameter optimization setting to obtain explicit formulas providing simple and interpretable explanations of the effects of hyperparameters on the model performance. HPO is great, but… In the field of machine learning, hyperparameter optimization (HPO) […]

Self-Adjusting Bayesian Optimization with SAWEI

Posted on July 11, 2023 by Carolin Benjamins

By Carolin Benjamins, Elena Raponi, Anja Jankovic, Carola Doerr and Marius Lindauer TLDR: In BO: We self-adjust the exploration-exploitation trade-off online in the acquisition function, adapting to any problem landscape. Motivation Bayesian optimization (BO) encompasses a class of surrogate-based, sample-efficient algorithms for optimizing black-box problems with small evaluation budgets. However, BO itself has numerous design […]

Experience-Driven Algorithm Selection: Making better and cheaper selection decisions

Posted on June 27, 2023 by Aditya Mohan

Authors: Tim Ruhkopf, Aditya Mohan, Difan Deng, Alexander Tornede, Frank Hutter, Marius Lindauer TL;DR: We are augmenting classical algorithm selection with multi-fidelity information, which we make non-myopic through meta-learning – enabling us for the first time to interpret partial learning curves of varying lengths jointly and make good algorithm recommendations at low cost. Why should […]

PFNs4BO: In-Context Learning for Bayesian Optimization

Posted on June 25, 2023 by Samuel Müller

Can we replace the GP in BO with in-context learning? Absolutely. We achieve strong real-world performance on a variety of benchmarks with a PFN that uses only in-context learning to provide training values. This is what we found out in our ICML ‘23 paper PFNs4BO: In-Context Learning for Bayesian Optimization. Our models are trained only […]

Contextualize Me – The Case for Context in Reinforcement Learning

Posted on June 5, 2023 by Theresa Eimer

Carolin Benjamins, Theresa Eimer, Frederik Schubert, Aditya Mohan, Sebastian Döhler, André Biedenkapp, Bodo Rosenhahn, Frank Hutter and Marius Lindauer TLDR: We can model and investigate generalization in RL with contextual RL and our benchmark library CARL. In theory, without adding context we cannot achieve optimal performance and in the experiments we saw that using context […]

Hyperparameter Tuning in Reinforcement Learning is Easy, Actually

Posted on by Theresa Eimer

Hyperparameter Optimization tools perform well on Reinforcement Learning, outperforming Grid Searches with less than 10% of the budget. If not reported correctly, however, all hyperparameter tuning can heavily skew future comparisons.

Learning Activation Functions for Sparse Neural Networks: Improving Accuracy in Sparse Models

Posted on May 31, 2023 by Aditya Mohan

Authors: Mohammad Loni, Aditya Mohan, Mehdi Asadi, and Marius Lindauer TL, DR: Optimizing activation functions and hyperparameters of sparse neural networks help us squeeze more performance out of them; thus helping with deploying models in resource-constrained scenarios. We propose a 2-stage optimization pipeline to achieve this. Motivation: Sparse Neural Networks (SNNs) – the greener and […]

Understanding AutoRL Hyperparameter Landscapes

Posted on by Aditya Mohan

Authors: Aditya Mohan, Carolin Benjamins, Konrad Wienecke, Alexander Dockhorn, and Marius Lindauer TL;DR: We investigate hyperparameters in RL by building landscapes of algorithm performance for different hyperparameter values at different stages of training. Using these landscapes we empirically demonstrate that adjusting hyperparameters during training can improve performance, which opens up new avenues to build better […]