Minimum Fidelity for Reliable Architecture Ranking in Bayesian NAS for Object Detection
DOI:
https://doi.org/10.18372/1990-5548.88.20966Keywords:
neural architecture search, bayesian optimization, minimum fidelity, proxy metrics, learning curve, TPE, OptunaAbstract
This paper addresses the problem of reducing computational costs in Neural Architecture Search for object detection. The key question in low-fidelity approaches is: after what minimum number of epochs does the early training signal already provide acceptable architecture ranking? This paper presents a methodology for determining minimum fidelity based on out-of-sample rank correlation: trials are split into calibration (70%) and test (30%) sets, a composite proxy is built on the first set and evaluated on the second. An empirical study was conducted on 60 architectures trained for 25 epochs on an object detection dataset (6,772 images, 6 classes). The results show that a rank ensemble of three training metrics (val_loss, train_accuracy, val_accuracy) achieves Spearman ρ = 0.877 out-of-sample at epoch 3, providing 88% computational savings. A more complex 7-component metric underperforms the simple ensemble due to overfitting on a small sample (ρ_test = 0.70 vs. 0.88). The results are locally valid within the studied search space and the specific dataset.
References
T. Elsken, J. H. Metzen, and F. Hutter, “Neural architecture search: A survey,” JMLR, 2019. https://doi.org/10.1007/978-3-030-05318-5_11
C. Sciuto, K. Yu, M. Jaggi, C. Musat, and M. Salzmann, “Evaluating the search phase of neural architecture search,” ICLR, 2020.
E. Real, A. Aggarwal, Y. Huang, and Q. V. Le, “Regularized evolution for image classifier architecture search,” AAAI, 2019. https://doi.org/10.1609/aaai.v33i01.33014780
H. Liu, K. Simonyan, and Y. Yang, “DARTS: Differentiable architecture search,” ICLR, 2019.
J. Bergstra, R. Bardenet, Y. Bengio, and B. Kégl, “Algorithms for hyper-parameter optimization,” NeurIPS, 2011.
M. S. Abdelfattah, A. Mehrotra, Ł. Dudziak, and N. D. Lane, “Zero-cost proxies for lightweight NAS,” ICLR, 2021.
J. Mellor, J. Turner, A. Storkey, and E. J. Crowley, “Neural architecture search without training,” ICML, 2021.
T. Domhan, J. T. Springenberg, and F. Hutter, “Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves,” IJCAI, 2015.
L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar, “Hyperband: A novel bandit-based approach to hyperparameter optimization,” JMLR, 2017.
J. Siems, L. Zimmer, A. Zela, J. Lukasik, M. Keuper, and F. Hutter, “NAS-Bench-301 and the case for surrogate benchmarks for NAS,” NeurIPS (Workshop), 2020.
T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, “Optuna: A next-generation hyperparameter optimization framework,” KDD, 2019. https://doi.org/10.1145/3292500.3330701.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
The scientific journal “Electronics and control systems” adheres to the principles of Open Access and provides free, immediate, and permanent access to all published materials without financial, technical, or legal barriers for readers.
All articles are published in Open Access under the Creative Commons Attribution 4.0 International (CC BY 4.0) license.
Copyright
Authors who publish their works in the journal “Electronics and control systems”:
-
retain the copyright to their publications;
-
grant the journal the right of first publication of the article;
-
agree to the distribution of their materials under the CC BY 4.0 license;
-
have the right to reuse, archive, and distribute their works (including in institutional and subject repositories), provided that proper reference is made to the original publication in the journal.
