Paper Title
Efficient Deep Learning Hyperparameter Tuning on the Cloud

Abstract
The paper discusses how we can leverage cloud infrastructure for efficient training and hyperparameter tuning of deep neural networks on the cloud. With the introduction of Horovod framework distributed training of deep learning models has been made trivial on the cloud thereby reducing the time taken to run a single iteration, but the hyperparameter tuning exercise on high dimensional hyperparameter spaces remains a challenge. The paper experiments Bayesian Sequential Gaussian Process Optimization of hyperparameters on the cloud at different levels of concurrency for the warmup runs. Two different distributed hyper-parameter tuning approaches were experimented on the cloud – Training on multiple nodes with higher warm-up concurrency Vs Distributed Training on multiple nodes with Horovod and reduced number of warm-up runs. The results indicate that greater number of warm-up runs results in better exploration of the search space. The hyper parameter choices of every run were optimized using Bayesian optimization technique to take advantage of the learnings from previous runs. The hyper parameter tuning and distributed training with Horovod was performed using Azure Machine Learning Service for Video Activity Recognition problem using LRCN network with transfer learning from Resnet50 backbone. Keywords - Distributed Training, Horovod, Hyperparameter Tuning, Deep Learning, Bayesian Optimization, Automated Machine Learning, Neural Architecture Search