Create optimizer object using the tf.keras.optimizers.get function

Question

I am trying to have the type of optimizer as a variable in the hyperparameter tuning phase. For that reason I am trying to use the tf.keras.optimizer.get function defined in https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/keras/optimizers.py.

However, I am getting different kind of errors depending on how I am calling the function and I cannot figure out the problem. For example, when I run the following command:

opt = tf.keras.optimizers.get({"class_name": hyperparams['optimizer'],
                                      "learning_rate": hyperparams['learning_rate']})

I get an error saying "ValueError: Improper config format: {'class_name': 'adagrad', 'learning_rate': 0.01} ".

Does anybody know what is the proper way of calling this function with a configuration dictionary if I want to have a configurable optimizer type and learning rate?

Answer 1

In order to have a configurable optimizer and configurable hyperparameters for it you need to make the function call in the following way:

opt = tf.keras.optimizers.get({"class_name": hyperparams['optimizer'],
                               "config": {"learning_rate": hyperparams['learning_rate']}})

where class_name is the name of the optimizer you want to use and the config dictionary contains the hyperparameters and their values for the given optimizer.