4
$\begingroup$

I have read through many of the papers and articles linked in this thread but I haven't been able to find an answer to my question.

I have built some small RL networks and I understand how REINFORCE works. I don't quite understand how they are applied to NAS though. Usually RL agents map a state to an action and get a reward so they can improve their decision making (which action to choose). I understand that the reward comes from the accuracy of the child network and the action is a series of digits encoding the network architecture.

What is passed as the state to the RL agent? This doesn't seem to be mentioned in the papers and articles I read. Is it the previous network? Example input data?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

(I will repeat a few details that you're already aware of, so that other users can also understand the context).

In the Neural Architecture Search (NAS) paper (that I mention in my answer to the question you link to in your question), the agent is the controller (see also this question Is there any difference between a control and an action in reinforcement learning?), which is implemented as a recurrent neural network (RNN). This controller produces actions (or controls), which are strings that represent the hyper-parameters of a neural network (see e.g. section 3.2), based on the reward that it receives, which, in this case, is the accuracy on the validation dataset that the designed (by the controller) and trained neural network obtains.

In this context, the controller is thus the reinforcement learning agent (or policy). The controller is an RNN that is represented by a vector of parameters, $\boldsymbol{\theta}_\text{c}$. This controller is trained using a policy gradient method to maximize the expected accuracy on the validation dataset (see e.g. section 3 of the paper). So $\boldsymbol{\theta}_\text{c}$ are adjusted using this policy gradient method so that the controller generates NN architectures that, after trained on some task, produce higher accuracy on a validation dataset.

The objective function of the controller is thus

$$ J(\boldsymbol{\theta}_\text{c}) = \mathbb{E}[R] $$

where $R$ is the accuracy on the validation dataset. Why expected? You will be performing this operation multiple times, so, intuitively, you want an average of the accuracy that you obtain on the validation dataset using multiple neural network architectures that the controller might produce.

The accuracy on the validation dataset, represented by $R$, is actually non-differentiable, so the authors use the famous REINFORCE algorithm (which I will not explain here). The authors actually use an approximation of the REINFORCE algorithm (see section 3.2).

$$ \frac{1}{m} \sum_{k=1}^m \sum_{t=1}^T \nabla_{\boldsymbol{\theta}_\text{c}} \log P (a_t \mid a_{{t-1}:1}; \boldsymbol{\theta}_\text{c}) R_k $$

So, given this formulation and taking into account the formulation of the REINFORCE algorithm, then the state of the agent seems to be $a_{{t-1}:1}$, that is, the previous actions of the agent. (Recall that the agent is a controller that is implemented as a recurrent neural network).

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for the great answer! Just one follow up question: So calculation for the next hyperparameter would be done like $ \text{activation_function}(\boldsymbol{\theta} \times a_{t-1} + \text{bias}) $ where usually softmax is chosen I guess? And the last action at step 0 would be just a 0 vector right? $\endgroup$
    – jaaq
    Commented Jul 19, 2019 at 10:24
  • 1
    $\begingroup$ @jaaq I think that your reasoning makes sense. However, I would suggest you look at this implementation of ENAS (efficient NAS), https://github.com/carpedm20/ENAS-pytorch, so that you have an idea of what people have done. $\endgroup$
    – nbro
    Commented Jul 19, 2019 at 17:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .