1
$\begingroup$

I’m using a simple neural network to solve a reinforcement learning problem.

The configuration is:

X-inputs: The current state Y-outputs: The possible actions

Whenever the network yields a “good” solution, i “reward” the network by training it a number of times.

Whenever the network yields a “bad” or “neutral” solution, i ignore it.

This seems to be working somewhat, but from what i read, everyone else (in broad terms) seems to be using a 2 neural network configuration for similar tasks. (Policy network and value network)

Am i missing something? - and are there any obvious caveats of the “single network” method i am using?

Supplemental question: Are there other methods of “rewarding” a network, aside from simply training it?

Thanks,

|improve this question
$\endgroup$
  • $\begingroup$ Is the output in your case a softmax over action choices? If that's the case, you are using a well-known RL method, and an answer can give its strengths and weaknesses compared to other methods. $\endgroup$ – Neil Slater Oct 31 at 9:33
1
$\begingroup$

From your description, it seems that you are implementing a version of an algorithm called REINFORCE. This algorithm belongs to a family called Policy Gradient methods, which directly optimizes the policy network $\pi(a_t|s_t)$ from rewards without ever worrying about estimating a value function. This type of algorithm is usually pretty slow and presents high variance.

The methods that you recognize as the ones using two neural networks correspond to a family called Actor-Critic methods. This type of algorithm uses the trajectories of rewards to estimate a value network $q(s_t,a_t)$ (called the critic), and, contrary to the previous family of methods, it uses the value network to train the policy network $\pi(a_t|s_t)$ (called the actor), instead of directly using the trajectory of rewards. This indirect dependence usually makes variance smaller and also learning faster. I recommend you have a look at chapter 13 of the book An Introduction to Reinforcement Learning.

So, to answer your first question: it seems you are missing the family of Actor-Critic methods. I recommend you learn about them since they are very powerful (e.g., read about DDPG or SAC).

About your second question, the standard method to "reward" a policy network is not by training it. Usually, you have a reward function $r(s_t,a_t)$ that depends on your current state $s_t$ and action $a_t$ and you modify the parameters $\theta$ of your network in such a way that the probability of an action $\pi(a_t|s_t)$ increases if the reward is positive or decreases if it is negative. More specifically, you perform stochastic gradient ascend steps like this one:

$$\theta_t\leftarrow\theta_t+\alpha\mathbb E\left[\sum_{k=t}^{T+t} \gamma^{k-t}r(s_k,a_k)\right]\nabla \log\pi(a_t|s_t,\theta_t)$$

What this formula says is that if in the time-step $t$ you take an action $a_t$ in the state $s_t$, wait $T$ steps and collect the rewards from $r(s_t,a_t)$ to $r(s_{t+T},a_{t+T})$, then, you should modify your parameters in the direction that the policy increases the most (i.e., $\nabla \log\pi(a_t|s_t,\theta_t)$) if the expected return $E\left[\sum_{k=t}^{T+t} \gamma^{k-t}r(s_k,a_k)\right]$ is positive, or in the direction that the policy decreases the most if that expected return is negative.

|improve this answer
$\endgroup$
  • $\begingroup$ So, @NeilSlater are you saying that there exists a target distribution that we know beforehand? I considered that, but I found it very weird that one could know that target distribution, which is precisely the solution of the RL problem. If you take your reward as 1 for "good" results and 0 for "bad" ones, you can see that the only difference is that in REINFORCE the loss function is the loglikelihood of a specific action, while in the Cross Entropy Method it is the dot product between the loglikelihood and that target distribution. $\endgroup$ – Diego Gomez Nov 2 at 21:01
  • $\begingroup$ @NeilSlater, but according to you exactly what is the loss function being used to train the network? If it is the Cross Entropy with the action itself (a vector with 0s in all positions except for the position corresponding to the action taken, say the i-th position, and a 1 in this i-th position), then as I already explained in the previous comment, it is exactly the same as REINFORCE using a reward function that has outcomes 0 and 1. This is the case since the dot product between the loglikelihood and a vector of 0s and a 1 in the i-th entry is precisely log pi(a_t|s_t). $\endgroup$ – Diego Gomez Nov 2 at 22:42
  • $\begingroup$ I still do not understand what would be the loss function. As far as I understand, it is some kind of distance between your output distribution and the target distribution that you build with your actions. Well, a typical loss function when your output is a distribution and your target is also a distribution is the Cross Entropy. Whether you derive CEM or not from the policy gradient theorem is not relevant. I'm telling you (just look at the formula) that CEM turns out to be the same as REINFORCE in the especial case that you choose a specific binary reward function and the Cross Entropy loss. $\endgroup$ – Diego Gomez Nov 3 at 18:48
  • $\begingroup$ I see your point in calling my characterization "innacurate" and I will edit those parts. I could argue that CEM is a funny version of REINFORCE in the context of RL, though XD. However, I think it is more appropriate to the principal question to introduce the Actor-Critic methods as opposed to the Policy Gradient methods, not the particular CEM algorithm that turns out to be equivalent to a specific version of REINFORCE $\endgroup$ – Diego Gomez Nov 3 at 18:54
  • $\begingroup$ To make CEM weight updates look the same as basic REINFORCE, you need to filter down to episodes where the only and final reward is +1 (which is not a great use for CEM - it works much better when returns can be differentiated), and also set discount factor to 1. Thanks for the edit anyway - it doesn't go quite as far as I'd like by identifying the algorithm the OP appears to be using, but at least it doesn't tell the OP that they are doing something "funny" :-) $\endgroup$ – Neil Slater Nov 3 at 20:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.