New answers tagged reinforcement-learning
0
Assume you are the snake.
In front of you is empty. Left of you is empty. Right of you is empty. The distance to the apple is 4. The apple straight in front of you. Your length is 20.
Can you make a good decision with this input? In which direction would you go to achieve maximum score?
From the given input, you could go straight forward to the apple. But ...
1
I thought about my input-layer. I had the 500 states one hot encoded. So 499 of every input node would be 0. And 0 is very bad in an neural network. I tried the same code with the "CardPole-v0" and it worked.
So think about your input guys
2
If $\pi$ is a random policy, and after running through this algorithm, and for each state take the $\max Q(s,a)$ for all possible actions, why would that not be equal to $Q_{\pi^*}(s, a)$ (optimal Q function)?
Assuming that the estimates for $Q_{\pi}(s,a)$ have converged to close to correct values from many samples, then a policy based on $\pi'(s) = \text{...
0
I think the problem is that you are not defining the weights and biases as parameters. So, when you backpropagate, they are not modified.
These lines should do the trick:
self.weight_mu = Parameter(torch.Tensor(out_features, in_features))
self.weight_sigma = Parameter(torch.Tensor(out_features, in_features))
self.bias_mu = Parameter(torch.Tensor(...
1
It is quite common in DQN to instead of having the neural network represent function $f(s,a) = \hat{q}(s,a,\theta)$ directly, it actually represents $f(s)= [\hat{q}(s,1,\theta), \hat{q}(s,2,\theta), \hat{q}(s,3,\theta) . . . \hat{q}(s,N_a,\theta)]$ where $N_a$ is the maximum action, and the input the current state. That is what is going on here. It is ...
0
From what I understood in a classifier a common method is that you sample a mini-batch, calculate the loss for every example, calculate the average loss over the whole batch and adjust the weights w.r.t to average loss? (Please correct me if I'm wrong)
You are wrong.
The weights are adjusted w.r.t. to average gradient, and this must be calculated using ...
2
Well, the way to know that the agent is actually learning is by looking at its behavior while it performs the task, and by comparing against a known optimal performance.
So, does your agent reaches the goal quickly? Does it step out of the grid frequently? What is the maximum possible sum of rewards / minimum number of steps attainable? Is the agent close ...
2
I think you can break this problem down into two parts to try and find the solution.
1. Can the neural network model the desired function?
Take the tabular function you have learned in the exact agent, and treat it as training data for the neural network model, using the same loss function and other hyperparameters as you intend to use when the NN is being ...
0
A trajectory ist just a sequence of states and actions. In RL, the goal is to maximize the reward, by finding the right trajectories.
$$
\operatorname{max}_\tau R(\tau)
$$
This means
maximizing not immediate reward (caused by one action from a state), but cumulative reward (all states and actions: trajectory)
2
About the first question, you are right. The $i$ denotes a sample trajectory corresponding to a whole episode. However, Sutton's version is exactly the same one as Levine's if you choose $N=1$.
About the second question, the Policy Gradient theorem only tells you what is the gradient up to a constant, so basically any constant is irrelevant. Now, even if ...
3
It doesn't seem that it is a "proper" symbol.
I guess that $\sup$ simply refers to the supremum, that is, you want to select actions that maximize the quantity that comes to the right of $\sup$, while $\text{dist}$ is simply a proxy for any possible distance between distributions. For example, you can replace $\text{dist}$ with the Kullback-Leibler ...
3
I think you are a bit confused about what is the update function and the target.
The equation you have there, and what is done in the video is the estimation of the true value of a certain state. In Temporal-Difference algorithms this is called the TD-Target.
The reason for your confusion might be that in the video he starts from the end state and goes ...
3
This is simply from definition of return in average reward setting (look at equation $10.9$). The "standard" TD error is defined as
\begin{equation}
TD_{\text{error}} = R_{t+1} + V(S_{t+1}) - V(S_t)
\end{equation}
In average reward setting, average reward $r(\pi)$ is subtracted from reward at $t$, $R_t$, so TD error in this case is
\begin{equation}
TD_{\text{...
0
The popular Q-learning algorithm is known to overestimate action values under certain conditions. It was not previously known whether, in practice, such overestimations are common, whether they harm performance, and whether they can generally be prevented. In this paper, we answer all these questions affirmatively. In particular, we first show that the ...
0
Welcome to the mine-field of semantic definitions within AI! According to Encyclopedia Britannica ML is a “discipline concerned with the implementation of computer software that can learn autonomously.” There are a bunch of other definitions for ML but generally they are all this vague, saying something about “learning”, “experience”, “autonomous”, etc. in ...
Top 50 recent answers are included
