# Tag Info

3

The update form $\theta^{\prime} \leftarrow \tau \theta+(1-\tau) \theta^{\prime}$ (where $\theta'$ and $\theta$ represent the weights of the target network and the current network, respectively) does exist and is correct. It is called soft update and it has been used in the Deep Deterministic Policy Gradient (DDPG) paper, which uses the concept of a target ...

2

You have two options, either interpolate or restrict the actions only to values that produce states which are in your state vector. The simplest interpolation scheme is a linear interpolation, which works as follows (assuming DS contains a set of grid points in increasing order). For a state $s'$ you can locate its closest neighbours from the array DS and ...

3

I know that a seed can be set to incorporate more determinism into the training. However, there could be other pseudo-random sequences that produce slightly better results? That is correct. If you fix the seed for a process which inherently has stochastic behaviour by design (such as initialising neural network params), then what you know about the model is ...

3

Can deep reinforcement learning algorithms be deterministic in their reproducibility in results? Yes, but only if you control all places in the code where stochastic methods are used (typically by seeding the affected RNGs): Neural network weight initialisation Action choice for $\epsilon$-greedy or other behaviour policy (does not apply in your case, ...

1

In a toy environment, this is a choice you can make relatively freely, depending on what you want to achieve with the learning challenge. It may help if you think through what the actual consequences for making the "wrong" move are in your environment. There are a few self-consistent options: The move simply cannot be made and count as playing the game as ...

2

Q-values represent expected return after taking action $a$ in state $s$, so they do tell you how good it is to take an action in the specific state. Better actions will have larger Q-values. Q-values can be used to compares actions but they are not very meaningful in representing performance of the agent since you have nothing to compare them with. You don't ...

0

No, mainly because these are all stochastic approximations and may not represent the true values. Almost nothing good can be said about NN approximations to value and Q functions(at least according to a professor I have had).

1

I'm not familiar with the ins and outs of self driving cars but I imagine that the action space is not discrete. For instance, the car may want to decide what angle it needs to turn (rather than left or right). The update in Q-Learning involves taking $\max_aQ(s,a)$; this is theoretically possible for a continuous action space but it would itself require ...

0

I think it says something about the training progress, while another approach you can make sure is to look at the gradient norm. Sometimes, the training loss is really noisy while the gradient norm is much more clear.

2

When doing gradient descent update for this single example, should the target output to set for the network be equivalent to $Q(s_1,a_0), Q(s_1,a_1), r_2 + \gamma max_aQ(s',a',\theta) , Q(s_1,a_3),...$ ? Other than what looks like a couple of small typos, then yes. This is an implementation issue for DQN, where you have decided to create a function that ...

1

You are looking for the best actions which minimize the loss function. You sample a batch of memory buffer uniformly and define a loss function based on that batch. The memory buffer consists of trajectories. Each trajectory consists of an state and the action taken in that state which results in next state and an immediate reward. If the trajectory is ...

Top 50 recent answers are included