New answers tagged deep-rl
1
Simple policy parametizations, including linear functions in some cases, can solve continous control tasks in RL. It's therefore not necessary to have a complex approximator for the function to be expressive enough in capturing the desired agent's behaviour in popular RL benchmarks
Towards Generalization and Simplicity in Continous Control tries to answer ...
1
The purpose of the input network is to embed the input tuple into a state/task representation, that can then be fed into the RNN hidden state at each time step.
$(o^a_t,m^a′_{t−1},u^a_{t−1},a)$ (input) $\rightarrow$ input network (embedding) $\rightarrow$ $z_t$ (task representation)
According to to section 6.1 of the paper, the input is a tuple represented ...
1
This is not necessarily the only way to do this but it would be the approach I'd take.
Assuming your agents position is a vector in $\mathbb{R}^d$, then I would have the network take as input this position vector and pass it through a fully connected layer. I would also take as input the matrix and pass it through a convolutional layer(s) and flatten the ...
0
Is the second binary plane all zeros or all ones? Or, something else? How is it known if the move is off the board? For my game, I know if it is a legal move on the board, but do not know if the move is off the board.
The second binary plane is one-hot by definition, there is a single one and everything else is zero. If this definition is not met, it's no ...
1
My understanding is that you are first training a policy network using imitation learning. Then you are adjusting that trained network in some way to be a value network for DQN. The most obvious change would be to remove softmax activation whilst keeping the network layer sizes identical. This would then present Q estimates for all actions from any given ...
0
The loss function is designed in a way to approximate the bellman optimality for $Q^*(s,a)$. Given an optimal policy $\pi^*$, $Q^*(s,a)$ satisfies the equation $$Q^*(s,a) = r(s) + \gamma max_{a'}\sum_{s'}P(s'|s,a)Q^*(s',a')$$
At convergence, the highest $Q$ value that I can get taking action $a$ in state $s$ is equal to the reward I get for taking action $a$ ...
1
Some sources say MCTS (or planning in general) increases the sample efficiency.
If we're thinking purely about experiments run in simulations, then I'd estimate there may be cases where a combination of pure learning + MCTS (or some other form of planning / model-based aspect) may be more efficient, and there may be different cases where only a single one ...
1
Target network is not more stable. Both networks are the same in the regard that no one is more stable than the other. The reason for using a target network is that your current network after each step is updated. So, by not using a target network and using just the current network, after each update the rewards for many states will be modified slightly. So, ...
1
The learning model and target model are only different by N steps (typically a few thousand) out the entire taining process. If the process is near complete, they will also be quite similar.
The target model is not inherently more stable in terms of producing "correct" or "better" Q values. Instead it is kept static for a period of time ...
0
What is it?
An experience replay (ER) buffer is an array/list (or buffer) $D = [e_1, \dots, e_N ]$ where you store the transitions that the agent collects while interacting with the environment. These transitions are usually represented as tuples of the form $e_t = (s_t, a_t, r_t, s_{t+1})$, where
$s_t$ is the state of the agent at time step $t$,
$a_t$ is ...
Top 50 recent answers are included
