New answers tagged deep-rl
0
votes
How many training steps does it usually take to train an RL model?
It depends on the the problem you're applying PPO to. To get an idea, you can have a look at the CleanRL benchmarks, there are a few of them where they use PPO: https://wandb.ai/openrlbenchmark/...
- 1
2
votes
Accepted
How do I create an AI controller for Pacman?
About the environments
For the controller part of your question, I would advice looking at openAI gym.
https://www.gymlibrary.ml/content/environment_creation/ #how to make your own gym enviroment
...
- 349
1
vote
How many training steps does it usually take to train an RL model?
This is not possible to know in advance precisely, only approximately, but it also strongly depends on the environment, hyperparameters and algorithm. For hard environments, e.g. the ones learning ...
- 351
1
vote
Accepted
How to compare memory requirements for tabular Q-learning vs deep neural network?
You don't say, but I suspect from your description, that you have designed the neural network to operate over one-hot-encoding representations of states and actions. Using such a representation offers ...
- 23.8k
3
votes
Accepted
What can we learn from AlphaZero in the development towards AGI?
Some learnings from AlphaZero:
Self-play, and more generally sandbox training, is effective. This indicates that given the right enviroment and enough computational power we can build highly ...
- 351
1
vote
Replay buffer action range in DDPG
It has an obvious answer: Network is conditioned to use tanh activation. Hence the buffer should be [-1, 1], or unscaled values before action execution. As I am not using openai gym or other baselines ...
- 13
1
vote
Accepted
How to preserve Markov Property in Deep Reinforcement Learning when using "mixup" or "mixreg"?
Because each environment individually satisfies the Markov property, the distribution of the next state $s_{t+1}$ in any transition depends only on $s_t$, $a_t$, and the transition probabilities of ...
- 451
0
votes
Accepted
How to tune hypeparametes in A2C-ppo?
Human performance on Breakout is ~30, if you refer to the original DQN paper (Table 1). In the original A3C paper, it takes around 5 epochs to reach that score, so 20 millions frames (Figure 3).
Is ...
2
votes
Why is a large replay buffer inefficient?
I read the same thing recently, and my interpretation was this:
If you only use the very-most recent data, you will overfit to that and things will break
We'd like to train the network to predict ...
- 451
2
votes
Do I need to normalize all state-space variables? If so, how?
I'll start with the literal question in the title:
Do I need to normalize all state-space variables?
You don't strictly need to in theory. It's often really useful, or sometimes borderline necessary,...
- 9,369
2
votes
Accepted
Do I need to normalize all state-space variables? If so, how?
The way I've seen most codes treat the state normalization is that they simply take a running mean and standard deviation for each dimension of the state space. As you point out, this normalization ...
- 876
1
vote
Accepted
How to implement PPO without using a Critic
Your justification for not wanting to use a critic (that the episodes are short) does not make sense to me. I would expect that including the critic would result in a substantial variance reduction (...
- 876
2
votes
Do I need to normalize all state-space variables? If so, how?
It's likely to train as long as they're reasonably within the orders of magnitude of other normalized variables. The network can adjust for that.
But it might cause problems later, if the values move ...
- 451
1
vote
Accepted
How to deal with changing rewards in Q-learning? DQN?
Is my definition of 'state' or 'action' wrong?
I hesitate to say 'wrong', but that's not how state and action are defined in RL, and that mismatch might make the algorithms hard to understand.
In RL ...
- 451
7
votes
Accepted
Mathematically, what is happening differently in the neural net during exploration vs. exploitation?
Typically, the NN is trained the same way whether an action is chosen for exploration or exploitation. Look at the objective (AKA loss) function for any algorithm you're interested in and you'll ...
- 451
1
vote
Accepted
Why aren't neural networks contractions?
why neural networks aren't considered contractions, as Geoffrey J. Gordon says in his paper.
I am not sure how strongly you mean aren't considered, if you mean it in a strong sense or weak sense.
...
- 81
1
vote
Accepted
Why does the average-reward estimator for continuing tasks use the TD error?
Mystery solved thanks to Exercise 10.8 in the book. The reason is that we want the running mean to converge to the actual value of the average reward.
With $\bar{R}_{t + 1} = \bar{R}_t + \beta \delta$...
- 66
2
votes
Accepted
In RL, is it possible to design a multiplicative/exponential reward function? A reward func that depends on current accumulated reward?
The main thing you will need to do is add the accumulated reward (total_score_so_far) to the state. In order to predict future reward with any accuracy, the agent ...
- 23.8k
2
votes
Accepted
How to perform the back propagation step in Semi-gradient SARSA using a deep neural network?
If so, then would it be correct to take the gradient of this expression of $\hat{q}(S,A,w)$ with respect to the weights of the network
Yes, your expression for $\hat{q}(S,A,w)$ looks correct for your ...
- 23.8k
2
votes
Accepted
Why do we use "true labels" that are based on the output of our network in Deep Q-Learning?
The labels in DQN, and in Q-learning in general, are not "true" in the sense that they represent optimal action value functions. Instead they represent approximate action values of a current ...
- 23.8k
Top 50 recent answers are included