# Tag Info

### How many training steps does it usually take to train an RL model?

It depends on the the problem you're applying PPO to. To get an idea, you can have a look at the CleanRL benchmarks, there are a few of them where they use PPO: https://wandb.ai/openrlbenchmark/...
Accepted

### How do I create an AI controller for Pacman?

About the environments For the controller part of your question, I would advice looking at openAI gym. https://www.gymlibrary.ml/content/environment_creation/ #how to make your own gym enviroment ...
• 349
1 vote

### How many training steps does it usually take to train an RL model?

This is not possible to know in advance precisely, only approximately, but it also strongly depends on the environment, hyperparameters and algorithm. For hard environments, e.g. the ones learning ...
• 351
1 vote
Accepted

### How to compare memory requirements for tabular Q-learning vs deep neural network?

You don't say, but I suspect from your description, that you have designed the neural network to operate over one-hot-encoding representations of states and actions. Using such a representation offers ...
• 23.8k
Accepted

### What can we learn from AlphaZero in the development towards AGI?

Some learnings from AlphaZero: Self-play, and more generally sandbox training, is effective. This indicates that given the right enviroment and enough computational power we can build highly ...
• 351
1 vote

### Replay buffer action range in DDPG

It has an obvious answer: Network is conditioned to use tanh activation. Hence the buffer should be [-1, 1], or unscaled values before action execution. As I am not using openai gym or other baselines ...
1 vote
Accepted

### How to preserve Markov Property in Deep Reinforcement Learning when using "mixup" or "mixreg"?

Because each environment individually satisfies the Markov property, the distribution of the next state $s_{t+1}$ in any transition depends only on $s_t$, $a_t$, and the transition probabilities of ...
• 451
Accepted

### How to tune hypeparametes in A2C-ppo?

Human performance on Breakout is ~30, if you refer to the original DQN paper (Table 1). In the original A3C paper, it takes around 5 epochs to reach that score, so 20 millions frames (Figure 3). Is ...

### Why is a large replay buffer inefficient?

I read the same thing recently, and my interpretation was this: If you only use the very-most recent data, you will overfit to that and things will break We'd like to train the network to predict ...
• 451

### Do I need to normalize all state-space variables? If so, how?

I'll start with the literal question in the title: Do I need to normalize all state-space variables? You don't strictly need to in theory. It's often really useful, or sometimes borderline necessary,...
• 9,369
Accepted

### Do I need to normalize all state-space variables? If so, how?

The way I've seen most codes treat the state normalization is that they simply take a running mean and standard deviation for each dimension of the state space. As you point out, this normalization ...
• 876
1 vote
Accepted

### How to implement PPO without using a Critic

Your justification for not wanting to use a critic (that the episodes are short) does not make sense to me. I would expect that including the critic would result in a substantial variance reduction (...
• 876

### Do I need to normalize all state-space variables? If so, how?

It's likely to train as long as they're reasonably within the orders of magnitude of other normalized variables. The network can adjust for that. But it might cause problems later, if the values move ...
• 451
1 vote
Accepted

### How to deal with changing rewards in Q-learning? DQN?

Is my definition of 'state' or 'action' wrong? I hesitate to say 'wrong', but that's not how state and action are defined in RL, and that mismatch might make the algorithms hard to understand. In RL ...
• 451
Accepted

### Mathematically, what is happening differently in the neural net during exploration vs. exploitation?

Typically, the NN is trained the same way whether an action is chosen for exploration or exploitation. Look at the objective (AKA loss) function for any algorithm you're interested in and you'll ...
• 451
1 vote
Accepted

### Why aren't neural networks contractions?

why neural networks aren't considered contractions, as Geoffrey J. Gordon says in his paper. I am not sure how strongly you mean aren't considered, if you mean it in a strong sense or weak sense. ...
• 81
1 vote
Accepted

### Why does the average-reward estimator for continuing tasks use the TD error?

Mystery solved thanks to Exercise 10.8 in the book. The reason is that we want the running mean to converge to the actual value of the average reward. With $\bar{R}_{t + 1} = \bar{R}_t + \beta \delta$...
• 66
Accepted

### In RL, is it possible to design a multiplicative/exponential reward function? A reward func that depends on current accumulated reward?

The main thing you will need to do is add the accumulated reward (total_score_so_far) to the state. In order to predict future reward with any accuracy, the agent ...
• 23.8k
Accepted

### How to perform the back propagation step in Semi-gradient SARSA using a deep neural network?

If so, then would it be correct to take the gradient of this expression of $\hat{q}(S,A,w)$ with respect to the weights of the network Yes, your expression for $\hat{q}(S,A,w)$ looks correct for your ...
• 23.8k