New answers tagged

0 votes

How many training steps does it usually take to train an RL model?

It depends on the the problem you're applying PPO to. To get an idea, you can have a look at the CleanRL benchmarks, there are a few of them where they use PPO: https://wandb.ai/openrlbenchmark/...
user avatar
  • 1
2 votes
Accepted

How do I create an AI controller for Pacman?

About the environments For the controller part of your question, I would advice looking at openAI gym. https://www.gymlibrary.ml/content/environment_creation/ #how to make your own gym enviroment ...
user avatar
  • 349
1 vote

How many training steps does it usually take to train an RL model?

This is not possible to know in advance precisely, only approximately, but it also strongly depends on the environment, hyperparameters and algorithm. For hard environments, e.g. the ones learning ...
user avatar
  • 351
1 vote
Accepted

How to compare memory requirements for tabular Q-learning vs deep neural network?

You don't say, but I suspect from your description, that you have designed the neural network to operate over one-hot-encoding representations of states and actions. Using such a representation offers ...
user avatar
  • 23.8k
3 votes
Accepted

What can we learn from AlphaZero in the development towards AGI?

Some learnings from AlphaZero: Self-play, and more generally sandbox training, is effective. This indicates that given the right enviroment and enough computational power we can build highly ...
user avatar
  • 351
1 vote

Replay buffer action range in DDPG

It has an obvious answer: Network is conditioned to use tanh activation. Hence the buffer should be [-1, 1], or unscaled values before action execution. As I am not using openai gym or other baselines ...
user avatar
1 vote
Accepted

How to preserve Markov Property in Deep Reinforcement Learning when using "mixup" or "mixreg"?

Because each environment individually satisfies the Markov property, the distribution of the next state $s_{t+1}$ in any transition depends only on $s_t$, $a_t$, and the transition probabilities of ...
user avatar
0 votes
Accepted

How to tune hypeparametes in A2C-ppo?

Human performance on Breakout is ~30, if you refer to the original DQN paper (Table 1). In the original A3C paper, it takes around 5 epochs to reach that score, so 20 millions frames (Figure 3). Is ...
user avatar
2 votes

Why is a large replay buffer inefficient?

I read the same thing recently, and my interpretation was this: If you only use the very-most recent data, you will overfit to that and things will break We'd like to train the network to predict ...
user avatar
2 votes

Do I need to normalize all state-space variables? If so, how?

I'll start with the literal question in the title: Do I need to normalize all state-space variables? You don't strictly need to in theory. It's often really useful, or sometimes borderline necessary,...
user avatar
  • 9,369
2 votes
Accepted

Do I need to normalize all state-space variables? If so, how?

The way I've seen most codes treat the state normalization is that they simply take a running mean and standard deviation for each dimension of the state space. As you point out, this normalization ...
user avatar
  • 876
1 vote
Accepted

How to implement PPO without using a Critic

Your justification for not wanting to use a critic (that the episodes are short) does not make sense to me. I would expect that including the critic would result in a substantial variance reduction (...
user avatar
  • 876
2 votes

Do I need to normalize all state-space variables? If so, how?

It's likely to train as long as they're reasonably within the orders of magnitude of other normalized variables. The network can adjust for that. But it might cause problems later, if the values move ...
user avatar
1 vote
Accepted

How to deal with changing rewards in Q-learning? DQN?

Is my definition of 'state' or 'action' wrong? I hesitate to say 'wrong', but that's not how state and action are defined in RL, and that mismatch might make the algorithms hard to understand. In RL ...
user avatar
7 votes
Accepted

Mathematically, what is happening differently in the neural net during exploration vs. exploitation?

Typically, the NN is trained the same way whether an action is chosen for exploration or exploitation. Look at the objective (AKA loss) function for any algorithm you're interested in and you'll ...
user avatar
1 vote
Accepted

Why aren't neural networks contractions?

why neural networks aren't considered contractions, as Geoffrey J. Gordon says in his paper. I am not sure how strongly you mean aren't considered, if you mean it in a strong sense or weak sense. ...
user avatar
  • 81
1 vote
Accepted

Why does the average-reward estimator for continuing tasks use the TD error?

Mystery solved thanks to Exercise 10.8 in the book. The reason is that we want the running mean to converge to the actual value of the average reward. With $\bar{R}_{t + 1} = \bar{R}_t + \beta \delta$...
user avatar
  • 66
2 votes
Accepted

In RL, is it possible to design a multiplicative/exponential reward function? A reward func that depends on current accumulated reward?

The main thing you will need to do is add the accumulated reward (total_score_so_far) to the state. In order to predict future reward with any accuracy, the agent ...
user avatar
  • 23.8k
2 votes
Accepted

How to perform the back propagation step in Semi-gradient SARSA using a deep neural network?

If so, then would it be correct to take the gradient of this expression of $\hat{q}(S,A,w)$ with respect to the weights of the network Yes, your expression for $\hat{q}(S,A,w)$ looks correct for your ...
user avatar
  • 23.8k
2 votes
Accepted

Why do we use "true labels" that are based on the output of our network in Deep Q-Learning?

The labels in DQN, and in Q-learning in general, are not "true" in the sense that they represent optimal action value functions. Instead they represent approximate action values of a current ...
user avatar
  • 23.8k

Top 50 recent answers are included