# Tag Info

## Hot answers tagged deep-rl

Accepted

### Mathematically, what is happening differently in the neural net during exploration vs. exploitation?

Typically, the NN is trained the same way whether an action is chosen for exploration or exploitation. Look at the objective (AKA loss) function for any algorithm you're interested in and you'll ...
• 451
Accepted

### Why do Q-values diverge without a target network?

I don't see how the target Q-value gets updated when the current Q-value is changed. Without a separate target network, this happens because the approximator will generalise, and the generalisation ...
• 24.5k
Accepted

### How do I design the network for Deep Q-Network?

What is the strategy to get to a better network? There are a few different strategies that you can use to search for good hyperparameters in reinforcement learning RL, but you should be aware that ...
• 24.5k

### What is the difference between a loss function and reward/penalty in Deep Reinforcement Learning?

1. Question: The difference between loss and reward/penalty So I see both the loss function and the reward/penalty are the quantitative way of measuring the output/action and making the model to ...
• 671

### What is the difference between a loss function and reward/penalty in Deep Reinforcement Learning?

Ultimately, in RL, the policy is what you want to find. It's the solution to the Markov Decision Process (MDP). But you don't want to find any policy, but the optimal policy, i.e. the one that will ...
• 34.4k
Accepted

### What can we learn from AlphaZero in the development towards AGI?

Some learnings from AlphaZero: Self-play, and more generally sandbox training, is effective. This indicates that given the right enviroment and enough computational power we can build highly ...
• 361
Accepted

### Determining to terminate at a reward or not

I am trying to understand how you would determine whether it is better for the agent to terminate at the state with the number 3 or to continue to the state with a number 4 to collect the more reward? ...
• 24.5k

### How to deal with small reward values

The numbers that a value-based neural network will predict are usually based on expected returns (sum of rewards by end of an episode, or a discounted infinite sum), although in some cases they might ...
• 24.5k
Accepted

### What do equations 1 and 3 describe in the "Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels" paper?

Equation 1 In normal Q-Learning your target is defined as $y_t = r_t + \gamma \mathrm{max_a}Q(s_{t+1}, a)$. Since you're training a regularized version, you construct the estimated value of the next ...
• 348

### Should I apply normalization to the observations in deep reinforcement learning?

The use of normalisation in neural networks and many other (but not all - decision trees are a notable exception) machine learning methods, is to improve the quality of the parameter space with ...
• 24.5k
Accepted

• 136
Accepted

### How should I initialize the weights of the neural network so that the initial policy is uniform?

You may be interested in section 3.2 of this paper What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study (2020) by Google Research. They claim that the initialization of the ...
• 931

### How Come My (D)DQN Fails To Learn?

I have two suggestions that you can look into. Based on my own work in RL, I believe the first one will require less work to implement. If the observability of the environment is not an issue, then ...
• 179

### Do I need to normalize all state-space variables? If so, how?

It's likely to train as long as they're reasonably within the orders of magnitude of other normalized variables. The network can adjust for that. But it might cause problems later, if the values move ...
• 451
Accepted

### Do I need to normalize all state-space variables? If so, how?

The way I've seen most codes treat the state normalization is that they simply take a running mean and standard deviation for each dimension of the state space. As you point out, this normalization ...
• 931

### Do I need to normalize all state-space variables? If so, how?

I'll start with the literal question in the title: Do I need to normalize all state-space variables? You don't strictly need to in theory. It's often really useful, or sometimes borderline necessary,...
• 9,419
Accepted

### Why do we use "true labels" that are based on the output of our network in Deep Q-Learning?

The labels in DQN, and in Q-learning in general, are not "true" in the sense that they represent optimal action value functions. Instead they represent approximate action values of a current ...
• 24.5k
Accepted

### How to perform the back propagation step in Semi-gradient SARSA using a deep neural network?

If so, then would it be correct to take the gradient of this expression of $\hat{q}(S,A,w)$ with respect to the weights of the network Yes, your expression for $\hat{q}(S,A,w)$ looks correct for your ...
• 24.5k
Accepted

### In RL, is it possible to design a multiplicative/exponential reward function? A reward func that depends on current accumulated reward?

The main thing you will need to do is add the accumulated reward (total_score_so_far) to the state. In order to predict future reward with any accuracy, the agent ...
• 24.5k

### Why is a large replay buffer inefficient?

I read the same thing recently, and my interpretation was this: If you only use the very-most recent data, you will overfit to that and things will break We'd like to train the network to predict ...
• 451
Accepted

### How do I create an AI controller for Pacman?

About the environments For the controller part of your question, I would advice looking at openAI gym. https://www.gymlibrary.ml/content/environment_creation/ #how to make your own gym enviroment ...
• 349
Accepted

### How to sample the tuples during the initial time steps of the DDPG algorithm?

You have a free choice to either: Wait until the replay buffer hits a minimum size for sampling. Take smaller samples from the buffer initially, until the buffer is large enough. On the first time ...
• 24.5k
Accepted

### Which paper describes the effect of learning_starts in Reinforcement Learning?

The replay buffer allows breaking the temporal dependence of the data and thus makes them more i.i.d. (which is what we want). The replay buffer needs to be at least filled with enough experience to ...
1 vote

### How to take gradient of log policy when actions are negative?

The policy gradient tells us that $$\nabla_\theta v_\pi(s) = \mathbb{E}_\pi\left[ G_t \nabla_\theta \log \pi_\theta(a | s) \right] \; ;$$ where as usual $G_t$ are the returns, $\pi$ is policy ...
• 4,180
1 vote
Accepted

### How to define actions on a list of values?

I had tried working on a problem similar to this using combinatorial scoring games. I ran into other issues with the players competing, but I think I can give some advice to how I handled this. In my ...
• 46
1 vote
Accepted

### Why does my actor-critic network always give either -1 or 1 at the output layer?

Most probably your network is underfitted. In that case, the network outputs values randomly. Hyperbolic tangent tanh converges very quickly towards $-1$ or $1$, so that is why you always find $-1$ ...
• 522
1 vote

### Does AlphaGo play random moves in a real competition?

The core mechanics of AlphaZero during selfplay and real tournament games are the same: something similar to Monte Carlo Tree Search is done but guided by the current neural network instead of random ...
• 146

Only top scored, non community-wiki answers of a minimum length are eligible