Hot answers tagged

7 votes
Accepted

Mathematically, what is happening differently in the neural net during exploration vs. exploitation?

Typically, the NN is trained the same way whether an action is chosen for exploration or exploitation. Look at the objective (AKA loss) function for any algorithm you're interested in and you'll ...
user avatar
5 votes
Accepted

Why do Q-values diverge without a target network?

I don't see how the target Q-value gets updated when the current Q-value is changed. Without a separate target network, this happens because the approximator will generalise, and the generalisation ...
user avatar
  • 24.5k
4 votes
Accepted

How do I design the network for Deep Q-Network?

What is the strategy to get to a better network? There are a few different strategies that you can use to search for good hyperparameters in reinforcement learning RL, but you should be aware that ...
user avatar
  • 24.5k
3 votes

What is the difference between a loss function and reward/penalty in Deep Reinforcement Learning?

1. Question: The difference between loss and reward/penalty So I see both the loss function and the reward/penalty are the quantitative way of measuring the output/action and making the model to ...
user avatar
  • 671
3 votes

What is the difference between a loss function and reward/penalty in Deep Reinforcement Learning?

Ultimately, in RL, the policy is what you want to find. It's the solution to the Markov Decision Process (MDP). But you don't want to find any policy, but the optimal policy, i.e. the one that will ...
user avatar
  • 34.4k
3 votes
Accepted

What can we learn from AlphaZero in the development towards AGI?

Some learnings from AlphaZero: Self-play, and more generally sandbox training, is effective. This indicates that given the right enviroment and enough computational power we can build highly ...
user avatar
  • 361
3 votes
Accepted

Determining to terminate at a reward or not

I am trying to understand how you would determine whether it is better for the agent to terminate at the state with the number 3 or to continue to the state with a number 4 to collect the more reward? ...
user avatar
  • 24.5k
3 votes

How to deal with small reward values

The numbers that a value-based neural network will predict are usually based on expected returns (sum of rewards by end of an episode, or a discounted infinite sum), although in some cases they might ...
user avatar
  • 24.5k
2 votes
Accepted

What do equations 1 and 3 describe in the "Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels" paper?

Equation 1 In normal Q-Learning your target is defined as $y_t = r_t + \gamma \mathrm{max_a}Q(s_{t+1}, a)$. Since you're training a regularized version, you construct the estimated value of the next ...
user avatar
  • 348
2 votes

Should I apply normalization to the observations in deep reinforcement learning?

The use of normalisation in neural networks and many other (but not all - decision trees are a notable exception) machine learning methods, is to improve the quality of the parameter space with ...
user avatar
  • 24.5k
2 votes
Accepted

Why exactly was previously believed that the deterministic policy gradient did not exist?

Actually, your result that the gradient is 0 is correct given your formulation. Indeed, that is why one might have believed that the deterministic policy gradient didn't exist. The term $\nabla_{\...
user avatar
  • 931
2 votes

Does AlphaGo play random moves in a real competition?

Question 1: I don't think they ran AlphaGo or AlphaGoZero in training mode during tournament matches because the computing power required for this is really large. I don't recall if this is described ...
user avatar
  • 179
2 votes
Accepted

Is it the high probability action that is always selected by the agent in REINFORCE algorithm?

You sample according to the probability distribution $\pi(a \mid s, \theta)$, so you do not always take the action with the highest probability (otherwise there would be no exploration but just ...
user avatar
  • 34.4k
2 votes
Accepted

Are the Q-values of DQN bounded at a single timestep?

Yes, I believe you can. Assume that you want to upper bound your difference to $k$. Use the following function: $$ y_{t}^{\pi} = \frac{k}{2}*\tanh(Q_{t}^{\pi}(s,a)) $$ Here, $y_{t}^{\pi} \in [-\frac{k}...
user avatar
2 votes
Accepted

How should I initialize the weights of the neural network so that the initial policy is uniform?

You may be interested in section 3.2 of this paper What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study (2020) by Google Research. They claim that the initialization of the ...
user avatar
  • 931
2 votes

How Come My (D)DQN Fails To Learn?

I have two suggestions that you can look into. Based on my own work in RL, I believe the first one will require less work to implement. If the observability of the environment is not an issue, then ...
user avatar
  • 179
2 votes

Do I need to normalize all state-space variables? If so, how?

It's likely to train as long as they're reasonably within the orders of magnitude of other normalized variables. The network can adjust for that. But it might cause problems later, if the values move ...
user avatar
2 votes
Accepted

Do I need to normalize all state-space variables? If so, how?

The way I've seen most codes treat the state normalization is that they simply take a running mean and standard deviation for each dimension of the state space. As you point out, this normalization ...
user avatar
  • 931
2 votes

Do I need to normalize all state-space variables? If so, how?

I'll start with the literal question in the title: Do I need to normalize all state-space variables? You don't strictly need to in theory. It's often really useful, or sometimes borderline necessary,...
user avatar
  • 9,419
2 votes
Accepted

Why do we use "true labels" that are based on the output of our network in Deep Q-Learning?

The labels in DQN, and in Q-learning in general, are not "true" in the sense that they represent optimal action value functions. Instead they represent approximate action values of a current ...
user avatar
  • 24.5k
2 votes
Accepted

How to perform the back propagation step in Semi-gradient SARSA using a deep neural network?

If so, then would it be correct to take the gradient of this expression of $\hat{q}(S,A,w)$ with respect to the weights of the network Yes, your expression for $\hat{q}(S,A,w)$ looks correct for your ...
user avatar
  • 24.5k
2 votes
Accepted

In RL, is it possible to design a multiplicative/exponential reward function? A reward func that depends on current accumulated reward?

The main thing you will need to do is add the accumulated reward (total_score_so_far) to the state. In order to predict future reward with any accuracy, the agent ...
user avatar
  • 24.5k
2 votes

Why is a large replay buffer inefficient?

I read the same thing recently, and my interpretation was this: If you only use the very-most recent data, you will overfit to that and things will break We'd like to train the network to predict ...
user avatar
2 votes
Accepted

How do I create an AI controller for Pacman?

About the environments For the controller part of your question, I would advice looking at openAI gym. https://www.gymlibrary.ml/content/environment_creation/ #how to make your own gym enviroment ...
user avatar
  • 349
2 votes
Accepted

How to sample the tuples during the initial time steps of the DDPG algorithm?

You have a free choice to either: Wait until the replay buffer hits a minimum size for sampling. Take smaller samples from the buffer initially, until the buffer is large enough. On the first time ...
user avatar
  • 24.5k
2 votes
Accepted

Which paper describes the effect of learning_starts in Reinforcement Learning?

The replay buffer allows breaking the temporal dependence of the data and thus makes them more i.i.d. (which is what we want). The replay buffer needs to be at least filled with enough experience to ...
user avatar
1 vote

How to take gradient of log policy when actions are negative?

The policy gradient tells us that $$\nabla_\theta v_\pi(s) = \mathbb{E}_\pi\left[ G_t \nabla_\theta \log \pi_\theta(a | s) \right] \; ;$$ where as usual $G_t$ are the returns, $\pi$ is policy ...
user avatar
1 vote
Accepted

How to define actions on a list of values?

I had tried working on a problem similar to this using combinatorial scoring games. I ran into other issues with the players competing, but I think I can give some advice to how I handled this. In my ...
user avatar
  • 46
1 vote
Accepted

Why does my actor-critic network always give either -1 or 1 at the output layer?

Most probably your network is underfitted. In that case, the network outputs values randomly. Hyperbolic tangent tanh converges very quickly towards $-1$ or $1$, so that is why you always find $-1$ ...
user avatar
  • 522
1 vote

Does AlphaGo play random moves in a real competition?

The core mechanics of AlphaZero during selfplay and real tournament games are the same: something similar to Monte Carlo Tree Search is done but guided by the current neural network instead of random ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible