Neil Slater
  • Member for 5 years, 4 months
  • Last seen this week
  • Durham, United Kingdom
Should RL rewards diminish over time?
Accepted answer
5 votes

RL agents - implemented correctly - do not take previous rewards into account when making decisions. For instance value functions only assess potential future reward. The state value or expected ...

View answer
Deep Q-Learning: why don't we use mini-batches during experience reply?
Accepted answer
5 votes

DQN "library" implementations that I have seen do use mini-batches to train, and I would generally recommend this, as it usually strikes a reasonable balance between number of weight updates and ...

View answer
What is the best programming language to learn to implement genetic algorithms?
5 votes

There is no "best language" for any problem. There are too many variables to consider, even when advising a single person with a single project plan. If the choice is between Python and C++, I would ...

View answer
Does it make sense to apply softmax on top of relu?
5 votes

Does it make sense? In general, yes it is interpretable, back propagation will work, and the NN can be optimised. By using ReLU, the default network has a minimum logit of $0$ for the softmax input, ...

View answer
What does the agent in reinforcement learning exactly do?
Accepted answer
5 votes

The agent in RL is the component that makes the decision of what action to take. In order to make that decision, the agent is allowed to use any observation from the environment, and any internal ...

View answer
How do you program fear into a neural network?
5 votes

There are a lot of approaches you could take for this. Creating a realistic artificial analog for fear as implemented biologically in animals might be possible, but there is quite a lot involved in a ...

View answer
Is true random number generation an AI concept?
Accepted answer
5 votes

As it can be easily pointed out that true random numbers cannot be generated fully by programming and some random seed is required. This is true. In fact, it is impossible to solve using software. No ...

View answer
Is random initialization of the weights the only choice to break the symmetry?
Accepted answer
5 votes

Randomising just b sort of works, but setting w to all zero causes severe problems with vanishing gradients, especially at the start of learning. Using backpropagation, the gradient at the outputs ...

View answer
Is the optimal policy the one with the highest accumulative reward (Q-Learning vs SARSA)?
Accepted answer
4 votes

It is important to note that the graph shows reward received during training. This includes rewards due to exploratory moves, which sometimes involve the agent falling off the cliff, even if it has ...

View answer
Which policy do I need to use in updating Q function?
Accepted answer
4 votes

I am going to stick with Q learning here to keep things simple. Most value-based reinforcement learning used for optimal control will have some statement similar to: Choose $a$ from $s$ using policy ...

View answer
What is the effect of parallel environments in reinforcement learning?
Accepted answer
4 votes

Do parallel environments improve the agent's ability to learn or does it not really make a difference? Yes they can make a difference. There are two ways improvement is seen: Collecting data from ...

View answer
What exactly are partially observable environments?
4 votes

You are correct in the question that in RL terms chess a game of chess where the agent is one player, and the other player has an unknown state is a partially observable environment. Chess played like ...

View answer
Can a large discrete action space be represented using Gaussian distributions?
Accepted answer
4 votes

The answer is "it depends". Once you have arranged the actions into order, a key trait is whether the action value function has a simple enough shape that sampling from a Gaussian policy ...

View answer
How should I generate datasets for a SARSA agent when the environment is not simple?
Accepted answer
4 votes

I am wondering how to generate datasets when the environment is not as simple as a tic-tac-toe or a maze problem There is no difference in concept, which is why tic-tac-toe and maze problems are used ...

View answer
What constitutes a large space state (in Q-learning)?
4 votes

I know this might be specific to different problems but does anyone know if there is any rule of thumb or references on what constitutes a large state space? Not really, it is all relative. There are ...

View answer
How do we compute the target value when the agent ends up in the terminal state?
Accepted answer
4 votes

Now, let us assume the agent is in the penultimate state, $S_1$, and chooses the action $A$ that leads him to the completion state, $S_2$, and gets a reward $R$. How do we form the target value $Q_\...

View answer
Is the state transition matrix known to the agents in a Markov decision processes?
Accepted answer
4 votes

In reinforcement learning (RL), there are some agents that need to know the state transition probabilities, and other agents that do not need to know. In addition, some agents may need to be able to ...

View answer
How do we define the reward function for an environment?
Accepted answer
4 votes

In Reinforcement Learning (RL), a reward function is part of the problem definition and should: Be based primarily on the goals of the agent. Take into account any combination of starting state $s$, ...

View answer
Why is regret so defined in MABs?
Accepted answer
4 votes

In short, you don't regret your bad luck that you could do nothing about, you regret your bad choices that you could have done something about if only you knew. The point of regret as a metric ...

View answer
What happens when you select actions using softmax instead of epsilon greedy in DQN?
Accepted answer
4 votes

DQN on the other hand, explores using epsilon greedy exploration. Either selecting the best action or a random action. This is a very common choice, because it is simple to implement and quite robust....

View answer
How to express $v_\pi(s)$ in terms of $q_\pi(s,a)$?
Accepted answer
4 votes

isn't then $v_\pi(s)$ just the expected action value function at $s$ over all actions $a$ that are given by the policy $\pi$, namely $v_\pi(s) = E_{a \sim \pi}[q_\pi(s,a) | S_t = s, A_t = a] = \sum_{...

View answer
Is the PyTorch official tutorial really about Q-learning?
4 votes

TL;DR: It is Q learning. However Q learning is basically sample-based value iteration, so not surprising you see a similarity. Q learning* and value iteration are very strongly related. When ...

View answer
Why not replacing the simple linear functions that neurons compute with more complex functions?
Accepted answer
4 votes

It is definitley possible to make the links between neurons use more complex functions. Provided those functions are differentiable, backpropagation still works, and the resulting compound function ...

View answer
Why is update rule of the value function different in policy evaluation and policy iteration?
Accepted answer
4 votes

Yes, the two update equations are equivalent. As an aside, technically the equation you give is not the Bellman equation, but the update step re-written as an equation - in the Bellman equation ...

View answer
In RL, if I assign the rewards for better positional play, the algorithm is learning nothing?
4 votes

What you are proposing is closer to a heuristic for searching than a reward for RL. This is a blurred line, but generally if you start analysing the problem yourself, breaking it down into components ...

View answer
Finding the optimal combination of inputs which return maximal output
4 votes

If your model is gradient-based, such as a neural network, then may also be able to use gradient methods to drive virtual inputs: Freeze all network weights to the trained version Define a loss ...

View answer
Does using the softmax function in Q learning not defeat the purpose of Q learning?
Accepted answer
4 votes

However, to me, it makes intuitive sense to have the final layer of the network be a softmax function for some games. This is because in a lot of cases (like Go for example), only one "move" can be ...

View answer
What is the double sample problem in reinforcement learning?
4 votes

The double sampling problem is referenced in Chaper 11.5 Gradient Descent in the Bellman Error in Reinforcement Learning: An Introduction (2nd edition). From the book, this is the full gradient ...

View answer
Confusion about temporal difference learning
Accepted answer
4 votes

My first question is whether the following "implementation" of the 𝑇𝐷(0) algorithm for the first two of the above observed trajectories correct? $V(a)\leftarrow0 + 0.1(1+0-0)= 0.1; \quad ...

View answer
How can we find find the input image which maximizes the class-probability for an ANN?
4 votes

Probably the simplest way to search for an image with the highest probability of being a cat is to use a technique similar to Deep Dream: Load the network for training, but freeze all the network ...

View answer
1 2
3
4 5
20