Neil Slater
  • Member for 5 years, 5 months
  • Last seen this week
  • Durham, United Kingdom
How can the sum of squared errors have negative gradient if it's defined as the squared of the error?
Accepted answer
3 votes

If $t^i - o^i$ is negative, doesn't the power of 2 eliminate any negative result? In the loss function, yes that is correct, and is what you want - a measurement that gets higher due to any ...

View answer
What is a good convergence criterion for Q-learning in a stochastic environment?
Accepted answer
3 votes

To obtain guarantees of convergence for Q table values, you need to decay the learning rate, $\alpha$, at a suitable rate. Too fast and convergence will be to inaccurate values. Too slow and ...

View answer
Should we feed a greater fraction of terminal states to the value network so that their values are learned first?
Accepted answer
3 votes

The basis of Q-learning is recursive (similar to dynamic programming), where only the absolute value of the terminal state is known. This may be true in some environments. Many environments do not ...

View answer
What is the relation between the context in contextual bandits and the state in reinforcement learning?
3 votes

Conceptually, in general, how is the context being handled in CB, compared to states in RL? In terms of its place in the description of Contextual Bandits and Reinforcement Learning, context in CB is ...

View answer
Can the rewards be matrices when using DQN?
Accepted answer
3 votes

Generally speaking, is it better for rewards to be a scalar, or is using matrices okay? Rewards need to be scalar, real values to match to standard theory of Markov decision processes (MDPs) and ...

View answer
What is a "learned policy" in Q-learning?
Accepted answer
3 votes

A Q table allows you to look up any state/action pair in it and find the associated action value. It is not itself a policy. However, in order to calculate the action values, you will have assumed ...

View answer
What is the target Q-value in DQNs?
3 votes

What does the target Q-values represent? In a DQN, which uses off-policy learning, they represent a refined estimate for the expected future reward from taking an action $a$ in state $s$, and from ...

View answer
Is a team of ML scientists an "intelligent agent"?
3 votes

Is there a term for the humans who do [machine] learning? Typically you will see "AI researchers" for people studying machine intelligence in general, or "data scientists" for ...

View answer
Is the expected value we sample in TD-learning action-value Q or state-value V?
Accepted answer
3 votes

However, from the blogs and texts I read, the equations are expressed in terms of V and NOT Q. Why is that? MC and TD are methods for associating value estimates to time step based on experienced ...

View answer
In DQN, is it possible to make some actions more likely?
Accepted answer
3 votes

For single-step Q learning, the behaviour policy can be any stochastic policy without any further adjustment to the update rules. You don't have to use $\epsilon$-greedy based on current Q function ...

View answer
How to implement exploration function and learning rate in Q Learning
3 votes

Your main problem is that you need to separate out what is driving the behaviour policy from the Q-table. Q Learning is an off-policy algorithm. The Q-table that it eventually learns is for an optimal ...

View answer
In reinforcement learning, is the value of terminal/goal state always zero?
Accepted answer
3 votes

In reinforcement learning, is the value of terminal/goal state always zero? Yes, always for episodic problems, the value of a terminal state is always zero, from the definition. The value of a state $...

View answer
In reinforcement learning, does the optimal value correspond to performing the best action in a given state?
Accepted answer
3 votes

I am wondering which definition is correct. The asterisk * in both the definitions stands for "optimal" in the sense of "value when following the optimal policy" So this one is ...

View answer
Why would the lookup table (of a table-driven artificial agent) need to store data at pixel precision?
Accepted answer
3 votes

A tabular system for agent decisions is a direct and simple map of percept to control choice. For each percept received, the agent looks up the percept and cross-references it to the action it should ...

View answer
How should I take into consideration the number of steps in the reward function?
Accepted answer
3 votes

How would you implement this "Number of Steps" cost? What the paper is referring to is the reward discounting process which is a standard way of formulating RL problems, either continuous ...

View answer
What does $r : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$ mean in the article Hindsight Experience Replay, section 2.1?
Accepted answer
3 votes

This answer assumes that you only have a problem with this notation from the article: $r : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$ This is a standard notation, used in many ...

View answer
Why do we need to go back to policy evaluation after policy improvement if the policy is not stable?
3 votes

There is a difference between accurate value function estimates, and optimal value functions. An optimal value function is more specifically the value function of an optimal policy. Value functions ...

View answer
What's a good neural network for this problem?
Accepted answer
3 votes

A simple feed-forward neural network with at least one hidden layer would suffice in your problem, and can deal with arbitrary non-linear relationships between input and output. If you expect ...

View answer
Why doesn't value iteration use $\pi(a \mid s)$ while policy evaluation does?
Accepted answer
3 votes

You appear to comparing the value table update steps in policy iteration and value iteration, which are both derived from Bellman equations. Policy iteration In policy iteration, a policy lookup table ...

View answer
How should I compute the target for updating in a DQN at the terminal state if I have pseudo-episodes?
Accepted answer
3 votes

If the episode does not terminate naturally, then if you are breaking it up into pseudo-episodes for training purposes, the one thing you should not do is use the TD target $G_{T-1} = R_T$ used for an ...

View answer
Why is tic-tac-toe considered a non-deterministic environment?
Accepted answer
3 votes

The game of TIC-TAC-TOE can be modelled as a non-deterministic Markov decision process (MDP) if, and only if: The opponent is considered part of the environment. This is a reasonable approach when ...

View answer
What is meant by "generate the data" in describing the difference between on-policy and off-policy?
Accepted answer
3 votes

In the book, the phrase "generate the data" refers to the data from observations about states, actions, next states and rewards, that then get used to make value estimate updates. In both ...

View answer
How is the reward in reinforcement learning different from the label in supervised learning problems?
Accepted answer
3 votes

Reward in reinforcement learning (RL) is entirely different from a supervised learning (SL) label, but can be related to it indirectly. In a RL control setting, you can imagine that you had a data ...

View answer
Why is 100% exploration bad during the learning stage in reinforcement learning?
Accepted answer
3 votes

Why can’t we during the first 1000 episodes allow our agent perform only exploration You can do this. It is fine to do so either to learn the value function of a simple random policy, or when ...

View answer
Two DQNs in two different time scales
Accepted answer
3 votes

From comments, you say there is no "outer" goal for picking an adversary other than scoring highly in an individual episode. You could potentially model the initial adversary choice as a ...

View answer
Equation not satisfied in Policy Iteration Algorithm
Accepted answer
3 votes

Your calculations are correct, but you have misinterpreted the equations and the diagram. The index $k$ in $v_k$ for the diagram refers to the policy evaluation update iteration only, and is not ...

View answer
Why is $G_{t+1}$ is replaced with $v_*(S_{t+1})$ in the Bellman optimality equation?
Accepted answer
3 votes

Can someone provide the reasoning behind why $G_{t+1}$ is equal to $v_*(S_{t+1})$? The two things are not usually exactly equal, because $G_{t+1}$ is a probability distribution over all possible ...

View answer
If deep Q-learning starts to choose only one action, is this a sign that the algorithm diverged?
Accepted answer
3 votes

Is this a sign that the algorithm diverged? It is a common sign of a problem with learning process. That includes divergence due to poor hyper-parameters, even just bad luck. But it can also point to ...

View answer
Is there a general file type associated with AI projects?
3 votes

No, there is no file type associated with AI projects in general. Your examples of Photoshop and Excel are specific corporate branded products. These store bespoke data that only works with those ...

View answer
How would researchers determine the best deep learning model if every run of the code yields different results?
Accepted answer
3 votes

I know that a seed can be set to incorporate more determinism into the training. However, there could be other pseudo-random sequences that produce slightly better results? That is correct. If you ...

View answer
1
3 4
5
6 7
20