Search Results
| Search type | Search syntax |
|---|---|
| Tags | [tag] |
| Exact | "words here" |
| Author |
user:1234 user:me (yours) |
| Score |
score:3 (3+) score:0 (none) |
| Answers |
answers:3 (3+) answers:0 (none) isaccepted:yes hasaccepted:no inquestion:1234 |
| Views | views:250 |
| Code | code:"if (foo != bar)" |
| Sections |
title:apples body:"apples oranges" |
| URL | url:"*.example.com" |
| Saves | in:saves |
| Status |
closed:yes duplicate:no migrated:no wiki:no |
| Types |
is:question is:answer |
| Exclude |
-[tag] -apples |
| For more details on advanced search visit our help page | |
Results tagged with q-learning
Search options not deleted
user 36821
For questions related to the Q-learning algorithm, which is a model-free and temporal-difference reinforcement learning algorithm that attempts to approximate the Q function, which is a function that, given a state s and an action a, returns a real number that represents the return (or value) of state s when action a is taken from s. Q-learning was introduced in the PhD thesis "Learning from Delayed Rewards" (1989) by Watkins.
11
votes
Accepted
What exactly is the advantage of double DQN over DQN?
In $Q$-learning there is what is known as a maximisation bias. That is because the update target is $r + \gamma \max_a Q(s,a)$. If you slightly overestimate your $Q$-value then this error gets compoun …
- 5,030
9
votes
Accepted
What is the difference between Q-learning, Deep Q-learning and Deep Q-network?
In Q-learning (and in general value based reinforcement learning) we are typically interested in learning a Q-function, $Q(s, a)$. This is defined as
$$Q(s, a) = \mathbb{E}_\pi\left[ G_t | S_t = s, A_ …
- 5,030
8
votes
Are Q-learning and SARSA the same when action selection is greedy?
If we write the pseudo-code for the SARSA algorithm we first initialise our hyper-parameters etc. and then initialise $S_t$, which we use to choose $A_t$ from our policy $\pi(a|s)$. Then for each $t$ …
- 5,030
7
votes
Why do we need target network in deep Q learning?
In DQN that was presented in the original paper the update target for the Q-Network is $\left(r_t + \max_aQ(s_{t+1},a;\theta^-) - Q(s_t,a_t; \theta)\right)^2$ were $\theta^-$ is some old version of th …
- 5,030
5
votes
Accepted
When do SARSA and Q-Learning converge to optimal Q values?
The true answers are 1 and 3.
1 is true because the required conditions for tabular Q-learning to converge is that each state action pair will be visited infinitely often, and Q-learning learns direct …
- 5,030
5
votes
Accepted
Why does Q-learning converge under 100% exploration rate?
Q-learning is guaranteed to converge (in the tabular case) under some mild conditions, one of which is that in the limit we visit each state-action tuple infinitely many times. If your random random p …
- 5,030
4
votes
Accepted
How should I decay $\epsilon$ in Q-learning?
The way you have described tends to be the common approach. There are of course other ways that you could do this e.g. using an exponential decay, or to only decay after a 'successful' episode, albeit …
- 5,030
4
votes
Accepted
How would I compute the optimal state-action value for a certain state and action?
It seems that you are getting confused between the definition of a Q-value and the update rule used to obtain these Q-values.
Remember that to simply obtain an optimal Q-value for a given state-action …
- 5,030
4
votes
Why does regular Q-learning (and DQN) overestimate the Q values?
The overestimation comes from the random initialisation of your Q-value estimates. Obviously these will not be perfect (if they were then we wouldn't need to learn the true Q-values!). In many value b …
- 5,030
4
votes
1
answer
1k
views
Why we don't use importance sampling in tabular Q-Learning?
Why don't we use an importance sampling ratio in Q-Learning, even though Q-Learning is an off-policy method?
Importance sampling is used to calculate expectation of a random variable by using data n …
- 5,030
4
votes
Accepted
Does the DoubleDQN algorithm use a target network or two separate policies?
The canonical DoubleDQN uses the target network. I've not seen the first version used anywhere in the deep RL literature, but it looks like what one would do if they were to take the original Double Q …
- 5,030
4
votes
Accepted
Reinforcement learning with action consisting of two discrete values
You would still be picking a single action. Your action space is now $\mathcal{A} = \mathcal{O} \times \mathcal{I}$ where I've chosen $\mathcal{O}$ to be the set of possible orders from your problem a …
- 5,030
3
votes
Accepted
When calculating the cost in deep Q-learning, do we use both the input and target states?
I will first explain briefly to you the difference between supervised learning and reinforcement learning to make sure that you don't have any misunderstandings. In supervised learning you are provide …
- 5,030
3
votes
Accepted
Why we don't use importance sampling in tabular Q-Learning?
In Tabular Q-learning the update is as follows
$$Q(s,a) = Q(s,a) + \alpha \left[R_{t+1} + \gamma \max_aQ(s',a) - Q(s,a) \right]\;.$$
Now, as we are interested in learning about the optimal policy, …
- 5,030
3
votes
Accepted
What is the target output for updating a Deep Q Network
As you say, the output of a $Q$ network is typically a value for all actions of the given state. Let us call this output $\mathbf{x} \in \mathbb{R}^{|\mathcal{A}|}$. To train your network using the sq …
- 5,030