Search Results
| Search type | Search syntax |
|---|---|
| Tags | [tag] |
| Exact | "words here" |
| Author |
user:1234 user:me (yours) |
| Score |
score:3 (3+) score:0 (none) |
| Answers |
answers:3 (3+) answers:0 (none) isaccepted:yes hasaccepted:no inquestion:1234 |
| Views | views:250 |
| Code | code:"if (foo != bar)" |
| Sections |
title:apples body:"apples oranges" |
| URL | url:"*.example.com" |
| Saves | in:saves |
| Status |
closed:yes duplicate:no migrated:no wiki:no |
| Types |
is:question is:answer |
| Exclude |
-[tag] -apples |
| For more details on advanced search visit our help page | |
Results tagged with q-learning
Search options not deleted
user 36821
For questions related to the Q-learning algorithm, which is a model-free and temporal-difference reinforcement learning algorithm that attempts to approximate the Q function, which is a function that, given a state s and an action a, returns a real number that represents the return (or value) of state s when action a is taken from s. Q-learning was introduced in the PhD thesis "Learning from Delayed Rewards" (1989) by Watkins.
2
votes
Accepted
How is it possible that Q-learning can learn a state-action value without taking into accoun...
Q-learning can learn about the greedy policy (the policy that we define as $\pi(s) = \arg\max_a Q(s, a)$) whilst following some arbitrary exploratory policy because Q-learning is an off-policy algorit …
- 5,030
4
votes
Accepted
How should I decay $\epsilon$ in Q-learning?
The way you have described tends to be the common approach. There are of course other ways that you could do this e.g. using an exponential decay, or to only decay after a 'successful' episode, albeit …
- 5,030
3
votes
Accepted
Q learning: How to create output layer in which actions are combinations of multiple sub-act...
What you're describing is known as a composite action space (or a factorisable action space). The key assumption is that the action space can be factorised in sub-action spaces, i.e. $\mathcal{A} = \m …
- 5,030
1
vote
How does one know that a problem is "model-free" in reinforcement learning?
A reinforcement learning algorithm is considered model based if it uses estimates of the environments dynamics to help learn. For instance, in the Tabular Dyna-Q algorithm, every time you visit a stat …
- 5,030
2
votes
Why is it not advisable to have a 100 percent exploration rate?
No - imagine if you were playing an Atari game and took completely random actions. Your games would not last very long and you would never get to experience all of the state space because the game wou …
- 5,030
1
vote
Why do we calculate the mean squared error loss to improve the value approximation in Advant...
I believe that the author is referring to how the networks are trained in Deep RL. Consider Deep Q-Learning where the $Q(s,a)$ is approximated using a neural network. Then the loss function used to tr …
- 5,030
1
vote
Accepted
Can tabular Q-learning converge even if it doesn't explore all state-action pairs?
In the tabular case, then the Q table will only converge if you have walked around the whole of the table. Note that to guarantee convergence we need $\sum\limits_{n=1}^{\infty}\alpha_n(a) = \infty$ a …
- 5,030
3
votes
Accepted
Why we don't use importance sampling in tabular Q-Learning?
In Tabular Q-learning the update is as follows
$$Q(s,a) = Q(s,a) + \alpha \left[R_{t+1} + \gamma \max_aQ(s',a) - Q(s,a) \right]\;.$$
Now, as we are interested in learning about the optimal policy, …
- 5,030
4
votes
Accepted
How would I compute the optimal state-action value for a certain state and action?
It seems that you are getting confused between the definition of a Q-value and the update rule used to obtain these Q-values.
Remember that to simply obtain an optimal Q-value for a given state-action …
- 5,030
5
votes
Accepted
When do SARSA and Q-Learning converge to optimal Q values?
The true answers are 1 and 3.
1 is true because the required conditions for tabular Q-learning to converge is that each state action pair will be visited infinitely often, and Q-learning learns direct …
- 5,030
2
votes
Accepted
Why is sampling non-uniformly from the replay memory an issue? (Prioritized experience replay)
The problem is not that we need importance sampling because the learning is off-policy -- you are correct in that for one step off-policy algorithms such as $Q$-learning we don't need importance sampl …
- 5,030
2
votes
Accepted
What are the differences between SARSA and Q-learning?
The main difference between the two is that Q-learning is an off policy algorithm. That is, we learn about an policy that is different to the one we choose to make actions. To see this, lets look at t …
- 5,030
4
votes
1
answer
1k
views
Why we don't use importance sampling in tabular Q-Learning?
Why don't we use an importance sampling ratio in Q-Learning, even though Q-Learning is an off-policy method?
Importance sampling is used to calculate expectation of a random variable by using data n …
- 5,030
3
votes
Accepted
What is the difference between A2C and Q-Learning, and when to use one over the other?
The major difference between A2C and Q-Learning are what the algorithms learn. In A2C, and policy gradient algorithms in general, the policy is directly parameterised, i.e. we have $\pi_\theta (a|s)$. …
- 5,030
2
votes
Is there an upper limit to the maximum cumulative reward in a deep reinforcement learning pr...
In any reinforcement learning problem, not just Deep RL, then there is an upper bound for the cumulative reward, provided that the problem is episodic and not continuing.
If the problem is episodic an …
- 5,030