Questions tagged [q-learning]
For questions related to the Q-learning algorithm, which is a model-free and temporal-difference reinforcement learning algorithm that attempts to approximate the Q function, which is a function that, given a state s and an action a, returns a real number that represents the return (or value) of state s when action a is taken from s. Q-learning was introduced in the PhD thesis "Learning from Delayed Rewards" (1989) by Watkins.
378
questions
1
vote
0
answers
12
views
Why slow-changing policy invalidates Double DQN approach in TD3 paper?
In the paper describing TD3 (https://arxiv.org/abs/1802.09477), the authors say that they could not effectively address the Q-learning overestimation bias by using different networks for maximizing ...
- 131
2
votes
1
answer
94
views
What are the similarities between Q-learning and Value Iteration?
This is the explanation of value iteration in our notes where you keep applying bellman optimality equation till it stops changing and then acting greedily wrt the value function gives the optimal ...
- 23
1
vote
1
answer
61
views
Can Q-learning be used to create new creative solutions by combining different factors and characteristics?
References from Wikipedia:
https://en.wikipedia.org/wiki/Q-learning
https://en.wikipedia.org/wiki/Markov_decision_process
Q-learning can be used to create new creative solutions, combining different ...
- 233
1
vote
1
answer
39
views
In Q-learning, states need to be just X and Y positions of the agent, or a state can be several other characteristics?
For example, in this article: https://www.learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/, which explains Q-learnig, teaches the Smartcab problem, the environment is a ...
- 233
2
votes
1
answer
49
views
Is Q-learning limited to just visual scenarios, or is it much broader and can it be used to solve non-visual problems as well?
For example, in this article: https://www.learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/, which explains Q-learnig, teaches the Smartcab problem, it has a visual ...
- 233
1
vote
2
answers
298
views
In Q-learning, Am I the one who will define the way in which actions allow the agent to interact with the environment? And the interactions will vary?
In Q-learning, am I the one who will define the way in which actions allow the agent to interact with the environment, so that the way in which actions allow the agent to interact with the environment ...
- 233
1
vote
1
answer
60
views
Could someone give a very simple example of Q-learning in a very small environment? [closed]
I would really like to see an example of Q-learning that I could read, so that I can learn Q-learning from scratch. I read some articles on the internet, but I found it a little difficult to ...
- 233
-1
votes
1
answer
81
views
in simple words, what is the Q-learning algortimn steps? [closed]
I read this article about Q-learning:
https://www.learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/
It teaches how to implement the algorithm using the Gym Python library.
...
- 233
0
votes
0
answers
26
views
Multi-Agent DQN not learning for Clean Up Game - Reward slowly decreasing
The environment of the Clean Up game is simple: in a 25*18 grid world, there's dirt spawning on the left side and apples spawning on the other. Agents get a +1 reward for eating an apple (by stepping ...
- 1
0
votes
1
answer
91
views
My first experience with gym environment has raised many questions, and I need some guidance [closed]
As I'm new to the AI/ML field, I'm still learning from various online materials. In this particular instance, I've been studying the Reinforcement Learning tutorial by deeplizard, specifically ...
- 109
2
votes
1
answer
57
views
Find maximum value of unknown functions f(x,y)=z using reinforcement learning & neural network
is it possible to train a neural network to find the global maximum value of unknown functions like f(x,y)=z with reinforcement learning?
Up until now I had only had experience with simple ...
- 23
4
votes
1
answer
94
views
Finding the true Q-values in gymnaiusm
I'm very interested in the true Q-values of state-action pairs in the classic control environments in gymnasium. Contrary to the usual goal, the ordering of the Q-values itself is irrelevant; a very ...
- 43
0
votes
3
answers
51
views
In Q-Learning the Q-Table is not considered a model of the game?
In a QTable you keep states and actions for the ongoing decision making, it somehow represents the knowledge of the world and your future decisions for this and any future instance of a game. In the ...
3
votes
1
answer
107
views
In Q-learning, how are Q values updated for the last state in the Q table?
In Q-learning, I know that the Q-values are updated using the Bellman equation.
$$
Q^{new}(S_t,A_t) \leftarrow Q(S_t,A_t) + \alpha [R_{t+1} + \gamma \underset{a}{max} Q(S_{t+1},a) - Q(S_t,A_t)]
$$
...
- 45
1
vote
1
answer
78
views
Q learning achieves small reward in simple dice game
I am trying to train a Q learning agent on the following game: The states are parametrised by an integer $S \geq 0$ (representing the sum of the previous die rolls). In each step the player can choose ...
- 111
2
votes
1
answer
46
views
Should the Q-Value of a state-action tuple be updated, if $s_{t} == s_{t+1}$
Assuming the agent in my environment does an action, however the agent's state does not change. Does that mean the Q-Table gets updated regardless of respective states (current and next) being the ...
- 21
0
votes
1
answer
85
views
Q learning: How to create output layer in which actions are combinations of multiple sub-actions
Suppose in my example I want an agent to learn a behavior that is made up of a combination of actions.
So consider the following example with a tamagotchi like game: There are 5 pets and 3 actions ...
- 103
0
votes
0
answers
64
views
How to use "states" as vectors in Q-Learning?
How do I use states as vectors in Q-Learning or any other RL Algorithms. Let's say I have state as a vector with probabilities[0,1], and I have to take an action if the state is valid (with ...
0
votes
0
answers
54
views
Problem of extremely varied reward value in DDQN
I am trying to train my model by DDQN agent after creating a customized environment in gym. I am stating my hyper-parameters and other details here.
...
- 101
0
votes
0
answers
24
views
Dealing with long running tasks in Q Learning
Assuming Q Learning is applied not to low level behaviors (such as taking a step; drawing/playing a card; moving one piece on a board), but rather longer running high-level behaviors (e.g. Moving from ...
- 21
2
votes
1
answer
57
views
Which Q function do we use to make our policy when using double Q learning?
I know this might be arbitrary, but I couldn't find any good information on this. As we update 2 q function in double q learning I was curios whether we average, or sum them together to get our policy....
- 23
3
votes
1
answer
153
views
What is the motivation for using Q-Learning in RL?
In Spinning Up by OpenAI, it says the following regarding policy optimization methods and Q-Learning as ways of getting a good policy for RL.
Trade-offs Between Policy Optimization and Q-Learning. ...
- 145
2
votes
1
answer
163
views
Why does Advantage Learning help function approximators?
Many later RL algorithms like PPO or Duelling DQN estimate the advantage. I am not very sure of how that really helps.
For instance, the actor loss for a simple actor critic algorithm is given by -
<...
- 608
0
votes
0
answers
23
views
Problem on evaluating DQN, on a Vehicle Routing Problem (VRP)
I am running this DQN algorithm that is trying to minimize the total distance traveled by a vehicle (VRP). In the training, as you can see in the images, everything works fine: the loss is decreasing, ...
0
votes
1
answer
112
views
Will my Q values keep going up forever?
In Q-learning,the q values can be updated by the bellman equation. What happens with my Q values is that they keep going up forever, in accordance with the more I train. After 10,000 training episodes,...
- 1
0
votes
1
answer
68
views
How can I get Q-Learning (1 step off policy) update from n-step off policy learning update?
In Sutton and Barto we have expressions for Q-Learning and n-step Off policy learning. The former ought to be the 1-step limit of the latter but I cannot see it working out that way. What am I missing?...
- 191
0
votes
0
answers
37
views
Relationship between TD control algorithm (SARSA) and logistic regression in two-armed bandit task
I have been looking for a way to model behavioral data (from rodents) in a nonstationary 2-armed bandit task. In this task the rodent can nose poke either on a left or a right port, and it will get a ...
0
votes
1
answer
116
views
Negative action-state values found during deep Q-learning
I'm training a simple deep q-learning algorithm with no experience buffer to solve the CartPole-v5 environment.
I want to check for overestimation, therefore I'm ...
- 103
1
vote
1
answer
214
views
Effects of hyperparameters in Q-learning
While playing around with the learning rate and discount factor in the Q-learning algorithm, I noticed some behavior that I could not really understand myself.
Firstly, I noticed that increasing the ...
1
vote
1
answer
181
views
Is it necessary to have a constant reward in the terminal state?
I have downloaded the grid world project form this link. I have executed the project multiple times using:
python gridworld.py -k 20 -a q -r -0.2 -s 90
I have ...
- 111
0
votes
1
answer
48
views
What does the figure in Q-learning vs Expected SARSA actually show?
I might be blind.
But I wasn't able to find or figure out what the small difference between Q-learn and SARSA depicts in the following image;
(src).
What does the semi-circle show? and what does the ...
- 206
0
votes
1
answer
146
views
Are there better loss functions than MSE for maze solver using deep learning?
I am a newbie in reinforcement learning, and I was doing a project on solving an agent maze solver using deep Q Learning. Currently, I am using the MSE loss function, but the agent is very slow or ...
- 1
1
vote
2
answers
167
views
How do you apply Q-learning when there are too many possible actions?
When the number of states in the Q-learning is large, we can refer to approximate Q-learning, but what should we do when we have a large number of actions?
- 11
1
vote
1
answer
54
views
What is the meaning of $ (I - \gamma P^{\pi})^{-1} \left[\frac{\mu(a|s)}{\hat \pi_{\beta}(a|s)} \right](s, a)$?
In Theorem 3.1 of the conservative q-learning paper, what is the meaning of
$$ (I - \gamma P^{\pi})^{-1} \left[\frac{\mu(a|s)}{\hat \pi_{\beta}(a|s)} \right](s, a)$$?
I thought $(I - \gamma P^{\pi})^{-...
- 118
5
votes
0
answers
69
views
What exactly is non-delusional Q-learning?
Problems occur when we combine Q-learning with a function approximator.
What exactly is the delusional-bias and non-delusional Q-learning? I am talking about the neurIPS 18 best paper Non-delusional Q-...
- 183
0
votes
1
answer
76
views
What would the "state space" and its Python implementation be for my simulation?
Context
I'm trying to build a social-consensus simulation involving two intelligent agents. The simulation involves a graph/network of nodes. Nearly all of these nodes (> 90%) will be green agents. ...
- 547
3
votes
1
answer
289
views
For which problem sizes is Deep Q-Learning suitable and why?
I am wondering for which problem sizes a Deep Q-Learning algorithm is most appropriate. For example, whether it is particularly suited for low complexity problems or not for high complexity problems. ...
anon
2
votes
2
answers
2k
views
What is the difference between A2C and Q-Learning, and when to use one over the other?
I'm trying to get an accurate answer about the difference between A2C and Q-Learning. And when can we use each of them?
- 23
0
votes
1
answer
222
views
How to deal with delay in reinforcement learning, an unclear case
According to the question in How to deal with the time delay in reinforcement learning?, we can tell the delay in the reinforcement learning can be observation delay, action delay and reward delay.
I ...
- 1
1
vote
2
answers
503
views
Does maximizing the value function and maximizing the state-action value function generate the same optimal policy?
In reinforcement learning, we define the optimal policy $\pi^*$ as the policy that maximizes the value of the state:
$$
\pi_v^*=\underset{\pi}{\operatorname{argmax}} {V_{\pi}(s)}
$$
In Q-learning, we ...
- 213
0
votes
1
answer
440
views
How do I create an AI controller for Pacman?
How do I create an AI controller, which can play pacman - by taking in pixel values (or some other data by represents the state) which perhaps runs on a separate thread, which can control the game?
It ...
- 9
1
vote
1
answer
248
views
Is the described Q-table considered large?
I never saw any rule of thumb as to what size is said as large for a q-table but I have a Q-table with like 2500 entries. Is it considered large for a tabular approach? Anyone from experience can ...
0
votes
1
answer
241
views
How to compare memory requirements for tabular Q-learning vs deep neural network?
I want to compare the space complexity/memory requirement of tabular Q-learning v.s. deep neural Q-network (DQN). I think DQN would be faster and Q-table has a disadvantage at large table sizes but ...
0
votes
0
answers
71
views
What if we modify some Q-values while taking the action?
Just a passing thought about Q-learning. In the tabular Q-learning, what if I play around and modify any Q-values as I am using them to take actions? Would it be a violation of any (1) theoretical ...
0
votes
1
answer
188
views
Is it possible to add states to the Q-table after the game has started?
I would like to implement Q-learning in a game.
Here is the board:
It's a 2 player game. At each turn, each player can put a pawn on a line of their choice. They can't choose the column. The right ...
- 103
0
votes
1
answer
389
views
How to deal with changing rewards in Q-learning? DQN?
I read the working of Q-learning through a grid-based taxi routing wherein a taxi has to pick and drop off a passenger from source to destination. Likewise, I have a routing problem and hence, I tried ...
2
votes
1
answer
733
views
Is Q-learning only capable of learning a deterministic policy?
I was following a reinforcement learning course on coursera and in this video at 2:57 the instructor says
Expected SARSA and SARSA both allow us to learn an optimal
$\epsilon$-soft policy, but, Q-...
0
votes
1
answer
26
views
Using reinforcement learning for human-robot interaction [closed]
I have a scenario where a user is wanting to exercise and improve over time. They attend around 10 exercise sessions, doing 20 repititions of an exercise each session.
I want to develop a ...
- 101
0
votes
1
answer
184
views
How to manage impossible actions? [closed]
I am using Q-learning in julia language.
Because of the solver’s configuration, actions have to be defined as the whole action space and impossible actions have to be also considered. It means that I ...
- 33
0
votes
1
answer
252
views
Is the case of a big state space, should we use a softmax exploration policy rather than $\epsilon$-greedy for Q-Learning?
In Reinforcement Learning, epsilon-greedy policies are the most used exploration policies, but in case there is a big state space with impossible actions, wouldn't it be better to use soft-max ...
- 33