Questions tagged [q-learning]
For questions related to the Q-learning algorithm, which is a model-free and temporal-difference reinforcement learning algorithm that attempts to approximate the Q function, which is a function that, given a state s and an action a, returns a real number that represents the return (or value) of state s when action a is taken from s. Q-learning was introduced in the PhD thesis "Learning from Delayed Rewards" (1989) by Watkins.
370
questions
0
votes
1
answer
24
views
My first experience with gym environment has raised many questions, and I need some guidance
As I'm new to the AI/ML field, I'm still learning from various online materials. In this particular instance, I've been studying the Reinforcement Learning tutorial by deeplizard, specifically ...
- 111
1
vote
1
answer
44
views
Find maximum value of unknown functions f(x,y)=z using reinforcement learning & neural network
is it possible to train a neural network to find the global maximum value of unknown functions like f(x,y)=z with reinforcement learning?
Up until now I had only had experience with simple ...
- 13
3
votes
1
answer
88
views
Finding the true Q-values in gymnaiusm
I'm very interested in the true Q-values of state-action pairs in the classic control environments in gymnasium. Contrary to the usual goal, the ordering of the Q-values itself is irrelevant; a very ...
- 33
0
votes
3
answers
37
views
In Q-Learning the Q-Table is not considered a model of the game?
In a QTable you keep states and actions for the ongoing decision making, it somehow represents the knowledge of the world and your future decisions for this and any future instance of a game. In the ...
2
votes
1
answer
97
views
In Q-learning, how are Q values updated for the last state in the Q table?
In Q-learning, I know that the Q-values are updated using the Bellman equation.
$$
Q^{new}(S_t,A_t) \leftarrow Q(S_t,A_t) + \alpha [R_{t+1} + \gamma \underset{a}{max} Q(S_{t+1},a) - Q(S_t,A_t)]
$$
...
- 35
1
vote
1
answer
75
views
Q learning achieves small reward in simple dice game
I am trying to train a Q learning agent on the following game: The states are parametrised by an integer $S \geq 0$ (representing the sum of the previous die rolls). In each step the player can choose ...
- 111
2
votes
1
answer
40
views
Should the Q-Value of a state-action tuple be updated, if $s_{t} == s_{t+1}$
Assuming the agent in my environment does an action, however the agent's state does not change. Does that mean the Q-Table gets updated regardless of respective states (current and next) being the ...
- 21
0
votes
1
answer
72
views
Q learning: How to create output layer in which actions are combinations of multiple sub-actions
Suppose in my example I want an agent to learn a behavior that is made up of a combination of actions.
So consider the following example with a tamagotchi like game: There are 5 pets and 3 actions ...
- 103
0
votes
0
answers
48
views
How to use "states" as vectors in Q-Learning?
How do I use states as vectors in Q-Learning or any other RL Algorithms. Let's say I have state as a vector with probabilities[0,1], and I have to take an action if the state is valid (with ...
0
votes
0
answers
50
views
Problem of extremely varied reward value in DDQN
I am trying to train my model by DDQN agent after creating a customized environment in gym. I am stating my hyper-parameters and other details here.
...
- 101
0
votes
0
answers
22
views
Dealing with long running tasks in Q Learning
Assuming Q Learning is applied not to low level behaviors (such as taking a step; drawing/playing a card; moving one piece on a board), but rather longer running high-level behaviors (e.g. Moving from ...
- 21
2
votes
1
answer
49
views
Which Q function do we use to make our policy when using double Q learning?
I know this might be arbitrary, but I couldn't find any good information on this. As we update 2 q function in double q learning I was curios whether we average, or sum them together to get our policy....
- 23
3
votes
1
answer
141
views
What is the motivation for using Q-Learning in RL?
In Spinning Up by OpenAI, it says the following regarding policy optimization methods and Q-Learning as ways of getting a good policy for RL.
Trade-offs Between Policy Optimization and Q-Learning. ...
- 145
2
votes
1
answer
151
views
Why does Advantage Learning help function approximators?
Many later RL algorithms like PPO or Duelling DQN estimate the advantage. I am not very sure of how that really helps.
For instance, the actor loss for a simple actor critic algorithm is given by -
<...
- 588
0
votes
0
answers
21
views
Problem on evaluating DQN, on a Vehicle Routing Problem (VRP)
I am running this DQN algorithm that is trying to minimize the total distance traveled by a vehicle (VRP). In the training, as you can see in the images, everything works fine: the loss is decreasing, ...
0
votes
1
answer
88
views
Will my Q values keep going up forever?
In Q-learning,the q values can be updated by the bellman equation. What happens with my Q values is that they keep going up forever, in accordance with the more I train. After 10,000 training episodes,...
- 1
0
votes
1
answer
51
views
How can I get Q-Learning (1 step off policy) update from n-step off policy learning update?
In Sutton and Barto we have expressions for Q-Learning and n-step Off policy learning. The former ought to be the 1-step limit of the latter but I cannot see it working out that way. What am I missing?...
- 191
0
votes
0
answers
34
views
Relationship between TD control algorithm (SARSA) and logistic regression in two-armed bandit task
I have been looking for a way to model behavioral data (from rodents) in a nonstationary 2-armed bandit task. In this task the rodent can nose poke either on a left or a right port, and it will get a ...
0
votes
1
answer
82
views
Negative action-state values found during deep Q-learning
I'm training a simple deep q-learning algorithm with no experience buffer to solve the CartPole-v5 environment.
I want to check for overestimation, therefore I'm ...
- 103
1
vote
1
answer
185
views
Effects of hyperparameters in Q-learning
While playing around with the learning rate and discount factor in the Q-learning algorithm, I noticed some behavior that I could not really understand myself.
Firstly, I noticed that increasing the ...
1
vote
1
answer
147
views
Is it necessary to have a constant reward in the terminal state?
I have downloaded the grid world project form this link. I have executed the project multiple times using:
python gridworld.py -k 20 -a q -r -0.2 -s 90
I have ...
- 111
0
votes
0
answers
33
views
What is the difference between fitted Q Iteration algorithms and traditional off-policy Q-learning algorithms?
I am unclear about what the difference between FQI-type algorithms and traditional Q-learning algorithms is. Is the only difference that FQI methods are not sequential, in the sense that they except a ...
- 1
0
votes
1
answer
42
views
What does the figure in Q-learning vs Expected SARSA actually show?
I might be blind.
But I wasn't able to find or figure out what the small difference between Q-learn and SARSA depicts in the following image;
(src).
What does the semi-circle show? and what does the ...
- 206
0
votes
1
answer
121
views
Are there better loss functions than MSE for maze solver using deep learning?
I am a newbie in reinforcement learning, and I was doing a project on solving an agent maze solver using deep Q Learning. Currently, I am using the MSE loss function, but the agent is very slow or ...
- 1
1
vote
2
answers
148
views
How do you apply Q-learning when there are too many possible actions?
When the number of states in the Q-learning is large, we can refer to approximate Q-learning, but what should we do when we have a large number of actions?
- 11
1
vote
1
answer
46
views
What is the meaning of $ (I - \gamma P^{\pi})^{-1} \left[\frac{\mu(a|s)}{\hat \pi_{\beta}(a|s)} \right](s, a)$?
In Theorem 3.1 of the conservative q-learning paper, what is the meaning of
$$ (I - \gamma P^{\pi})^{-1} \left[\frac{\mu(a|s)}{\hat \pi_{\beta}(a|s)} \right](s, a)$$?
I thought $(I - \gamma P^{\pi})^{-...
- 108
5
votes
0
answers
63
views
What exactly is non-delusional Q-learning?
Problems occur when we combine Q-learning with a function approximator.
What exactly is the delusional-bias and non-delusional Q-learning? I am talking about the neurIPS 18 best paper Non-delusional Q-...
- 183
0
votes
1
answer
73
views
What would the "state space" and its Python implementation be for my simulation?
Context
I'm trying to build a social-consensus simulation involving two intelligent agents. The simulation involves a graph/network of nodes. Nearly all of these nodes (> 90%) will be green agents. ...
- 527
3
votes
1
answer
254
views
For which problem sizes is Deep Q-Learning suitable and why?
I am wondering for which problem sizes a Deep Q-Learning algorithm is most appropriate. For example, whether it is particularly suited for low complexity problems or not for high complexity problems. ...
anon
1
vote
2
answers
2k
views
What is the difference between A2C and Q-Learning, and when to use one over the other?
I'm trying to get an accurate answer about the difference between A2C and Q-Learning. And when can we use each of them?
- 13
0
votes
1
answer
189
views
How to deal with delay in reinforcement learning, an unclear case
According to the question in How to deal with the time delay in reinforcement learning?, we can tell the delay in the reinforcement learning can be observation delay, action delay and reward delay.
I ...
- 1
1
vote
2
answers
432
views
Does maximizing the value function and maximizing the state-action value function generate the same optimal policy?
In reinforcement learning, we define the optimal policy $\pi^*$ as the policy that maximizes the value of the state:
$$
\pi_v^*=\underset{\pi}{\operatorname{argmax}} {V_{\pi}(s)}
$$
In Q-learning, we ...
- 203
0
votes
1
answer
357
views
How do I create an AI controller for Pacman?
How do I create an AI controller, which can play pacman - by taking in pixel values (or some other data by represents the state) which perhaps runs on a separate thread, which can control the game?
It ...
- 9
1
vote
1
answer
217
views
Is the described Q-table considered large?
I never saw any rule of thumb as to what size is said as large for a q-table but I have a Q-table with like 2500 entries. Is it considered large for a tabular approach? Anyone from experience can ...
0
votes
1
answer
224
views
How to compare memory requirements for tabular Q-learning vs deep neural network?
I want to compare the space complexity/memory requirement of tabular Q-learning v.s. deep neural Q-network (DQN). I think DQN would be faster and Q-table has a disadvantage at large table sizes but ...
0
votes
0
answers
65
views
What if we modify some Q-values while taking the action?
Just a passing thought about Q-learning. In the tabular Q-learning, what if I play around and modify any Q-values as I am using them to take actions? Would it be a violation of any (1) theoretical ...
0
votes
1
answer
164
views
Is it possible to add states to the Q-table after the game has started?
I would like to implement Q-learning in a game.
Here is the board:
It's a 2 player game. At each turn, each player can put a pawn on a line of their choice. They can't choose the column. The right ...
- 103
0
votes
1
answer
348
views
How to deal with changing rewards in Q-learning? DQN?
I read the working of Q-learning through a grid-based taxi routing wherein a taxi has to pick and drop off a passenger from source to destination. Likewise, I have a routing problem and hence, I tried ...
2
votes
1
answer
705
views
Is Q-learning only capable of learning a deterministic policy?
I was following a reinforcement learning course on coursera and in this video at 2:57 the instructor says
Expected SARSA and SARSA both allow us to learn an optimal
$\epsilon$-soft policy, but, Q-...
0
votes
1
answer
26
views
Using reinforcement learning for human-robot interaction [closed]
I have a scenario where a user is wanting to exercise and improve over time. They attend around 10 exercise sessions, doing 20 repititions of an exercise each session.
I want to develop a ...
- 101
0
votes
1
answer
171
views
How to manage impossible actions? [closed]
I am using Q-learning in julia language.
Because of the solver’s configuration, actions have to be defined as the whole action space and impossible actions have to be also considered. It means that I ...
- 33
0
votes
1
answer
235
views
Is the case of a big state space, should we use a softmax exploration policy rather than $\epsilon$-greedy for Q-Learning?
In Reinforcement Learning, epsilon-greedy policies are the most used exploration policies, but in case there is a big state space with impossible actions, wouldn't it be better to use soft-max ...
- 33
0
votes
1
answer
115
views
Could Softmax Action Selection be useful to solve an episodic task with more than 100000 possible states and 2000 actions?
I am new in the field of RL. I am trying to use tabular methods, Q-Learning for solving a problem that takes a lot of time for computation, so I would like to know if there are more efficient methods ...
- 33
2
votes
1
answer
216
views
Is it possible learning convergence is lost in Reinforcement Learning as the state space grows?
I am new in the AI field and I am trying to use Reinforcement Learning. Specifically, I am using tabular Q-Learning and SARSA algorithms to solve a sequential decision making problem. (I am using <...
- 33
0
votes
1
answer
69
views
Does $S_{t+1}$ denote the future information in Q-learning?
In Q-learning, $Q(S_t,a)$ is updated by the Bellman equation. $Q(S_t,a) = r + \max_{a'}(Q(S_{t+1},a'))$ where $S_{t+1}$ is the future state.
Let's say $S$ denotes the stock price, does it mean we are ...
- 1
0
votes
1
answer
345
views
When calculating the max in DQN, do I have to calculate the Q for every possible action for a particular state?
I'm trying to implement the DQN paper using python/pytorch for my needs (https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf). I'm studying the main algorithm:
I am a bit confused about the $\gamma* \max ...
1
vote
1
answer
128
views
Can directly using expert policy in epsilon-greedy speed-up Q-learning?
In deep Q-learning we typically use epsilon-greedy policy during training. We choose a random action for a certain probability $\epsilon$, and choose the action that maximize the current Q-value ...
- 203
1
vote
1
answer
365
views
Is using Monte-Carlo estimate of returns in Deep Q Learning possible?
In all the tutorials of deep Q-learning (using neural networks) I have read so far, the state-action value function $Q(s,a)$ is learned by temporal difference learning. However, in policy gradient ...
- 203
-1
votes
1
answer
632
views
what does the OpenAI ALE/Breakout-RAM-V5 observation return [closed]
I haven't been able to understand the output that OpenAI gym return for observation from this snippet
...
2
votes
1
answer
212
views
Deep Q-Learning Model Effectiveness Improves then Crashes
I am implementing a Deep Q-Learning Algorithm. The model appears to improve but after awhile it just crashes and does just as well as if an agent was making random decisions. Shouldn't the behavior ...
- 23