Questions tagged [q-learning]

For questions related to the Q-learning algorithm, which is a model-free and temporal-difference reinforcement learning algorithm that attempts to approximate the Q function, which is a function that, given a state s and an action a, returns a real number that represents the return (or value) of state s when action a is taken from s. Q-learning was introduced in the PhD thesis "Learning from Delayed Rewards" (1989) by Watkins.

Filter by
Sorted by
Tagged with
0 votes
1 answer
24 views

My first experience with gym environment has raised many questions, and I need some guidance

As I'm new to the AI/ML field, I'm still learning from various online materials. In this particular instance, I've been studying the Reinforcement Learning tutorial by deeplizard, specifically ...
Boris L.'s user avatar
  • 111
1 vote
1 answer
44 views

Find maximum value of unknown functions f(x,y)=z using reinforcement learning & neural network

is it possible to train a neural network to find the global maximum value of unknown functions like f(x,y)=z with reinforcement learning? Up until now I had only had experience with simple ...
Bubble's user avatar
  • 13
3 votes
1 answer
88 views

Finding the true Q-values in gymnaiusm

I'm very interested in the true Q-values of state-action pairs in the classic control environments in gymnasium. Contrary to the usual goal, the ordering of the Q-values itself is irrelevant; a very ...
Mark B's user avatar
  • 33
0 votes
3 answers
37 views

In Q-Learning the Q-Table is not considered a model of the game?

In a QTable you keep states and actions for the ongoing decision making, it somehow represents the knowledge of the world and your future decisions for this and any future instance of a game. In the ...
Raul Lapeira Herrero's user avatar
2 votes
1 answer
97 views

In Q-learning, how are Q values updated for the last state in the Q table?

In Q-learning, I know that the Q-values are updated using the Bellman equation. $$ Q^{new}(S_t,A_t) \leftarrow Q(S_t,A_t) + \alpha [R_{t+1} + \gamma \underset{a}{max} Q(S_{t+1},a) - Q(S_t,A_t)] $$ ...
gondorian's user avatar
1 vote
1 answer
75 views

Q learning achieves small reward in simple dice game

I am trying to train a Q learning agent on the following game: The states are parametrised by an integer $S \geq 0$ (representing the sum of the previous die rolls). In each step the player can choose ...
deepfloe's user avatar
  • 111
2 votes
1 answer
40 views

Should the Q-Value of a state-action tuple be updated, if $s_{t} == s_{t+1}$

Assuming the agent in my environment does an action, however the agent's state does not change. Does that mean the Q-Table gets updated regardless of respective states (current and next) being the ...
Ralph's user avatar
  • 21
0 votes
1 answer
72 views

Q learning: How to create output layer in which actions are combinations of multiple sub-actions

Suppose in my example I want an agent to learn a behavior that is made up of a combination of actions. So consider the following example with a tamagotchi like game: There are 5 pets and 3 actions ...
T. Kau's user avatar
  • 103
0 votes
0 answers
48 views

How to use "states" as vectors in Q-Learning?

How do I use states as vectors in Q-Learning or any other RL Algorithms. Let's say I have state as a vector with probabilities[0,1], and I have to take an action if the state is valid (with ...
rainarashika's user avatar
0 votes
0 answers
50 views

Problem of extremely varied reward value in DDQN

I am trying to train my model by DDQN agent after creating a customized environment in gym. I am stating my hyper-parameters and other details here. ...
Subhajit Saha's user avatar
0 votes
0 answers
22 views

Dealing with long running tasks in Q Learning

Assuming Q Learning is applied not to low level behaviors (such as taking a step; drawing/playing a card; moving one piece on a board), but rather longer running high-level behaviors (e.g. Moving from ...
Ralph's user avatar
  • 21
2 votes
1 answer
49 views

Which Q function do we use to make our policy when using double Q learning?

I know this might be arbitrary, but I couldn't find any good information on this. As we update 2 q function in double q learning I was curios whether we average, or sum them together to get our policy....
IloveR's user avatar
  • 23
3 votes
1 answer
141 views

What is the motivation for using Q-Learning in RL?

In Spinning Up by OpenAI, it says the following regarding policy optimization methods and Q-Learning as ways of getting a good policy for RL. Trade-offs Between Policy Optimization and Q-Learning. ...
Justin T's user avatar
  • 145
2 votes
1 answer
151 views

Why does Advantage Learning help function approximators?

Many later RL algorithms like PPO or Duelling DQN estimate the advantage. I am not very sure of how that really helps. For instance, the actor loss for a simple actor critic algorithm is given by - <...
desert_ranger's user avatar
0 votes
0 answers
21 views

Problem on evaluating DQN, on a Vehicle Routing Problem (VRP)

I am running this DQN algorithm that is trying to minimize the total distance traveled by a vehicle (VRP). In the training, as you can see in the images, everything works fine: the loss is decreasing, ...
elizabeth's user avatar
0 votes
1 answer
88 views

Will my Q values keep going up forever?

In Q-learning,the q values can be updated by the bellman equation. What happens with my Q values is that they keep going up forever, in accordance with the more I train. After 10,000 training episodes,...
Kyotiq's user avatar
  • 1
0 votes
1 answer
51 views

How can I get Q-Learning (1 step off policy) update from n-step off policy learning update?

In Sutton and Barto we have expressions for Q-Learning and n-step Off policy learning. The former ought to be the 1-step limit of the latter but I cannot see it working out that way. What am I missing?...
Borun Chowdhury's user avatar
0 votes
0 answers
34 views

Relationship between TD control algorithm (SARSA) and logistic regression in two-armed bandit task

I have been looking for a way to model behavioral data (from rodents) in a nonstationary 2-armed bandit task. In this task the rodent can nose poke either on a left or a right port, and it will get a ...
Alex Legaria's user avatar
0 votes
1 answer
82 views

Negative action-state values found during deep Q-learning

I'm training a simple deep q-learning algorithm with no experience buffer to solve the CartPole-v5 environment. I want to check for overestimation, therefore I'm ...
Gello's user avatar
  • 103
1 vote
1 answer
185 views

Effects of hyperparameters in Q-learning

While playing around with the learning rate and discount factor in the Q-learning algorithm, I noticed some behavior that I could not really understand myself. Firstly, I noticed that increasing the ...
perceptronEnthusiast420's user avatar
1 vote
1 answer
147 views

Is it necessary to have a constant reward in the terminal state?

I have downloaded the grid world project form this link. I have executed the project multiple times using: python gridworld.py -k 20 -a q -r -0.2 -s 90 I have ...
AAA's user avatar
  • 111
0 votes
0 answers
33 views

What is the difference between fitted Q Iteration algorithms and traditional off-policy Q-learning algorithms?

I am unclear about what the difference between FQI-type algorithms and traditional Q-learning algorithms is. Is the only difference that FQI methods are not sequential, in the sense that they except a ...
JoshML's user avatar
  • 1
0 votes
1 answer
42 views

What does the figure in Q-learning vs Expected SARSA actually show?

I might be blind. But I wasn't able to find or figure out what the small difference between Q-learn and SARSA depicts in the following image; (src). What does the semi-circle show? and what does the ...
nammerkage's user avatar
0 votes
1 answer
121 views

Are there better loss functions than MSE for maze solver using deep learning?

I am a newbie in reinforcement learning, and I was doing a project on solving an agent maze solver using deep Q Learning. Currently, I am using the MSE loss function, but the agent is very slow or ...
Lim's user avatar
  • 1
1 vote
2 answers
148 views

How do you apply Q-learning when there are too many possible actions?

When the number of states in the Q-learning is large, we can refer to approximate Q-learning, but what should we do when we have a large number of actions?
znb's user avatar
  • 11
1 vote
1 answer
46 views

What is the meaning of $ (I - \gamma P^{\pi})^{-1} \left[\frac{\mu(a|s)}{\hat \pi_{\beta}(a|s)} \right](s, a)$?

In Theorem 3.1 of the conservative q-learning paper, what is the meaning of $$ (I - \gamma P^{\pi})^{-1} \left[\frac{\mu(a|s)}{\hat \pi_{\beta}(a|s)} \right](s, a)$$? I thought $(I - \gamma P^{\pi})^{-...
hongshan.li's user avatar
5 votes
0 answers
63 views

What exactly is non-delusional Q-learning?

Problems occur when we combine Q-learning with a function approximator. What exactly is the delusional-bias and non-delusional Q-learning? I am talking about the neurIPS 18 best paper Non-delusional Q-...
wrek's user avatar
  • 183
0 votes
1 answer
73 views

What would the "state space" and its Python implementation be for my simulation?

Context I'm trying to build a social-consensus simulation involving two intelligent agents. The simulation involves a graph/network of nodes. Nearly all of these nodes (> 90%) will be green agents. ...
The Pointer's user avatar
3 votes
1 answer
254 views

For which problem sizes is Deep Q-Learning suitable and why?

I am wondering for which problem sizes a Deep Q-Learning algorithm is most appropriate. For example, whether it is particularly suited for low complexity problems or not for high complexity problems. ...
user avatar
1 vote
2 answers
2k views

What is the difference between A2C and Q-Learning, and when to use one over the other?

I'm trying to get an accurate answer about the difference between A2C and Q-Learning. And when can we use each of them?
Hani's user avatar
  • 13
0 votes
1 answer
189 views

How to deal with delay in reinforcement learning, an unclear case

According to the question in How to deal with the time delay in reinforcement learning?, we can tell the delay in the reinforcement learning can be observation delay, action delay and reward delay. I ...
CharlesC's user avatar
1 vote
2 answers
432 views

Does maximizing the value function and maximizing the state-action value function generate the same optimal policy?

In reinforcement learning, we define the optimal policy $\pi^*$ as the policy that maximizes the value of the state: $$ \pi_v^*=\underset{\pi}{\operatorname{argmax}} {V_{\pi}(s)} $$ In Q-learning, we ...
Cloudy's user avatar
  • 203
0 votes
1 answer
357 views

How do I create an AI controller for Pacman?

How do I create an AI controller, which can play pacman - by taking in pixel values (or some other data by represents the state) which perhaps runs on a separate thread, which can control the game? It ...
R3sist's user avatar
  • 9
1 vote
1 answer
217 views

Is the described Q-table considered large?

I never saw any rule of thumb as to what size is said as large for a q-table but I have a Q-table with like 2500 entries. Is it considered large for a tabular approach? Anyone from experience can ...
knowledge_seeker's user avatar
0 votes
1 answer
224 views

How to compare memory requirements for tabular Q-learning vs deep neural network?

I want to compare the space complexity/memory requirement of tabular Q-learning v.s. deep neural Q-network (DQN). I think DQN would be faster and Q-table has a disadvantage at large table sizes but ...
knowledge_seeker's user avatar
0 votes
0 answers
65 views

What if we modify some Q-values while taking the action?

Just a passing thought about Q-learning. In the tabular Q-learning, what if I play around and modify any Q-values as I am using them to take actions? Would it be a violation of any (1) theoretical ...
knowledge_seeker's user avatar
0 votes
1 answer
164 views

Is it possible to add states to the Q-table after the game has started?

I would like to implement Q-learning in a game. Here is the board: It's a 2 player game. At each turn, each player can put a pawn on a line of their choice. They can't choose the column. The right ...
Dunno's user avatar
  • 103
0 votes
1 answer
348 views

How to deal with changing rewards in Q-learning? DQN?

I read the working of Q-learning through a grid-based taxi routing wherein a taxi has to pick and drop off a passenger from source to destination. Likewise, I have a routing problem and hence, I tried ...
knowledge_seeker's user avatar
2 votes
1 answer
705 views

Is Q-learning only capable of learning a deterministic policy?

I was following a reinforcement learning course on coursera and in this video at 2:57 the instructor says Expected SARSA and SARSA both allow us to learn an optimal $\epsilon$-soft policy, but, Q-...
ketan dhanuka's user avatar
0 votes
1 answer
26 views

Using reinforcement learning for human-robot interaction [closed]

I have a scenario where a user is wanting to exercise and improve over time. They attend around 10 exercise sessions, doing 20 repititions of an exercise each session. I want to develop a ...
caaax's user avatar
  • 101
0 votes
1 answer
171 views

How to manage impossible actions? [closed]

I am using Q-learning in julia language. Because of the solver’s configuration, actions have to be defined as the whole action space and impossible actions have to be also considered. It means that I ...
Aquila's user avatar
  • 33
0 votes
1 answer
235 views

Is the case of a big state space, should we use a softmax exploration policy rather than $\epsilon$-greedy for Q-Learning?

In Reinforcement Learning, epsilon-greedy policies are the most used exploration policies, but in case there is a big state space with impossible actions, wouldn't it be better to use soft-max ...
Aquila's user avatar
  • 33
0 votes
1 answer
115 views

Could Softmax Action Selection be useful to solve an episodic task with more than 100000 possible states and 2000 actions?

I am new in the field of RL. I am trying to use tabular methods, Q-Learning for solving a problem that takes a lot of time for computation, so I would like to know if there are more efficient methods ...
Aquila's user avatar
  • 33
2 votes
1 answer
216 views

Is it possible learning convergence is lost in Reinforcement Learning as the state space grows?

I am new in the AI field and I am trying to use Reinforcement Learning. Specifically, I am using tabular Q-Learning and SARSA algorithms to solve a sequential decision making problem. (I am using <...
Aquila's user avatar
  • 33
0 votes
1 answer
69 views

Does $S_{t+1}$ denote the future information in Q-learning?

In Q-learning, $Q(S_t,a)$ is updated by the Bellman equation. $Q(S_t,a) = r + \max_{a'}(Q(S_{t+1},a'))$ where $S_{t+1}$ is the future state. Let's say $S$ denotes the stock price, does it mean we are ...
L.Chau's user avatar
  • 1
0 votes
1 answer
345 views

When calculating the max in DQN, do I have to calculate the Q for every possible action for a particular state?

I'm trying to implement the DQN paper using python/pytorch for my needs (https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf). I'm studying the main algorithm: I am a bit confused about the $\gamma* \max ...
Antonis Karvelas's user avatar
1 vote
1 answer
128 views

Can directly using expert policy in epsilon-greedy speed-up Q-learning?

In deep Q-learning we typically use epsilon-greedy policy during training. We choose a random action for a certain probability $\epsilon$, and choose the action that maximize the current Q-value ...
Cloudy's user avatar
  • 203
1 vote
1 answer
365 views

Is using Monte-Carlo estimate of returns in Deep Q Learning possible?

In all the tutorials of deep Q-learning (using neural networks) I have read so far, the state-action value function $Q(s,a)$ is learned by temporal difference learning. However, in policy gradient ...
Cloudy's user avatar
  • 203
-1 votes
1 answer
632 views

what does the OpenAI ALE/Breakout-RAM-V5 observation return [closed]

I haven't been able to understand the output that OpenAI gym return for observation from this snippet ...
Jirawat Zhou's user avatar
2 votes
1 answer
212 views

Deep Q-Learning Model Effectiveness Improves then Crashes

I am implementing a Deep Q-Learning Algorithm. The model appears to improve but after awhile it just crashes and does just as well as if an agent was making random decisions. Shouldn't the behavior ...
Josh Majors's user avatar

1
2 3 4 5
8