Questions tagged [q-learning]

For questions related to the Q-learning algorithm, which is a model-free and temporal-difference reinforcement learning algorithm that attempts to approximate the Q function, which is a function that, given a state s and an action a, returns a real number that represents the return (or value) of state s when action a is taken from s. Q-learning was introduced in the PhD thesis "Learning from Delayed Rewards" (1989) by Watkins.

Filter by
Sorted by
Tagged with
0 votes
2 answers
38 views

Can DQN lead with discrete state spaces?

For example in Cart Pole v1 gym environment the state space is continuous, but we discretize it to apply the Q-Learning algorithm because Q-Learning is a tabular method and only works with discrete ...
Vitor Martins's user avatar
0 votes
1 answer
79 views

Can Q(s,a) be replaced by V(s) when certrain requirements are met?

I read this post, was thinking about it and now I have a hypothesis but I am not sure whether or not its correct. I claim that in Q-learning $Q(s,a)$ can be replaced by V(s) when $p(s'|a,s)$ is ...
NMO's user avatar
  • 161
0 votes
0 answers
62 views

Bias of multi-step Q Learning on/off policy

This is comes from cs2852023Fall, hw3. I'm learning RL by myself and I cann't find answers related to this question. Althrough it's from a homework, I believe it would be beneficial to solve the ...
yeebo xie's user avatar
3 votes
1 answer
169 views

Does the DoubleDQN algorithm use a target network or two separate policies?

I've been looking for ways to improve my DQN. That is when I found the Double DQN algorithm. After looking at explanatory videos and posts, I've seen conflicting information: The Double DQN algorithm ...
Vladislav Korecký's user avatar
2 votes
1 answer
144 views

Q learning (DQN) strategy for a multiplayer zero-sum game

I have been looking for ways to train a Q-learning agent for a multiplayer zero-sum game (a variation of Tic-Tac-Toe in my case). I came up with a learning strategy I haven't found anywhere else, and ...
Vladislav Korecký's user avatar
1 vote
1 answer
40 views

Expectile regression in Implicit Q-Learning

I am reading Kostrikov et al.'s "Offline Reinforcement Learning with Implicit Q-Learning" but got stuck understanding one particular transformation they use. They describe the loss function ...
user118967's user avatar
0 votes
0 answers
21 views

Summing up rewards when encountering terminal state in n-step DQN

I'm trying to implement n-step DQN using deque for n-step experience buffer and I am not sure how to handle the terminal state in calculation For step = n = 5, Sx = state number x, T - terminal state, ...
Question's user avatar
0 votes
0 answers
30 views

Tabular Q-Learning & TicTacToe

I'm currently implementing tabular q-learning for 3x3 tictactoe in python and I'm new to RL and still have a hard time to understand RL. Therefore, I would like to know one thing: In (tabular) q-...
Hans123's user avatar
  • 11
1 vote
1 answer
55 views

Tabular Q-Learning: Is a variable for "action_history" needed for backpropagating the q-value for all previous actions?

I am implementing a Tabular Q-Learning algorithm in Python and have questions regarding the use of an 'action_history' variable. Necessity of 'action_history': In Q-Learning, the Q-value update is ...
Hans123's user avatar
  • 11
0 votes
0 answers
49 views

How do I deal with dynamic, parameterized, action spaces?

I want to design an AI Learning Algorithm for a Student made, round based Game. Let me first explain the Game/Environment You have a round based HTTP Game, in which multiple Players can participate. ...
Andre's user avatar
  • 1
0 votes
1 answer
109 views

Can Q-learning rewards and next states be non-deterministic?

I am working in a team to develop a Q-learning based approach for hyperparameter tuning. I have a disagreement with one of my teammates on how they defined this problem. They defined it as follows: ...
Ahmed Mokhtar's user avatar
1 vote
0 answers
63 views

Resulting quantiles from Quantile Regression DQN

In my QR-DQN application, the resulting quantiles for a state s and action a take the form of the blue line in the figure. The ...
Amav's user avatar
  • 11
5 votes
2 answers
281 views

DQN arXiv 10-year anniversary: What are the outstanding problems being actively researched in deep Q-learning since 2019?

Background As of today (12-19-2023), the arXiv submission of the original deep Q-learning approach to achieve superhuman performance on ATARI games has turned a decade old. The original approach, ...
DeepQZero's user avatar
  • 1,379
0 votes
1 answer
76 views

How do I update Q-values in Q-learning when rewards may only be received after many actions?

I am working on a Q-learning system where the agent may well (and almost always) have to take many actions before a reward can be given to the agent (or more so, the notion of a reward in my context ...
Darcy Sutton's user avatar
5 votes
2 answers
590 views

Does increasing the number of Q functions in Q-Learning scale?

Q-Learning (Watkins, 1989) uses a single function to estimate the value of actions and to choose the next action. Double Q-Learning (Hasselt, 2010) extends this and uses two functions which are ...
foreverska's user avatar
  • 1,018
2 votes
2 answers
130 views

Can we solve the environment with only the linear and angular position through Q-Learning?

I'm trying to solve the cartpole-v1 gym environment with only the linear and angular position, but the mean reward of the last 100 episodes isn't greater than 20 rewards. The longest train i made was ...
Vitor Martins's user avatar
1 vote
0 answers
30 views

Why slow-changing policy invalidates Double DQN approach in TD3 paper?

In the paper describing TD3 (https://arxiv.org/abs/1802.09477), the authors say that they could not effectively address the Q-learning overestimation bias by using different networks for maximizing ...
Jerry Ding's user avatar
2 votes
1 answer
173 views

What are the similarities between Q-learning and Value Iteration?

This is the explanation of value iteration in our notes where you keep applying bellman optimality equation till it stops changing and then acting greedily wrt the value function gives the optimal ...
ace239's user avatar
  • 23
1 vote
1 answer
64 views

Can Q-learning be used to create new creative solutions by combining different factors and characteristics?

References from Wikipedia: https://en.wikipedia.org/wiki/Q-learning https://en.wikipedia.org/wiki/Markov_decision_process Q-learning can be used to create new creative solutions, combining different ...
will The J's user avatar
1 vote
1 answer
49 views

In Q-learning, states need to be just X and Y positions of the agent, or a state can be several other characteristics?

For example, in this article: https://www.learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/, which explains Q-learnig, teaches the Smartcab problem, the environment is a ...
will The J's user avatar
2 votes
1 answer
57 views

Is Q-learning limited to just visual scenarios, or is it much broader and can it be used to solve non-visual problems as well?

For example, in this article: https://www.learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/, which explains Q-learnig, teaches the Smartcab problem, it has a visual ...
will The J's user avatar
1 vote
2 answers
307 views

In Q-learning, Am I the one who will define the way in which actions allow the agent to interact with the environment? And the interactions will vary?

In Q-learning, am I the one who will define the way in which actions allow the agent to interact with the environment, so that the way in which actions allow the agent to interact with the environment ...
will The J's user avatar
1 vote
1 answer
80 views

Could someone give a very simple example of Q-learning in a very small environment? [closed]

I would really like to see an example of Q-learning that I could read, so that I can learn Q-learning from scratch. I read some articles on the internet, but I found it a little difficult to ...
will The J's user avatar
-1 votes
1 answer
86 views

in simple words, what is the Q-learning algortimn steps? [closed]

I read this article about Q-learning: https://www.learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/ It teaches how to implement the algorithm using the Gym Python library. ...
will The J's user avatar
0 votes
0 answers
37 views

Multi-Agent DQN not learning for Clean Up Game - Reward slowly decreasing

The environment of the Clean Up game is simple: in a 25*18 grid world, there's dirt spawning on the left side and apples spawning on the other. Agents get a +1 reward for eating an apple (by stepping ...
Charles's user avatar
0 votes
1 answer
548 views

My first experience with gym environment has raised many questions, and I need some guidance [closed]

As I'm new to the AI/ML field, I'm still learning from various online materials. In this particular instance, I've been studying the Reinforcement Learning tutorial by deeplizard, specifically ...
Boris L.'s user avatar
  • 109
2 votes
1 answer
113 views

Find maximum value of unknown functions f(x,y)=z using reinforcement learning & neural network

is it possible to train a neural network to find the global maximum value of unknown functions like f(x,y)=z with reinforcement learning? Up until now I had only had experience with simple ...
Bubble's user avatar
  • 23
4 votes
1 answer
101 views

Finding the true Q-values in gymnaiusm

I'm very interested in the true Q-values of state-action pairs in the classic control environments in gymnasium. Contrary to the usual goal, the ordering of the Q-values itself is irrelevant; a very ...
Mark B's user avatar
  • 43
0 votes
3 answers
65 views

In Q-Learning the Q-Table is not considered a model of the game?

In a QTable you keep states and actions for the ongoing decision making, it somehow represents the knowledge of the world and your future decisions for this and any future instance of a game. In the ...
Raul Lapeira Herrero's user avatar
3 votes
1 answer
156 views

In Q-learning, how are Q values updated for the last state in the Q table?

In Q-learning, I know that the Q-values are updated using the Bellman equation. $$ Q^{new}(S_t,A_t) \leftarrow Q(S_t,A_t) + \alpha [R_{t+1} + \gamma \underset{a}{max} Q(S_{t+1},a) - Q(S_t,A_t)] $$ ...
gondorian's user avatar
1 vote
1 answer
85 views

Q learning achieves small reward in simple dice game

I am trying to train a Q learning agent on the following game: The states are parametrised by an integer $S \geq 0$ (representing the sum of the previous die rolls). In each step the player can choose ...
deepfloe's user avatar
  • 111
2 votes
1 answer
50 views

Should the Q-Value of a state-action tuple be updated, if $s_{t} == s_{t+1}$

Assuming the agent in my environment does an action, however the agent's state does not change. Does that mean the Q-Table gets updated regardless of respective states (current and next) being the ...
Ralph's user avatar
  • 21
0 votes
1 answer
105 views

Q learning: How to create output layer in which actions are combinations of multiple sub-actions

Suppose in my example I want an agent to learn a behavior that is made up of a combination of actions. So consider the following example with a tamagotchi like game: There are 5 pets and 3 actions ...
T. Kau's user avatar
  • 103
0 votes
0 answers
64 views

Problem of extremely varied reward value in DDQN

I am trying to train my model by DDQN agent after creating a customized environment in gym. I am stating my hyper-parameters and other details here. ...
Subhajit Saha's user avatar
2 votes
1 answer
66 views

Which Q function do we use to make our policy when using double Q learning?

I know this might be arbitrary, but I couldn't find any good information on this. As we update 2 q function in double q learning I was curios whether we average, or sum them together to get our policy....
IloveR's user avatar
  • 23
3 votes
1 answer
168 views

What is the motivation for using Q-Learning in RL?

In Spinning Up by OpenAI, it says the following regarding policy optimization methods and Q-Learning as ways of getting a good policy for RL. Trade-offs Between Policy Optimization and Q-Learning. ...
Justin T's user avatar
  • 145
2 votes
1 answer
170 views

Why does Advantage Learning help function approximators?

Many later RL algorithms like PPO or Duelling DQN estimate the advantage. I am not very sure of how that really helps. For instance, the actor loss for a simple actor critic algorithm is given by - <...
desert_ranger's user avatar
0 votes
1 answer
179 views

Will my Q values keep going up forever?

In Q-learning,the q values can be updated by the bellman equation. What happens with my Q values is that they keep going up forever, in accordance with the more I train. After 10,000 training episodes,...
Kyotiq's user avatar
  • 1
0 votes
1 answer
90 views

How can I get Q-Learning (1 step off policy) update from n-step off policy learning update?

In Sutton and Barto we have expressions for Q-Learning and n-step Off policy learning. The former ought to be the 1-step limit of the latter but I cannot see it working out that way. What am I missing?...
Borun Chowdhury's user avatar
0 votes
1 answer
174 views

Negative action-state values found during deep Q-learning

I'm training a simple deep q-learning algorithm with no experience buffer to solve the CartPole-v5 environment. I want to check for overestimation, therefore I'm ...
Gello's user avatar
  • 103
1 vote
1 answer
267 views

Effects of hyperparameters in Q-learning

While playing around with the learning rate and discount factor in the Q-learning algorithm, I noticed some behavior that I could not really understand myself. Firstly, I noticed that increasing the ...
perceptronEnthusiast420's user avatar
1 vote
1 answer
257 views

Is it necessary to have a constant reward in the terminal state?

I have downloaded the grid world project form this link. I have executed the project multiple times using: python gridworld.py -k 20 -a q -r -0.2 -s 90 I have ...
AAA's user avatar
  • 111
0 votes
1 answer
63 views

What does the figure in Q-learning vs Expected SARSA actually show?

I might be blind. But I wasn't able to find or figure out what the small difference between Q-learn and SARSA depicts in the following image; (src). What does the semi-circle show? and what does the ...
nammerkage's user avatar
0 votes
1 answer
196 views

Are there better loss functions than MSE for maze solver using deep learning?

I am a newbie in reinforcement learning, and I was doing a project on solving an agent maze solver using deep Q Learning. Currently, I am using the MSE loss function, but the agent is very slow or ...
Lim's user avatar
  • 1
1 vote
2 answers
249 views

How do you apply Q-learning when there are too many possible actions?

When the number of states in the Q-learning is large, we can refer to approximate Q-learning, but what should we do when we have a large number of actions?
znb's user avatar
  • 11
1 vote
1 answer
64 views

What is the meaning of $ (I - \gamma P^{\pi})^{-1} \left[\frac{\mu(a|s)}{\hat \pi_{\beta}(a|s)} \right](s, a)$?

In Theorem 3.1 of the conservative q-learning paper, what is the meaning of $$ (I - \gamma P^{\pi})^{-1} \left[\frac{\mu(a|s)}{\hat \pi_{\beta}(a|s)} \right](s, a)$$? I thought $(I - \gamma P^{\pi})^{-...
hongshan.li's user avatar
5 votes
0 answers
74 views

What exactly is non-delusional Q-learning?

Problems occur when we combine Q-learning with a function approximator. What exactly is the delusional-bias and non-delusional Q-learning? I am talking about the neurIPS 18 best paper Non-delusional Q-...
wrek's user avatar
  • 183
0 votes
1 answer
85 views

What would the "state space" and its Python implementation be for my simulation?

Context I'm trying to build a social-consensus simulation involving two intelligent agents. The simulation involves a graph/network of nodes. Nearly all of these nodes (> 90%) will be green agents. ...
The Pointer's user avatar
3 votes
1 answer
346 views

For which problem sizes is Deep Q-Learning suitable and why?

I am wondering for which problem sizes a Deep Q-Learning algorithm is most appropriate. For example, whether it is particularly suited for low complexity problems or not for high complexity problems. ...
user avatar
3 votes
2 answers
3k views

What is the difference between A2C and Q-Learning, and when to use one over the other?

I'm trying to get an accurate answer about the difference between A2C and Q-Learning. And when can we use each of them?
Hani's user avatar
  • 33

1
2 3 4 5
8