Questions tagged [q-learning]

For questions related to the Q-learning algorithm, which is a model-free and temporal-difference reinforcement learning algorithm that attempts to approximate the Q function, which is a function that, given a state s and an action a, returns a real number that represents the return (or value) of state s when action a is taken from s. Q-learning was introduced in the PhD thesis "Learning from Delayed Rewards" (1989) by Watkins.

Filter by
Sorted by
Tagged with
1 vote
0 answers
12 views

Why slow-changing policy invalidates Double DQN approach in TD3 paper?

In the paper describing TD3 (https://arxiv.org/abs/1802.09477), the authors say that they could not effectively address the Q-learning overestimation bias by using different networks for maximizing ...
Jerry Ding's user avatar
2 votes
1 answer
94 views

What are the similarities between Q-learning and Value Iteration?

This is the explanation of value iteration in our notes where you keep applying bellman optimality equation till it stops changing and then acting greedily wrt the value function gives the optimal ...
ace239's user avatar
  • 23
1 vote
1 answer
61 views

Can Q-learning be used to create new creative solutions by combining different factors and characteristics?

References from Wikipedia: https://en.wikipedia.org/wiki/Q-learning https://en.wikipedia.org/wiki/Markov_decision_process Q-learning can be used to create new creative solutions, combining different ...
will The J's user avatar
1 vote
1 answer
39 views

In Q-learning, states need to be just X and Y positions of the agent, or a state can be several other characteristics?

For example, in this article: https://www.learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/, which explains Q-learnig, teaches the Smartcab problem, the environment is a ...
will The J's user avatar
2 votes
1 answer
49 views

Is Q-learning limited to just visual scenarios, or is it much broader and can it be used to solve non-visual problems as well?

For example, in this article: https://www.learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/, which explains Q-learnig, teaches the Smartcab problem, it has a visual ...
will The J's user avatar
1 vote
2 answers
298 views

In Q-learning, Am I the one who will define the way in which actions allow the agent to interact with the environment? And the interactions will vary?

In Q-learning, am I the one who will define the way in which actions allow the agent to interact with the environment, so that the way in which actions allow the agent to interact with the environment ...
will The J's user avatar
1 vote
1 answer
60 views

Could someone give a very simple example of Q-learning in a very small environment? [closed]

I would really like to see an example of Q-learning that I could read, so that I can learn Q-learning from scratch. I read some articles on the internet, but I found it a little difficult to ...
will The J's user avatar
-1 votes
1 answer
81 views

in simple words, what is the Q-learning algortimn steps? [closed]

I read this article about Q-learning: https://www.learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/ It teaches how to implement the algorithm using the Gym Python library. ...
will The J's user avatar
0 votes
0 answers
26 views

Multi-Agent DQN not learning for Clean Up Game - Reward slowly decreasing

The environment of the Clean Up game is simple: in a 25*18 grid world, there's dirt spawning on the left side and apples spawning on the other. Agents get a +1 reward for eating an apple (by stepping ...
Charles's user avatar
0 votes
1 answer
91 views

My first experience with gym environment has raised many questions, and I need some guidance [closed]

As I'm new to the AI/ML field, I'm still learning from various online materials. In this particular instance, I've been studying the Reinforcement Learning tutorial by deeplizard, specifically ...
Boris L.'s user avatar
  • 109
2 votes
1 answer
57 views

Find maximum value of unknown functions f(x,y)=z using reinforcement learning & neural network

is it possible to train a neural network to find the global maximum value of unknown functions like f(x,y)=z with reinforcement learning? Up until now I had only had experience with simple ...
Bubble's user avatar
  • 23
4 votes
1 answer
94 views

Finding the true Q-values in gymnaiusm

I'm very interested in the true Q-values of state-action pairs in the classic control environments in gymnasium. Contrary to the usual goal, the ordering of the Q-values itself is irrelevant; a very ...
Mark B's user avatar
  • 43
0 votes
3 answers
51 views

In Q-Learning the Q-Table is not considered a model of the game?

In a QTable you keep states and actions for the ongoing decision making, it somehow represents the knowledge of the world and your future decisions for this and any future instance of a game. In the ...
Raul Lapeira Herrero's user avatar
3 votes
1 answer
107 views

In Q-learning, how are Q values updated for the last state in the Q table?

In Q-learning, I know that the Q-values are updated using the Bellman equation. $$ Q^{new}(S_t,A_t) \leftarrow Q(S_t,A_t) + \alpha [R_{t+1} + \gamma \underset{a}{max} Q(S_{t+1},a) - Q(S_t,A_t)] $$ ...
gondorian's user avatar
1 vote
1 answer
78 views

Q learning achieves small reward in simple dice game

I am trying to train a Q learning agent on the following game: The states are parametrised by an integer $S \geq 0$ (representing the sum of the previous die rolls). In each step the player can choose ...
deepfloe's user avatar
  • 111
2 votes
1 answer
46 views

Should the Q-Value of a state-action tuple be updated, if $s_{t} == s_{t+1}$

Assuming the agent in my environment does an action, however the agent's state does not change. Does that mean the Q-Table gets updated regardless of respective states (current and next) being the ...
Ralph's user avatar
  • 21
0 votes
1 answer
85 views

Q learning: How to create output layer in which actions are combinations of multiple sub-actions

Suppose in my example I want an agent to learn a behavior that is made up of a combination of actions. So consider the following example with a tamagotchi like game: There are 5 pets and 3 actions ...
T. Kau's user avatar
  • 103
0 votes
0 answers
64 views

How to use "states" as vectors in Q-Learning?

How do I use states as vectors in Q-Learning or any other RL Algorithms. Let's say I have state as a vector with probabilities[0,1], and I have to take an action if the state is valid (with ...
rainarashika's user avatar
0 votes
0 answers
54 views

Problem of extremely varied reward value in DDQN

I am trying to train my model by DDQN agent after creating a customized environment in gym. I am stating my hyper-parameters and other details here. ...
Subhajit Saha's user avatar
0 votes
0 answers
24 views

Dealing with long running tasks in Q Learning

Assuming Q Learning is applied not to low level behaviors (such as taking a step; drawing/playing a card; moving one piece on a board), but rather longer running high-level behaviors (e.g. Moving from ...
Ralph's user avatar
  • 21
2 votes
1 answer
57 views

Which Q function do we use to make our policy when using double Q learning?

I know this might be arbitrary, but I couldn't find any good information on this. As we update 2 q function in double q learning I was curios whether we average, or sum them together to get our policy....
IloveR's user avatar
  • 23
3 votes
1 answer
153 views

What is the motivation for using Q-Learning in RL?

In Spinning Up by OpenAI, it says the following regarding policy optimization methods and Q-Learning as ways of getting a good policy for RL. Trade-offs Between Policy Optimization and Q-Learning. ...
Justin T's user avatar
  • 145
2 votes
1 answer
163 views

Why does Advantage Learning help function approximators?

Many later RL algorithms like PPO or Duelling DQN estimate the advantage. I am not very sure of how that really helps. For instance, the actor loss for a simple actor critic algorithm is given by - <...
desert_ranger's user avatar
0 votes
0 answers
23 views

Problem on evaluating DQN, on a Vehicle Routing Problem (VRP)

I am running this DQN algorithm that is trying to minimize the total distance traveled by a vehicle (VRP). In the training, as you can see in the images, everything works fine: the loss is decreasing, ...
elizabeth's user avatar
0 votes
1 answer
112 views

Will my Q values keep going up forever?

In Q-learning,the q values can be updated by the bellman equation. What happens with my Q values is that they keep going up forever, in accordance with the more I train. After 10,000 training episodes,...
Kyotiq's user avatar
  • 1
0 votes
1 answer
68 views

How can I get Q-Learning (1 step off policy) update from n-step off policy learning update?

In Sutton and Barto we have expressions for Q-Learning and n-step Off policy learning. The former ought to be the 1-step limit of the latter but I cannot see it working out that way. What am I missing?...
Borun Chowdhury's user avatar
0 votes
0 answers
37 views

Relationship between TD control algorithm (SARSA) and logistic regression in two-armed bandit task

I have been looking for a way to model behavioral data (from rodents) in a nonstationary 2-armed bandit task. In this task the rodent can nose poke either on a left or a right port, and it will get a ...
Alex Legaria's user avatar
0 votes
1 answer
116 views

Negative action-state values found during deep Q-learning

I'm training a simple deep q-learning algorithm with no experience buffer to solve the CartPole-v5 environment. I want to check for overestimation, therefore I'm ...
Gello's user avatar
  • 103
1 vote
1 answer
214 views

Effects of hyperparameters in Q-learning

While playing around with the learning rate and discount factor in the Q-learning algorithm, I noticed some behavior that I could not really understand myself. Firstly, I noticed that increasing the ...
perceptronEnthusiast420's user avatar
1 vote
1 answer
181 views

Is it necessary to have a constant reward in the terminal state?

I have downloaded the grid world project form this link. I have executed the project multiple times using: python gridworld.py -k 20 -a q -r -0.2 -s 90 I have ...
AAA's user avatar
  • 111
0 votes
1 answer
48 views

What does the figure in Q-learning vs Expected SARSA actually show?

I might be blind. But I wasn't able to find or figure out what the small difference between Q-learn and SARSA depicts in the following image; (src). What does the semi-circle show? and what does the ...
nammerkage's user avatar
0 votes
1 answer
146 views

Are there better loss functions than MSE for maze solver using deep learning?

I am a newbie in reinforcement learning, and I was doing a project on solving an agent maze solver using deep Q Learning. Currently, I am using the MSE loss function, but the agent is very slow or ...
Lim's user avatar
  • 1
1 vote
2 answers
167 views

How do you apply Q-learning when there are too many possible actions?

When the number of states in the Q-learning is large, we can refer to approximate Q-learning, but what should we do when we have a large number of actions?
znb's user avatar
  • 11
1 vote
1 answer
54 views

What is the meaning of $ (I - \gamma P^{\pi})^{-1} \left[\frac{\mu(a|s)}{\hat \pi_{\beta}(a|s)} \right](s, a)$?

In Theorem 3.1 of the conservative q-learning paper, what is the meaning of $$ (I - \gamma P^{\pi})^{-1} \left[\frac{\mu(a|s)}{\hat \pi_{\beta}(a|s)} \right](s, a)$$? I thought $(I - \gamma P^{\pi})^{-...
hongshan.li's user avatar
5 votes
0 answers
69 views

What exactly is non-delusional Q-learning?

Problems occur when we combine Q-learning with a function approximator. What exactly is the delusional-bias and non-delusional Q-learning? I am talking about the neurIPS 18 best paper Non-delusional Q-...
wrek's user avatar
  • 183
0 votes
1 answer
76 views

What would the "state space" and its Python implementation be for my simulation?

Context I'm trying to build a social-consensus simulation involving two intelligent agents. The simulation involves a graph/network of nodes. Nearly all of these nodes (> 90%) will be green agents. ...
The Pointer's user avatar
3 votes
1 answer
289 views

For which problem sizes is Deep Q-Learning suitable and why?

I am wondering for which problem sizes a Deep Q-Learning algorithm is most appropriate. For example, whether it is particularly suited for low complexity problems or not for high complexity problems. ...
user avatar
2 votes
2 answers
2k views

What is the difference between A2C and Q-Learning, and when to use one over the other?

I'm trying to get an accurate answer about the difference between A2C and Q-Learning. And when can we use each of them?
Hani's user avatar
  • 23
0 votes
1 answer
222 views

How to deal with delay in reinforcement learning, an unclear case

According to the question in How to deal with the time delay in reinforcement learning?, we can tell the delay in the reinforcement learning can be observation delay, action delay and reward delay. I ...
CharlesC's user avatar
1 vote
2 answers
503 views

Does maximizing the value function and maximizing the state-action value function generate the same optimal policy?

In reinforcement learning, we define the optimal policy $\pi^*$ as the policy that maximizes the value of the state: $$ \pi_v^*=\underset{\pi}{\operatorname{argmax}} {V_{\pi}(s)} $$ In Q-learning, we ...
Cloudy's user avatar
  • 213
0 votes
1 answer
440 views

How do I create an AI controller for Pacman?

How do I create an AI controller, which can play pacman - by taking in pixel values (or some other data by represents the state) which perhaps runs on a separate thread, which can control the game? It ...
R3sist's user avatar
  • 9
1 vote
1 answer
248 views

Is the described Q-table considered large?

I never saw any rule of thumb as to what size is said as large for a q-table but I have a Q-table with like 2500 entries. Is it considered large for a tabular approach? Anyone from experience can ...
knowledge_seeker's user avatar
0 votes
1 answer
241 views

How to compare memory requirements for tabular Q-learning vs deep neural network?

I want to compare the space complexity/memory requirement of tabular Q-learning v.s. deep neural Q-network (DQN). I think DQN would be faster and Q-table has a disadvantage at large table sizes but ...
knowledge_seeker's user avatar
0 votes
0 answers
71 views

What if we modify some Q-values while taking the action?

Just a passing thought about Q-learning. In the tabular Q-learning, what if I play around and modify any Q-values as I am using them to take actions? Would it be a violation of any (1) theoretical ...
knowledge_seeker's user avatar
0 votes
1 answer
188 views

Is it possible to add states to the Q-table after the game has started?

I would like to implement Q-learning in a game. Here is the board: It's a 2 player game. At each turn, each player can put a pawn on a line of their choice. They can't choose the column. The right ...
Dunno's user avatar
  • 103
0 votes
1 answer
389 views

How to deal with changing rewards in Q-learning? DQN?

I read the working of Q-learning through a grid-based taxi routing wherein a taxi has to pick and drop off a passenger from source to destination. Likewise, I have a routing problem and hence, I tried ...
knowledge_seeker's user avatar
2 votes
1 answer
733 views

Is Q-learning only capable of learning a deterministic policy?

I was following a reinforcement learning course on coursera and in this video at 2:57 the instructor says Expected SARSA and SARSA both allow us to learn an optimal $\epsilon$-soft policy, but, Q-...
ketan dhanuka's user avatar
0 votes
1 answer
26 views

Using reinforcement learning for human-robot interaction [closed]

I have a scenario where a user is wanting to exercise and improve over time. They attend around 10 exercise sessions, doing 20 repititions of an exercise each session. I want to develop a ...
caaax's user avatar
  • 101
0 votes
1 answer
184 views

How to manage impossible actions? [closed]

I am using Q-learning in julia language. Because of the solver’s configuration, actions have to be defined as the whole action space and impossible actions have to be also considered. It means that I ...
Aquila's user avatar
  • 33
0 votes
1 answer
252 views

Is the case of a big state space, should we use a softmax exploration policy rather than $\epsilon$-greedy for Q-Learning?

In Reinforcement Learning, epsilon-greedy policies are the most used exploration policies, but in case there is a big state space with impossible actions, wouldn't it be better to use soft-max ...
Aquila's user avatar
  • 33

1
2 3 4 5
8