Questions tagged [q-learning]

For questions related to the Q-learning algorithm, which is a model-free and temporal-difference reinforcement learning algorithm that attempts to approximate the Q function, which is a function that, given a state s and an action a, returns a real number that represents the return (or value) of state s when action a is taken from s. Q-learning was introduced in the PhD thesis "Learning from Delayed Rewards" (1989) by Watkins.

Filter by
Sorted by
Tagged with
0 votes
0 answers
10 views

Does maximizing the value function and maximizing the state-action value function generate the same optimal policy?

In reinforcement learning, we define the optimal policy $\pi^*$ as the policy that maximizes the value of the state: $$ \pi_v^*=\underset{\pi}{\operatorname{argmax}} {V_{\pi}(s)} $$ In Q-learning, we ...
user avatar
  • 183
0 votes
1 answer
89 views

How do I create an AI controller for Pacman?

How do I create an AI controller, which can play pacman - by taking in pixel values (or some other data by represents the state) which perhaps runs on a separate thread, which can control the game? It ...
user avatar
  • 11
1 vote
1 answer
28 views

Is the described Q-table considered large?

I never saw any rule of thumb as to what size is said as large for a q-table but I have a Q-table with like 2500 entries. Is it considered large for a tabular approach? Anyone from experience can ...
user avatar
0 votes
1 answer
51 views

How to compare memory requirements for tabular Q-learning vs deep neural network?

I want to compare the space complexity/memory requirement of tabular Q-learning v.s. deep neural Q-network (DQN). I think DQN would be faster and Q-table has a disadvantage at large table sizes but ...
user avatar
0 votes
0 answers
30 views

What if we modify some Q-values while taking the action?

Just a passing thought about Q-learning. In the tabular Q-learning, what if I play around and modify any Q-values as I am using them to take actions? Would it be a violation of any (1) theoretical ...
user avatar
0 votes
1 answer
41 views

Is it possible to add states to the Q-table after the game has started?

I would like to implement Q-learning in a game. Here is the board: It's a 2 player game. At each turn, each player can put a pawn on a line of their choice. They can't choose the column. The right ...
user avatar
  • 3
0 votes
1 answer
33 views

How to deal with changing rewards in Q-learning? DQN?

I read the working of Q-learning through a grid-based taxi routing wherein a taxi has to pick and drop off a passenger from source to destination. Likewise, I have a routing problem and hence, I tried ...
user avatar
0 votes
0 answers
14 views

DDQN for Connect 4: Sudden explosion of Loss

I am trying to solve Connect 4 with DDQN through the self-play regime that was used for AlphaZero. That means, I let a student version play against a teacher version of itself and replace the teacher ...
user avatar
  • 31
2 votes
1 answer
428 views

Is Q-learning only capable of learning a deterministic policy?

I was following a reinforcement learning course on coursera and in this video at 2:57 the instructor says Expected SARSA and SARSA both allow us to learn an optimal $\epsilon$-soft policy, but, Q-...
user avatar
0 votes
1 answer
23 views

Using reinforcement learning for human-robot interaction [closed]

I have a scenario where a user is wanting to exercise and improve over time. They attend around 10 exercise sessions, doing 20 repititions of an exercise each session. I want to develop a ...
user avatar
  • 101
0 votes
1 answer
44 views

How to manage impossible actions? [closed]

I am using Q-learning in julia language. Because of the solver’s configuration, actions have to be defined as the whole action space and impossible actions have to be also considered. It means that I ...
user avatar
  • 33
0 votes
1 answer
55 views

Is the case of a big state space, should we use a softmax exploration policy rather than $\epsilon$-greedy for Q-Learning?

In Reinforcement Learning, epsilon-greedy policies are the most used exploration policies, but in case there is a big state space with impossible actions, wouldn't it be better to use soft-max ...
user avatar
  • 33
0 votes
1 answer
46 views

Could Softmax Action Selection be useful to solve an episodic task with more than 100000 possible states and 2000 actions?

I am new in the field of RL. I am trying to use tabular methods, Q-Learning for solving a problem that takes a lot of time for computation, so I would like to know if there are more efficient methods ...
user avatar
  • 33
2 votes
1 answer
74 views

Is it possible learning convergence is lost in Reinforcement Learning as the state space grows?

I am new in the AI field and I am trying to use Reinforcement Learning. Specifically, I am using tabular Q-Learning and SARSA algorithms to solve a sequential decision making problem. (I am using <...
user avatar
  • 33
0 votes
1 answer
43 views

Does $S_{t+1}$ denote the future information in Q-learning?

In Q-learning, $Q(S_t,a)$ is updated by the Bellman equation. $Q(S_t,a) = r + \max_{a'}(Q(S_{t+1},a'))$ where $S_{t+1}$ is the future state. Let's say $S$ denotes the stock price, does it mean we are ...
user avatar
  • 1
0 votes
1 answer
140 views

When calculating the max in DQN, do I have to calculate the Q for every possible action for a particular state?

I'm trying to implement the DQN paper using python/pytorch for my needs (https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf). I'm studying the main algorithm: I am a bit confused about the $\gamma* \max ...
user avatar
1 vote
1 answer
41 views

Can directly using expert policy in epsilon-greedy speed-up Q-learning?

In deep Q-learning we typically use epsilon-greedy policy during training. We choose a random action for a certain probability $\epsilon$, and choose the action that maximize the current Q-value ...
user avatar
  • 183
1 vote
1 answer
32 views

Is using Monte-Carlo estimate of returns in Deep Q Learning possible?

In all the tutorials of deep Q-learning (using neural networks) I have read so far, the state-action value function $Q(s,a)$ is learned by temporal difference learning. However, in policy gradient ...
user avatar
  • 183
0 votes
0 answers
19 views

Are temperature schedules used in Boltzmann Q-learning?

In Q-learning with the epsilon greedy method, I have seen implementations for training an agent that provide the possibility to pass epsilon schedules. Basically you can keep the epsilon constant or ...
user avatar
-1 votes
1 answer
207 views

what does the OpenAI ALE/Breakout-RAM-V5 observation return [closed]

I haven't been able to understand the output that OpenAI gym return for observation from this snippet ...
user avatar
2 votes
1 answer
76 views

Deep Q-Learning Model Effectiveness Improves then Crashes

I am implementing a Deep Q-Learning Algorithm. The model appears to improve but after awhile it just crashes and does just as well as if an agent was making random decisions. Shouldn't the behavior ...
user avatar
4 votes
2 answers
247 views

Is the optimal policy the one with the highest accumulative reward (Q-Learning vs SARSA)?

I was looking at the following diagram, The reward obtained with SARSA is higher. However, the path that Q learning chooses is eventually the optimal one, isn't it? Why is the SARSA reward higher if ...
user avatar
  • 282
1 vote
0 answers
65 views

How to use Actor-Critic RL with a categorical, state-dependent action space?

I have a problem where the agent is given an embedding vector to represent the state. Then it is also given a set of possible actions in the environment, let's say that the actions are each ...
user avatar
  • 276
1 vote
0 answers
31 views

What is the difference between the $Q_a$ calculated to update delta and those to select next action in the exploitation phase?

As the title suggests, I have a doubt about the computation of the $Q_a$ used to update the delta and the $Q_a$ used to select the next action in the exploitation phase, as shown below (source of ...
user avatar
2 votes
0 answers
72 views

Watkins' Q(λ) with function approximation: why is gradient not considered when updating eligibility traces for the exploitation phase?

I'm implementing the Watkins' Q(λ) algorithm with function approximation (in 2nd edition of Sutton & Barto). I am very confused about updating the eligibility traces because, at the beginning of ...
user avatar
0 votes
1 answer
128 views

How is $Q(s', a')$ calculated in SARSA and Q-Learning?

I have a question about how to update the Q-function in Q-learning and SARSA. Here (What are the differences between SARSA and Q-learning?) the following updating formulas are given: Q-Learning $$Q(s,...
user avatar
  • 213
0 votes
0 answers
59 views

Should you clip Q values if they start to grow indefinitely?

I am training the SAC algorithm for an environment where the rewards are small as shown below and the episode length is 84. I have a problem with the Q values that grow with each step. The following ...
user avatar
  • 167
1 vote
1 answer
126 views

What is meant by "two action selections" in SARSA?

I have some difficulties understanding the difference between Q-learning and SARSA. Here (What are the differences between SARSA and Q-learning?) the following updating formulas are given: Q-Learning $...
user avatar
  • 213
2 votes
0 answers
41 views

Reduction of state space of the game Connect Four to apply RL algorithms SARSA and Q-Learning

I would like to implement the reinforcement learning algorithms SARSA and Q-Learning for the board game Connect Four. I am familiar with the algorithms and know about their limitations regarding large ...
user avatar
  • 21
1 vote
0 answers
60 views

Can Q-learning be used for my scenario, and how might I do so?

I have already asked 2-3 general questions w.r.t Q learning and now I am asking a scenario specific one. I will try to be concise and understandable. I really really need help. Scenario: I have a ...
user avatar
0 votes
0 answers
35 views

Alternatives to neural networks for function approximation in Q learning?

I want to know if there is anything other than neural networks (or Deep NNs) that I can effectively use to perform function approximation? I am asking this w.r.t to the use of approximators in Q ...
user avatar
0 votes
0 answers
75 views

What do we actually 'approximate' when dealing with large state spaces in Q-learning?

I realized that my state space is very large in size. I had planned to use tabular Q-learning (Bellman equation to update the $Q(s, a)$ after each action taken). But this 'large space' realization has ...
user avatar
0 votes
0 answers
34 views

How to deal with Q-learning having low variance in predicted Q-values?

I have a neural network that takes the state (which contains a lot of data), and the possible action (which is very little data), and predicts the Q-value of the action. I am double Q-learning. I've ...
user avatar
  • 276
1 vote
1 answer
189 views

Why would SARSA diverge (but not Expected SARSA or Q-learning)?

In figure 6.3 (shown below) from Reinforcement Learning: An Introduction (second edition) by Sutton and Barto, SARSA is shown to perform worse asymptotically (after 100k episodes) than in the interim (...
user avatar
1 vote
0 answers
37 views

Multi-armed Bandit in optimization on graph edges selection

I have the problem, which I described below. I wonder if there exists a class of multi-armed bandit approaches that is related to it. I am working on computer networking optimization. In the simplest ...
user avatar
  • 21
1 vote
0 answers
21 views

Transferring a Q-learning policy to larger instances

How do I best transfer and fine-tune a Q-learning policy that was trained on small instances to large instances? Some more details on the problem: I am currently trying to derive a decision policy for ...
user avatar
4 votes
2 answers
2k views

When to use Value Iteration vs. Policy Iteration

Both value iteration and policy iteration are General Policy Iteration (GPI) algorithms. However, they differ in the mechanics of their updates. Policy Iteration seeks to first find a completed ...
user avatar
2 votes
1 answer
193 views

How to encourage the reinforcement-learning agent to reach the goal as quickly as possible, and what's the effect of discount factor?

I am trying to use reinforcement learning to solve a task and compare its performance to humans. The task is to find a single target in a fixed number of locations. At each step, the agent will pick ...
user avatar
  • 183
0 votes
0 answers
42 views

How do I solve a minimization problem with Q-learning learning?

I am trying to learn reinforcement learning by myself and so I have a lot of doubts. In particular, I am investigating how to use Q-learning in order to solve minimization problems. For example, ...
user avatar
  • 57
1 vote
1 answer
91 views

What is the difference between gradient decent in neural networks and temporal difference in reinforcement learning?

I am studying Q-learning in reinforcement learning. My question is about the Bellman equation. In Q-learning, the Bellman equation is often introduced as follows. \begin{align} Q_{new}(s,a) &= Q_{...
user avatar
2 votes
1 answer
136 views

How to handle invalid actions for next state in Q-learning loss

I am implementing an RL application in an environment with illegal moves. For handling the illegal moves, I am currently just picking an action as the maximum Q-value from the set of legal Q-values. ...
user avatar
2 votes
1 answer
102 views

What is the derivative of equation 1 in the paper "Conservative Q-Learning for Offline Reinforcement Learning"?

I am looking at the paper Conservative Q-Learning for Offline Reinforcement Learning, but I'm not sure how they proved theorem 3.1. Here is a screenshot of theorem 3.1. In the proof of theorem 3.1 ...
user avatar
  • 121
4 votes
1 answer
101 views

How to approach a blackjack-like card game with the possibility of cards being counted?

Consider a single-player card game which shares many characteristics to "unprofessional" (not being played in casino, refer point 2) Blackjack, i.e.: You're playing against a dealer with ...
user avatar
1 vote
1 answer
49 views

Why does Q-function training not query the Q-function value at unobserved states?

In the paper Conservative Q-Learning for Offline Reinforcement Learning, it is stated (section 3.1, page 3) that standard Q-function training does not query the Q-function value at unobserved states, ...
user avatar
  • 203
3 votes
1 answer
161 views

Is it really hard to learn in a stochastic environment?

I understand that a stochastic environment is one that does not always lead you to the desired state by giving a particular action $a$ (But the probability to change to a not desire state is fixed, ...
user avatar
  • 282
1 vote
1 answer
169 views

When to activate batch normalization and dropout in deep Q-learning?

In the vanilla version of deep Q-learning, there are three places where the Q-network is queried: When exploring. When training: a. When calculating the optimal value of the state reached by an ...
user avatar
1 vote
1 answer
154 views

In deep reinforcement learning, what is this model with state as input and value as output?

I was looking at this implementation for creating an agent for playing Tetris using DeepRL. This model uses "a state based on the statistics of the board after a potential action. All predictions ...
user avatar
  • 155
0 votes
0 answers
44 views

Defining states and possible actions in Q learning

I am trying to define the number of states and possible actions for a reinforcement learning problem that I want to solve with Q-learning, but I am a bit confused, as I'm totally new to reinforcement ...
user avatar
  • 1
0 votes
1 answer
49 views

Where can I find the original conference paper that introduced Q-learning and Deep Q-Learning?

I tried searching a lot, but I could neither find the paper that introduced Q-Learning nor the paper that introduced Deep Q Learning. If anyone knows anything about it please do tell me.
user avatar
1 vote
1 answer
76 views

Why do I get bad results no matter my neural network function approximator for parametrized Q-learning implementation for Contextual Bandits?

I'd like to ask you why, no matter my neural network function approximator for parametrized Q-learning implementation for a Contextual Bandits environment, I'm getting bad results. I don't know if it'...
user avatar
  • 509

1
2 3 4 5
7