Questions tagged [q-learning]

For questions related to the Q-learning algorithm, which is a model-free and temporal-difference reinforcement learning algorithm that attempts to approximate the Q function, which is a function that, given a state s and an action a, returns a real number that represents the return (or value) of state s when action a is taken from s. Q-learning was introduced in the PhD thesis "Learning from Delayed Rewards" (1989) by Watkins.

Filter by
Sorted by
Tagged with
19
votes
1answer
4k views

What is the relation between Q-learning and policy gradients methods?

As far as I understand, Q-learning and policy gradients (PG) are the two major approaches used to solve RL problems. While Q-learning aims to predict the reward of a certain action taken in a certain ...
11
votes
1answer
1k views

Why does DQN require two different networks?

I was going through this implementation of DQN and I see that on line 124 and 125 two different Q networks have been initialized. From my understanding, I think one network predicts the appropriate ...
9
votes
2answers
227 views

Why doesn't Q-learning converge when using function approximation?

The tabular Q-learning algorithm is guaranteed to find the optimal $Q$ function, $Q^*$, provided the following conditions regarding the learning rate are satisfied $\sum_{t} \alpha_t(s, a) = \infty$ $...
6
votes
1answer
150 views

Does AlphaZero use Q-Learning?

I was reading the AlphaZero paper Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm, and it seems they don't mention Q-Learning anywhere. So does AZ use Q-...
6
votes
1answer
337 views

How do we prove the n-step return error reduction property?

In section 7.1 (about the n-step bootstrapping) of the book Reinforcement Learning: An Introduction (2nd edition), by Andrew Barto and Richard S. Sutton, the authors write about what they call the "n-...
6
votes
1answer
93 views

Q-Learning the generic maze solution

After doing some exercices on Q-learning for maze solving, I wondered : my q-learning algorithms solve only ONE maze. The AI doesn't learn how to solve mazes, so how can I achieve it ? For instance ...
5
votes
1answer
273 views

Is the discount not needed in a deterministic environment for Reinforcement Learning?

I'm now reading a book titled as "Deep Reinforcement Learning Hands-On" and the author said the following on the chapter about AlphaGo Zero: Self-play In AlphaGo Zero, the NN is used to ...
5
votes
2answers
2k views

How to implement exploration function and learning rate in Q Learning

I'm trying to implement Q-learning (state-based representation and no neural / deep stuff) but I'm having a hard time getting it to learn anything. I believe my issue is with the exploration function ...
5
votes
2answers
162 views

Is Q-Learning suitable for continous (state or action) spaces?

Many examples work with a table based method for Q-Llearning. This may be suitable for discrete state(observation) or actions like a robot in a grid world but is there a way to use Q-Learning for ...
5
votes
1answer
111 views

Reinforcement Learning (Fitted Q): Qn on Concept & Implementation

I hope to get some clarifications on Fitted Q-Learning ('FQL'). My Research So Far I've read Sutton's book (specifically, chp 6 to 10), Ernst et al and this paper. I know that ...
5
votes
2answers
994 views

Why is the target $r + \gamma \max_{a'} Q(s', a'; \theta_i^-)$ in the loss function of the DQN architecture?

In the paper Human-level control through deep reinforcement learning, the DQN architecture is presented, where the loss function is as follows $$ L_i(\theta_i) = \mathbb{E}_{(s, a, r, s') \sim U(D)} \...
4
votes
2answers
139 views

What does the symbol $\mathbb E$ mean in these equations?

I came across some papers that use $\mathbb E$ in equations, in particular, this paper: https://arxiv.org/pdf/1511.06581.pdf. Here is some equations from the paper that uses it: $Q^\pi \left(s,a \...
4
votes
2answers
126 views

Deciding on a reward per each action in a given state (Q-learning)

I looked for existing posts on Stack Exchange, which kind of answer the questions about the reward system and reward function, but not specifically what I want to ask here, which is how do you ...
4
votes
1answer
58 views

Is tabular Q-learning considered interpretable?

I am working on a research project in a domain where other related works have always resorted to deep Q-learning. The motivation of my research stems from the fact that the domain has an inherent ...
4
votes
1answer
96 views

Q-learning, am I interpreting correctly $Q(s,a) = r + \gamma \max_{a'} Q(s',a')$?

Ok, due to previous question I was pointed to use reinfrocement learning. So far what I understood from random websites is the following: there is a Q(s,a) function involved I can assume my neural ...
4
votes
2answers
85 views

Can DQN perform better than Double DQN?

I'm training both kind of agents against an environment but DQN performs significantly better than Double DQN. As I've saw here, Double DQN use to perform better than DQN. Am I doing something wrong ...
4
votes
2answers
541 views

How does Q-learning work in stochastic environments?

The Q function uses the (current and future) states to determine the action that gets the highest reward. However, in a stochastic environment, the current action (at the current state) does not ...
4
votes
0answers
51 views

Inconsistent definitions of the retrace

In Section 4.3 of paper Learning by Playing - Solving Sparse Reward Tasks from Scratch, the authors define Retrace as $$ Q^{ret}=\sum_{j=i}^\infty\left(\gamma^{j-i}\prod_{k=i}^jc_k\right)[r(s_j,a_j)+\...
3
votes
2answers
585 views

Is Q-learning a type of model-based RL?

Model-based RL creates a model of the transition function. Tabular Q-Learning does this iteratively (without directly optimizing for the transition function). So, does this make tabular Q-learning a ...
3
votes
1answer
399 views

Number of Neuron in Q-Learning of Chess

So I just read about deep Q-Learning which is using a neural network for optimization instead of Q-table. I saw the example here: https://yanpanlau.github.io/2016/07/10/FlappyBird-Keras.html and he ...
3
votes
2answers
54 views

Why is the max a non-expansive operator?

In certain reinforcement learning (RL) proofs, the operators involved are assumed to be non-expansive. For example, on page 6 of the paper Generalized Markov Decision Processes: Dynamic-programming ...
3
votes
2answers
280 views

My DQN is stuck and can't see where the problem is

I'm trying to replicate the DeepMind paper results, so I implemented my own DQN. I left it training for more than 4 million frames (more than 2000 episodes) on SpaceInvaders-v4 (OpenAI-Gym) and it ...
3
votes
1answer
162 views

How q-learning solves the issue with value iteration in model-free settings

I can't understand what is the problem in applying value-iteration in reinforcement learning setting (where we don't the reward and transition probabilities). In one of the lectures, the guy said it ...
3
votes
1answer
279 views

State representation of position in 2D plane for Reinforcement Learning (Q Learning)

I recently finished Course on RL by David Silver (on YT) and thought about trying it out on simple application in Unity Game Engine, where I've built simple labyrint with ball and want to teach the ...
3
votes
1answer
37 views

Can exogenous variables be state features in reinforcement learning?

I have a question about state representation of Q-learning or DQN algorithm. I'm still a beginner of RL, so I'm not sure that is it suitable to take exogenous variables as state features. For example,...
3
votes
1answer
41 views

Is it possible to have a dynamic $Q$-function?

I am trying to use Q-learning for energy optimization. I only wish to have states that will be visited by the learning agent, and, for each state, I have a function that generates possible actions, so ...
3
votes
1answer
36 views

How do I represent a multi-dimensional state using a neural network?

I have a set of 15 unique playing cards from a deck of 52 playing cards. A given state is represented by the respective card values in the set of 15 cards, where the card value is a prime number ...
3
votes
1answer
106 views

Can Q-learning be used to derive a stochastic policy?

In my understanding, Q-learning gives you a deterministic policy. However, can we use some technique to build a meaningful stochastic policy from the learned Q values? I think that simply using a ...
3
votes
1answer
274 views

What are good learning strategies for Deep Q-Network with opponents?

I am trying to find out what are some good learning strategies for Deep Q-Network with opponents. Let's consider the well known game Tic-Tac-Toe as an example: How should an opponent be implemented ...
3
votes
1answer
69 views

Experience Replay Not Always Giving Better Results

I have recently started working on a control problem using a Deep Q Network as proposed by DeepMind (https://arxiv.org/abs/1312.5602). Initially, I implemented it without Experience Replay. The ...
3
votes
2answers
281 views

Can Q-learning be used in a POMDP?

Can Q-learning (and SARSA) be directly used in a Partially Observable Markov Decision Process (POMDP)? If not, why not? My intuition is that the policies learned will be terrible because of partial ...
3
votes
1answer
45 views

How to apply or extend the $Q(\lambda)$ algorithm to semi-MDPs?

I want to model an SMDP such that time is discretized and the transition time between the two states follows an exponential distribution and there would be no reward between the transition. Can I ...
3
votes
1answer
79 views

Reason for issues with correlation in the dataset in DQN

From the paper Human level Control through DeepRL, the correlation in the data causes instability in the network and may causes the network to ...
3
votes
3answers
391 views

Difficulty in understanding identifiability in the “Dueling Network Architectures for Deep Reinforcement Learning” paper

I have difficulty understanding the following paragraph in the below excerpts from page 4 to page 5 from the paper Dueling Network Architectures for Deep Reinforcement Learning. The author said "we ...
3
votes
1answer
200 views

Snake path finding variant : Algorithm choice

I am working on a project which maps to a variant of path finding problem. I am new to this area and I would be very grateful if you could give suggestions/ point to libraries for relevant algorithms. ...
3
votes
1answer
323 views

Which Reinforcement Learning algorithms are efficient for episodic problems?

I have some episodic datasets extracted from a turn-based RTS game in which the current actions leading to the next state doesn’t determine the final solution/outcome of the episode. The learning is ...
3
votes
0answers
69 views

How can I use Q-learning for inventory decision making?

I am trying to model operational decisions in inventory control. The control policy is base stock with a fixed stock level $S$. That is replenishment orders are placed for every demand arrival to take ...
3
votes
2answers
65 views

What is the difference between return and expected return?

At a time step $t$, for a state $S_{t}$, the return is defined as the discounted cumulative reward from that time step $t$. If an agent is following a policy (which in itself is a probability ...
3
votes
0answers
39 views

Deep Q-Learning agent poor performing actions. Need help optimizing

I'm trying to make deep q-learning agent from https://keon.io/deep-q-learning My environment looks like this: https://imgur.com/a/OnbiCtV As you can see my agent is a circle and there is one gray ...
3
votes
1answer
45 views

Why Q2 is a more or less independant estimate in Twin Delayed DDPG (TD3)?

TD3 is inspired from both double Q-learning and double DQN. In double Q-learning, I understand that Q1 and Q2 are independent because they are trained on different samples. In double DQN, I understand ...
3
votes
1answer
2k views

Training AI to play NES/SNES games on NN python

I am currently getting into Deep Learning and would like to set up an environment for training an Artificial Neural Network or NEAT to play simple video games on NES (Mario etc.) and SNES ( Donkey ...
2
votes
1answer
108 views

How do updates in SARSA and Q-learning differ in code?

The update rules for Q-learning and SARSA each are as follows: Q Learning: $$Q(s_t,a_t)←Q(s_t,a_t)+α[r_{t+1}+γ\max_{a'}Q(s_{t+1},a')−Q(s_t,a_t)]$$ SARSA: $$Q(s_t,a_t)←Q(s_t,a_t)+α[r_{t+1}+γQ(s_{t+...
2
votes
1answer
123 views

Should the exploration rate be reset after each trial in Q-learning?

As the title says, should I reset the exploration rate between trials? I am currently doing the Open AI pendulum task and after a number of trials my model started playing but did not take any ...
2
votes
1answer
24 views

Q Learning for FrozenLake environment not converging to V* values from Value Iteration

I am trying to learn tabular Q learning, value iteration using the classical algorithms (no neural networks) by using a table of states and actions. I was trying it out on FrozenLake environment in ...
2
votes
1answer
171 views

Meaning of Actor Output in Actor Critic Reinforcement Learning

In actor critic, The equations for calculating the loss in actor critic are an actor loss (parameterized by $\theta$) $$log[\pi_\theta(s_t,a_t)]Q_w(s_t,a_t)$$ and a critic loss (parameterized by ...
2
votes
1answer
65 views

Convergence in multi-agent environment

I have a multi-agent environment where agents are trying to optimise the overall energy consumption of their group. Agents can exchange energy between themselves (actions for exchange of energy ...
2
votes
1answer
806 views

How to use DQN to handle an imperfect but complete information game?

I'm currently having troubles to win against a random bot playing the Schieber Jass game. It is a imperfect card information game. (famous in switzerland https://www.schieber.ch/) The environement I'...
2
votes
1answer
45 views

What happens to the optimal value function if the reward is multiplied by a constant?

What happens to the optimal action-value function, $q_*$ if the reward is multiplied by a constant $c$? Is the optimal action-value function also multiplied by such a constant?
2
votes
1answer
41 views

If deep Q learning involves adjusting the value function for a specific policy, then how do I choose the right policy?

I wrote a simple implementation of Flappy Bird in Python, and now I'm trying to train an agent to play it at a reasonable skill level using TFLearn. I feed the network an input vector of size 4: ...
2
votes
1answer
43 views

Is there any grid world dataset or generator for reinforcement learning?

I would like to start programming a multi task reinforcement learning model. For this, I need not just one maze or grid world (or just model-based), but many with different reward functions. So, I am ...