Questions tagged [q-learning]

For questions related to the Q-learning algorithm, which is a model-free and temporal-difference reinforcement learning algorithm that attempts to approximate the Q function, which is a function that, given a state s and an action a, returns a real number that represents the return (or value) of state s when action a is taken from s. Q-learning was introduced in the PhD thesis "Learning from Delayed Rewards" (1989) by Watkins.

Filter by
Sorted by
Tagged with
3
votes
1answer
561 views

What are the differences between the DQN variants?

There are several variants of the DQN model. For example, double DQN, duelling DQN, prioritized DQN, distributed prioritized DQN, episodic memory DQN, asynchronous n-step DQN and multiple DQN. What ...
3
votes
2answers
89 views

Why is the max a non-expansive operator?

In certain reinforcement learning (RL) proofs, the operators involved are assumed to be non-expansive. For example, on page 6 of the paper Generalized Markov Decision Processes: Dynamic-programming ...
0
votes
0answers
22 views

How to serve a deep q network using tensorflow serving?

How to Serve a Deep Q Network using Tensorflow Serving. I have built a Deep Q Network using Multilayer Perceptron. Is it possible to serve it?
1
vote
0answers
38 views

Comparison and understanding of different version of DDQN?

There are several version of DDQN floating around. Sutton gives one that is a simple symmetric random update of the two Q functions. I think other papers (Silver paper for example) use a kind of ...
4
votes
1answer
69 views

How to apply or extend the $Q(\lambda)$ algorithm to semi-MDPs?

I want to model an SMDP such that time is discretized and the transition time between the two states follows an exponential distribution and there would be no reward between the transition. Can I ...
2
votes
1answer
109 views

Is there any grid world dataset or generator for reinforcement learning?

I would like to start programming a multi task reinforcement learning model. For this, I need not just one maze or grid world (or just model-based), but many with different reward functions. So, I am ...
2
votes
1answer
335 views

Is there a way to train an RL agent without any environment?

Following Deep Q-learning from Demonstrations, I'd like to avoid potentially unsafe behavior during early learning by making use of supervised learning with demonstration data. However, the ...
3
votes
1answer
2k views

My DQN is stuck and can't see where the problem is

I'm trying to replicate the DeepMind paper results, so I implemented my own DQN. I left it training for more than 4 million frames (more than 2000 episodes) on SpaceInvaders-v4 (OpenAI-Gym) and it ...
1
vote
1answer
586 views

What is happening when a reinforcement learning agent trains itself out of desired behavior?

I have a reinforcement learning agent with both a positive and a negative terminal state. After each episode during training, I am recording whether a success or failure occurred, and then I can ...
3
votes
1answer
602 views

Why does Deep Q Network outputs multiple Q values?

I am learning Deep RL following this tutorial: https://medium.freecodecamp.org/an-introduction-to-deep-q-learning-lets-play-doom-54d02d8017d8 I understand everything but one detail: This image shows ...
1
vote
1answer
89 views

Robot Arm Deep Q Learning Actions

Hello I am new to reinforcement learning and robotics. So far I have an understanding of the concept on 2D world. You can make agent move one step in one direction. However, how do you define movement ...
1
vote
2answers
116 views

Is there more than one Q-matrix update formula?

I asked a question a while ago here and since then I've been solving the issues within my code but I have just one question... This is the formula for updating the Q-Matrix in Q-Learning: $$Q(s_t, ...
2
votes
1answer
94 views

What is the next state for a two-player board game?

I'm using Q-learning to train an agent to play a board game (e.g. chess, draughts or go). The agent takes an action while in state $S$, but then what is the next state (that is, $S'$)? Is $S'$ now ...
3
votes
1answer
193 views

How do updates in SARSA and Q-learning differ in code?

The update rules for Q-learning and SARSA each are as follows: Q Learning: $$Q(s_t,a_t)←Q(s_t,a_t)+α[r_{t+1}+γ\max_{a'}Q(s_{t+1},a')−Q(s_t,a_t)]$$ SARSA: $$Q(s_t,a_t)←Q(s_t,a_t)+α[r_{t+1}+γQ(s_{t+...
2
votes
1answer
469 views

How do I convert table-based to neural network-based Q-learning?

I've used a table to represent the Q function, while an agent is being trained to catch the cheese without touching the walls. The first and last row (and column) of the matrix are associated with ...
3
votes
1answer
671 views

Can Q-learning be used to derive a stochastic policy?

In my understanding, Q-learning gives you a deterministic policy. However, can we use some technique to build a meaningful stochastic policy from the learned Q values? I think that simply using a ...
2
votes
1answer
105 views

Will Q-learning converge to the optimal state-action function when the reward periodically changes?

Imagine that the agent receives a positive reward upon reaching a state 𝑠. Once the state 𝑠 has been reached the positive reward associated with it vanishes and appears somewhere else in the state ...
3
votes
1answer
518 views

Meaning of Actor Output in Actor Critic Reinforcement Learning

In actor critic, The equations for calculating the loss in actor critic are an actor loss (parameterized by $\theta$) $$log[\pi_\theta(s_t,a_t)]Q_w(s_t,a_t)$$ and a critic loss (parameterized by ...
1
vote
2answers
107 views

When are Q values calculated in experience replay?

In experience replay, the update rule follows the loss: $$ L_i(\theta_i) = \mathbb{E}_{(s_t, a_t, r_t, s_{t+1}) \sim U(D)} \left[ \left(r_t + \gamma \max_{a_{t+1}} Q(s_{t+1}, a_{t+1}; \theta_i^-) - Q(...
2
votes
1answer
142 views

What will Q-values look like in self-play tic-tac-toe?

This corresponds to Exercise 1.1 of RLBook, and a discussion followed from here. Considering two reward schemes- Win = +1, Draw = 0, Loss = -1 Win = +1, Draw or Loss = 0 Can we say something about ...
1
vote
1answer
551 views

Why isn't my Q-Learning agent able to play tic-tac-toe?

I tried to build a Q-learning agent which you can play tic tac toe against after training. Unfortunately, the agent performs pretty poorly. He tries to win but does not try to make me 'not winning' ...
2
votes
1answer
387 views

Using the opponent's mixed strategy in estimating the state value in minimax Q learning

In the paper Markov games as a framework for multi-agent reinforcement learning (which introduces the minimax Q Learning algorithm), at the bottom left of page 3, my understanding is that the author ...
1
vote
1answer
56 views

How do I calculate $max_{a′}Q(s′,a′,w−)$ when it is represented as a neural network?

Consider the following loss function $$ L(\mathbf{w}) = [(r + \gamma max_{a'} Q(s', a', \mathbf{w^-})) - Q(s, a, \mathbf{w})]^2 $$ where $Q(s, a, \mathbf{w^-})$ and $Q(s, a, \mathbf{w})$ are ...
2
votes
1answer
224 views

Concrete Example for Q Learning

I am not sure if I understood the q learning algorithms correctly. Therefore I would give a concrete example and ask if someone can tell me how to update the q value correctly. First I initialized ...
0
votes
1answer
361 views

How do I update the Q values of a Deep Q Network when exploring?

I am trying to implement a Deep Q Network to play Asteroids. Unfortunately, I am not sure how to calculate the Q value exactly, if I am exploring. For example, the agent is exploring for 1 second (...
5
votes
3answers
1k views

What is a high dimensional state in reinforcement learning?

In the DQN paper, it is written that the state-space is high dimensional. I am a little bit confused about this terminology. Suppose my state is a high dimensional vector of length $N$, where $N$ is a ...
11
votes
2answers
1k views

How do we prove the n-step return error reduction property?

In section 7.1 (about the n-step bootstrapping) of the book Reinforcement Learning: An Introduction (2nd edition), by Andrew Barto and Richard S. Sutton, the authors write about what they call the "n-...
1
vote
0answers
103 views

Exploration rate decay and training in Q learning

I'm trying to replicate the results of the DeepMind's paper with Breakout included in OpenAI Gym. I wonder how much frames should I keep until I reach the fixed exploration rate. Actually it reaches ...
0
votes
1answer
940 views

DQN it's not working properly

I'm trying to build a DQN to replicate the DeepMind results. I'm doing with a simple DQN for the moment, but it isn't learning properly: after +5000 episodes, it couldn't get more than 9-10 points. ...
2
votes
1answer
374 views

Why does Q-learning converge to the optimal policy, even if the agent acts sub-optimally?

In Q-learning, during training, it doesn't matter how the agent selects actions. The algorithm always converges to the optimal policy. Why does this happen? What's the intuition?
0
votes
1answer
382 views

Is it possible to use a feed-forward neural network to predict the actions in reinforcement learning?

I have done a lot of research on the internet about Reinforcement Learning and I found encountered methods of Reinforcement Learning: Q-Learning and Deep Q-Learning. And I have developed a vague idea ...
2
votes
1answer
1k views

Why are Q values updated according to the greedy policy?

Apparently, in the Q-learning algorithm, the Q values are not updated according to the "current policy", but according to a "greedy policy". Why is that the case? I think this is related to the fact ...
3
votes
1answer
823 views

Deep Q-Learning poor convergence on Stochastic Environment

I'm trying to implement a Deep Q-network in Keras/TF that learns to play Minesweeper (our stochastic environment). I have noticed that the agent learns to play the game pretty well with both small and ...
3
votes
1answer
221 views

Reason for issues with correlation in the dataset in DQN

From the paper Human level Control through DeepRL, the correlation in the data causes instability in the network and may causes the network to ...
1
vote
1answer
402 views

Some RL algorithms (especially policy gradients) initialize with random policies, which often manifests as random jitter on spot for a long time?

I am reviewing a statement on the website for ES regarding structured exploration. https://blog.openai.com/evolution-strategies/ Structured exploration. Some RL algorithms (especially policy ...
6
votes
1answer
205 views

Q-Learning the generic maze solution

After doing some exercices on Q-learning for maze solving, I wondered : my q-learning algorithms solve only ONE maze. The AI doesn't learn how to solve mazes, so how can I achieve it ? For instance ...
3
votes
2answers
742 views

Difficulty in understanding identifiability in the “Dueling Network Architectures for Deep Reinforcement Learning” paper

I have difficulty understanding the following paragraph in the below excerpts from page 4 to page 5 from the paper Dueling Network Architectures for Deep Reinforcement Learning. The author said "we ...
1
vote
1answer
47 views

Can Q-learning be used to find the shortest distance from each source to destination?

Is it possible to form a table that will have simply the shortest distance from each source to destination using q learning? If not, suggest any other learning algorithm.
5
votes
1answer
506 views

Is the discount not needed in a deterministic environment for Reinforcement Learning?

I'm now reading a book titled as "Deep Reinforcement Learning Hands-On" and the author said the following on the chapter about AlphaGo Zero: Self-play In AlphaGo Zero, the NN is used to ...
5
votes
1answer
147 views

How is the fitted Q-iteration algorithm related to $Q^*(s, a)$, and how can we use function approximation with this algorithm?

I hope to get some clarifications on Fitted Q-Iteration (FQI). My Research So Far I've read Sutton's book (specifically, ch 6 to 10), Ernst et al and this paper. I know that $Q^*(s, a)$ expresses the ...
2
votes
1answer
115 views

Convergence in multi-agent environment

I have a multi-agent environment where agents are trying to optimise the overall energy consumption of their group. Agents can exchange energy between themselves (actions for exchange of energy ...
14
votes
1answer
3k views

Why does DQN require two different networks?

I was going through this implementation of DQN and I see that on line 124 and 125 two different Q networks have been initialized. From my understanding, I think one network predicts the appropriate ...
5
votes
1answer
2k views

How should I model all available actions of a chess game in deep Q-learning?

I just read about deep Q-learning, which is using a neural network for the value function instead of a table. I saw the example here: Using Keras and Deep Q-Network to Play FlappyBird and he used a ...
1
vote
1answer
84 views

Can Q-learning working in a multi agent environment where every agent learns a behaviour independently?

I am currently exploring multi-agent reinforcement learning. I have multiple agents that communicate with each other and a central service that maintains the environment state. The central service ...
1
vote
0answers
233 views

Training RL agent on timeseries trading data with Continous Deep Q or NAF

I am writing an MDP based agent that is supposed to learn to place bids and asks in a trading environment. The system requests 2 values (mWh energy and $, both being positive or negative). Every ...
2
votes
1answer
177 views

Should the exploration rate be reset after each trial in Q-learning?

As the title says, should I reset the exploration rate between trials? I am currently doing the Open AI pendulum task and after a number of trials my model started playing but did not take any ...
2
votes
2answers
62 views

Should we feed a greater fraction of terminal states to the value network so that their values are learned first?

The basis of Q-learning is recursive (similar to dynamic programming), where only the absolute value of the terminal state is known. Shouldn't it make sense to feed the model a greater proportion of ...
4
votes
0answers
2k views

Training AI to play NES/SNES games on NN python [closed]

I am currently getting into Deep Learning and would like to set up an environment for training an Artificial Neural Network or NEAT to play simple video games on NES (Mario etc.) and SNES ( Donkey ...
32
votes
2answers
13k views

What is the relation between Q-learning and policy gradients methods?

As far as I understand, Q-learning and policy gradients (PG) are the two major approaches used to solve RL problems. While Q-learning aims to predict the reward of a certain action taken in a certain ...
2
votes
0answers
291 views

Snake path finding variant : Algorithm choice

I am working on a project which maps to a variant of path finding problem. I am new to this area and I would be very grateful if you could give suggestions/ point to libraries for relevant algorithms. ...

1
3 4 5
6
7