The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Questions tagged [q-learning]

For questions related to the Q-learning algorithm, which is a model-free and temporal-difference reinforcement learning algorithm that attempts to approximate the Q function, which is a function that, given a state s and an action a, returns a real number that represents the return (or value) of state s when action a is taken from s. Q-learning was introduced in the PhD thesis "Learning from Delayed Rewards" (1989) by Watkins.

Filter by
Sorted by
Tagged with
1
vote
0answers
52 views

Do we need to reset the DQN network after every episode?

I was going through this implementation of Reinforcement learning where model is being trained to manage the number of bikes at a station. Here, line 78 represents the loop over all episodes (if I ...
1
vote
0answers
54 views

Reinforcement Learning with limited number of episodes

I try to implement RL to a case something like this: This game consist of several rounds. Every round the players need to generate a maze that consists of rooms. There are around 1000 different ...
3
votes
1answer
45 views

Why Q2 is a more or less independant estimate in Twin Delayed DDPG (TD3)?

TD3 is inspired from both double Q-learning and double DQN. In double Q-learning, I understand that Q1 and Q2 are independent because they are trained on different samples. In double DQN, I understand ...
2
votes
1answer
84 views

What are the differences between the DQN variants?

There are several variants of the DQN model. For example, double DQN, duelling DQN, prioritized DQN, distributed prioritized DQN, episodic memory DQN, asynchronous n-step DQN and multiple DQN. What ...
2
votes
0answers
27 views

Hindsight Experience Replay with multiple goals

What if there are multiple goals? For example, let's consider Bit-flipping environment as described in the paper HER with one small change: Now, goal is not some specific configuration, but let's say ...
3
votes
2answers
55 views

Why is the max a non-expansive operator?

In certain reinforcement learning (RL) proofs, the operators involved are assumed to be non-expansive. For example, on page 6 of the paper Generalized Markov Decision Processes: Dynamic-programming ...
0
votes
0answers
17 views

How to serve a deep q network using tensorflow serving?

How to Serve a Deep Q Network using Tensorflow Serving. I have built a Deep Q Network using Multilayer Perceptron. Is it possible to serve it?
1
vote
0answers
30 views

Comparison and understanding of different version of DDQN?

There are several version of DDQN floating around. Sutton gives one that is a simple symmetric random update of the two Q functions. I think other papers (Silver paper for example) use a kind of ...
3
votes
1answer
45 views

How to apply or extend the $Q(\lambda)$ algorithm to semi-MDPs?

I want to model an SMDP such that time is discretized and the transition time between the two states follows an exponential distribution and there would be no reward between the transition. Can I ...
2
votes
1answer
43 views

Is there any grid world dataset or generator for reinforcement learning?

I would like to start programming a multi task reinforcement learning model. For this, I need not just one maze or grid world (or just model-based), but many with different reward functions. So, I am ...
2
votes
1answer
112 views

Is there a way to train an RL agent without any environment?

Following Deep Q-learning from Demonstrations, I'd like to avoid potentially unsafe behavior during early learning by making use of supervised learning with demonstration data. However, the ...
3
votes
2answers
280 views

My DQN is stuck and can't see where the problem is

I'm trying to replicate the DeepMind paper results, so I implemented my own DQN. I left it training for more than 4 million frames (more than 2000 episodes) on SpaceInvaders-v4 (OpenAI-Gym) and it ...
1
vote
1answer
68 views

What is happening when a reinforcement learning agent trains itself out of desired behavior?

I have a reinforcement learning agent with both a positive and a negative terminal state. After each episode during training, I am recording whether a success or failure occurred, and then I can ...
2
votes
1answer
118 views

Why does Deep Q Network outputs multiple Q values?

I am learning Deep RL following this tutorial: https://medium.freecodecamp.org/an-introduction-to-deep-q-learning-lets-play-doom-54d02d8017d8 I understand everything but one detail: This image shows ...
1
vote
1answer
49 views

Robot Arm Deep Q Learning Actions

Hello I am new to reinforcement learning and robotics. So far I have an understanding of the concept on 2D world. You can make agent move one step in one direction. However, how do you define movement ...
1
vote
2answers
102 views

Is there more than one Q-matrix update formula?

I asked a question a while ago here and since then I've been solving the issues within my code but I have just one question... This is the formula for updating the Q-Matrix in Q-Learning: $$Q(s_t, ...
0
votes
0answers
21 views

Is an indirect policy superior to a normal one?

Before a robot can act with meaning, some planning is needed. The idea is, that the decision making process is independent from action. The task of figuring out what the best decision is, was ...
2
votes
1answer
72 views

What is the next state for a two-player board game?

I'm using Q-learning to train an agent to play a board game (e.g. chess, draughts or go). The agent takes an action while in state $S$, but then what is the next state (that is, $S'$)? Is $S'$ now ...
2
votes
1answer
108 views

How do updates in SARSA and Q-learning differ in code?

The update rules for Q-learning and SARSA each are as follows: Q Learning: $$Q(s_t,a_t)←Q(s_t,a_t)+α[r_{t+1}+γ\max_{a'}Q(s_{t+1},a')−Q(s_t,a_t)]$$ SARSA: $$Q(s_t,a_t)←Q(s_t,a_t)+α[r_{t+1}+γQ(s_{t+...
1
vote
1answer
147 views

How do I convert table-based to neural network-based Q-learning?

I've used a table to represent the Q function, while an agent is being trained to catch the cheese without touching the walls. The first and last row (and column) of the matrix are associated with ...
3
votes
1answer
108 views

Can Q-learning be used to derive a stochastic policy?

In my understanding, Q-learning gives you a deterministic policy. However, can we use some technique to build a meaningful stochastic policy from the learned Q values? I think that simply using a ...
2
votes
1answer
65 views

Will Q-learning converge to the optimal state-action function when the reward periodically changes?

Imagine that the agent receives a positive reward upon reaching a state 𝑠. Once the state 𝑠 has been reached the positive reward associated with it vanishes and appears somewhere else in the state ...
2
votes
1answer
174 views

Meaning of Actor Output in Actor Critic Reinforcement Learning

In actor critic, The equations for calculating the loss in actor critic are an actor loss (parameterized by $\theta$) $$log[\pi_\theta(s_t,a_t)]Q_w(s_t,a_t)$$ and a critic loss (parameterized by ...
1
vote
2answers
57 views

When are Q values calculated in experince replay?

In experience replay, the update rule follows the loss: $$ L_i(\theta_i) = \mathbb{E}_{(s_t, a_t, r_t, s_{t+1}) \sim U(D)} \left[ \left(r_t + \gamma \max_{a_{t+1}} Q(s_{t+1}, a_{t+1}; \theta_i^-) - ...
2
votes
1answer
301 views

How to define the final / terminal state for Q learning?

I'm training an agent using RL and the SARSA function to update a Q function, but I'm confused how you handle the final state. In this case when the game ends and there is no S'. For example, the ...
0
votes
1answer
79 views

What will Q-values look like in self-play tic-tac-toe?

This corresponds to Exercise 1.1 of RLBook, and a discussion followed from here. Considering two reward schemes- Win = +1, Draw = 0, Loss = -1 Win = +1, Draw or Loss = 0 Can we say something about ...
1
vote
1answer
234 views

Q-Learning tic tac toe - bad player

I tried to build an Q-learning agent which you can play tic tac toe against after training. Unfortunately the agent performs pretty poorly. He tries to win but does not try to make me 'not winning' ...
2
votes
1answer
196 views

Using the opponent's mixed strategy in estimating the state value in minimax Q learning

In the paper Markov games as a framework for multi-agent reinforcement learning (which introduces the minimax Q Learning algorithm), at the bottom left of page 3, my understanding is that the author ...
1
vote
1answer
49 views

How do I calculate $max_{a′}Q(s′,a′,w−)$ when it is represented as a neural network?

Consider the following loss function $$ L(\mathbf{w}) = [(r + \gamma max_{a'} Q(s', a', \mathbf{w^-})) - Q(s, a, \mathbf{w})]^2 $$ where $Q(s, a, \mathbf{w^-})$ and $Q(s, a, \mathbf{w})$ are ...
2
votes
2answers
149 views

Concrete Example for Q Learning

I am not sure if I understood the q learning algorithms correctly. Therefore I would give a concrete example and ask if someone can tell me how to update the q value correctly. First I initialized ...
0
votes
1answer
80 views

How do I update the Q values of a Deep Q Network when exploring?

I am trying to implement a Deep Q Network to play Asteroids. Unfortunately, I am not sure how to calculate the Q value exactly, if I am exploring. For example, the agent is exploring for 1 second (...
-1
votes
1answer
42 views

Q-Learning Algorithmus does not work

Hey I am training an initialized Neural Network with this Method ...
6
votes
1answer
339 views

How do we prove the n-step return error reduction property?

In section 7.1 (about the n-step bootstrapping) of the book Reinforcement Learning: An Introduction (2nd edition), by Andrew Barto and Richard S. Sutton, the authors write about what they call the "n-...
1
vote
0answers
38 views

Exploration rate decay and training in Q learning

I'm trying to replicate the results of the DeepMind's paper with Breakout included in OpenAI Gym. I wonder how much frames should I keep until I reach the fixed exploration rate. Actually it reaches ...
-1
votes
1answer
341 views

DQN it's not working properly

I'm trying to build a DQN to replicate the DeepMind results. I'm doing with a simple DQN for the moment, but it isn't learning properly: after +5000 episodes, it couldn't get more than 9-10 points. ...
2
votes
1answer
365 views

How to build AI bots for board games like monopoly?

I am trying to build a Q learning-based bot for board games, specifically monopoly. I am fairly new to Q-learning and currently, I have only implemented some bots that can play simple games like Tic-...
1
vote
0answers
70 views

Why does Q-learning converges to optimal policy even if I am acting suboptimally?

In Q-learning, during training, it doesn’t matter how I select actions. The algorithm always converges to optimal optimal policy. Why does this happen?
0
votes
1answer
142 views

Is it possible to use a feed-forward neural network to predict the actions in reinforcement learning?

I have done a lot of research on the internet about Reinforcement Learning and I found encountered methods of Reinforcement Learning: Q-Learning and Deep Q-Learning. And I have developed a vague idea ...
2
votes
1answer
267 views

Why are Q values updated according to the greedy policy?

Apparently, in the Q-learning algorithm, the Q values are not updated according to the "current policy", but according to a "greedy policy". Why is that the case? I think this is related to the fact ...
1
vote
1answer
294 views

Deep Q-Learning poor convergence on Stochastic Environment

I'm trying to implement a Deep Q-network in Keras/TF that learns to play Minesweeper (our stochastic environment). I have noticed that the agent learns to play the game pretty well with both small and ...
1
vote
1answer
101 views

How does Hindsight Experience Replay learn from unsuccessful trajectories

I am confused by how HER learns from unsuccessful trajectories. I understand that from failed trajectories it creates 'fake' goals that it can learn from. Ignoring HER for now, if in the case where ...
3
votes
1answer
80 views

Reason for issues with correlation in the dataset in DQN

From the paper Human level Control through DeepRL, the correlation in the data causes instability in the network and may causes the network to ...
1
vote
1answer
94 views

Some RL algorithms (especially policy gradients) initialize with random policies, which often manifests as random jitter on spot for a long time?

I am reviewing a statement on the website for ES regarding structured exploration. https://blog.openai.com/evolution-strategies/ Structured exploration. Some RL algorithms (especially policy ...
6
votes
1answer
93 views

Q-Learning the generic maze solution

After doing some exercices on Q-learning for maze solving, I wondered : my q-learning algorithms solve only ONE maze. The AI doesn't learn how to solve mazes, so how can I achieve it ? For instance ...
3
votes
3answers
393 views

Difficulty in understanding identifiability in the “Dueling Network Architectures for Deep Reinforcement Learning” paper

I have difficulty understanding the following paragraph in the below excerpts from page 4 to page 5 from the paper Dueling Network Architectures for Deep Reinforcement Learning. The author said "we ...
1
vote
1answer
42 views

Can Q-learning be used to find the shortest distance from each source to destination?

Is it possible to form a table that will have simply the shortest distance from each source to destination using q learning? If not, suggest any other learning algorithm.
5
votes
1answer
274 views

Is the discount not needed in a deterministic environment for Reinforcement Learning?

I'm now reading a book titled as "Deep Reinforcement Learning Hands-On" and the author said the following on the chapter about AlphaGo Zero: Self-play In AlphaGo Zero, the NN is used to ...
5
votes
1answer
111 views

Reinforcement Learning (Fitted Q): Qn on Concept & Implementation

I hope to get some clarifications on Fitted Q-Learning ('FQL'). My Research So Far I've read Sutton's book (specifically, chp 6 to 10), Ernst et al and this paper. I know that ...
2
votes
1answer
65 views

Convergence in multi-agent environment

I have a multi-agent environment where agents are trying to optimise the overall energy consumption of their group. Agents can exchange energy between themselves (actions for exchange of energy ...
11
votes
1answer
1k views

Why does DQN require two different networks?

I was going through this implementation of DQN and I see that on line 124 and 125 two different Q networks have been initialized. From my understanding, I think one network predicts the appropriate ...