Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [q-learning]

Use for questions that involve Q-learning, where Q is the value of a particular next action among a set of possible actions, based on a specified function of each action and its projected result.

2
votes
1answer
31 views

Why does Deep Q Network outputs multiple Q values?

I am learning Deep RL following this tutorial: https://medium.freecodecamp.org/an-introduction-to-deep-q-learning-lets-play-doom-54d02d8017d8 I understand everything but one detail: This image shows ...
1
vote
1answer
30 views

Robot Arm Deep Q Learning Actions

Hello I am new to reinforcement learning and robotics. So far I have an understanding of the concept on 2D world. You can make agent move one step in one direction. However, how do you define movement ...
1
vote
2answers
81 views

Is there more than one Q-matrix update formula?

I asked a question a while ago here and since then I've been solving the issues within my code but I have just one question... This is the formula for updating the Q-Matrix in Q-Learning: $$Q(s_t, ...
0
votes
0answers
17 views

Is an indirect policy superior to a normal one?

Before a robot can act with meaning, some planning is needed. The idea is, that the decision making process is independent from action. The task of figuring out what the best decision is, was ...
0
votes
0answers
37 views

Is it correct to define the equation below as q-learning multiple agent reinforcement learning models

I'm trying to define the q-learning multiple agent reinforcement learning models but I'm not sure if it is correct or not. N the number of agents and $i \in N$ and the k is the time episode. $Q^{i}_{...
2
votes
1answer
59 views

What is the next state for a two-player board game?

I'm using Q-learning to train an agent to play a board game (e.g. chess, draughts or go). The agent takes an action while in state $S$, but then what is the next state (that is, $S'$)? Is $S'$ now ...
1
vote
1answer
52 views

How do updates in SARSA and Q-learning differ in code?

The update rules for Q-learning and SARSA each are as follows: Q Learning: $$Q(s_t,a_t)←Q(s_t,a_t)+α[r_{t+1}+γ\max_{a'}Q(s_{t+1},a')−Q(s_t,a_t)]$$ SARSA: $$Q(s_t,a_t)←Q(s_t,a_t)+α[r_{t+1}+γQ(s_{t+...
1
vote
1answer
70 views

How do I convert table-based to neural network-based Q-learning?

I've used a table to represent the Q function, while an agent is being trained to catch the cheese without touching the walls. The first and last row (and column) of the matrix are associated with ...
3
votes
1answer
44 views

Can Q-learning be used to derive a stochastic policy?

In my understanding, Q-learning gives you a deterministic policy. However, can we use some technique to build a meaningful stochastic policy from the learned Q values? I think that simply using a ...
0
votes
0answers
33 views

Updating Q-Matrix in Q-Learning

I'm new to Reinforcement Learning and I'm having an issue trying to code a Q-Learning algorithm in Python. I don't understand why when I use this as my Q matrix update function (alp is my learning ...
2
votes
1answer
56 views

Will Q-learning converge to the optimal state-action function when the reward periodically changes?

Imagine that the agent receives a positive reward upon reaching a state 𝑠. Once the state 𝑠 has been reached the positive reward associated with it vanishes and appears somewhere else in the state ...
1
vote
1answer
44 views

Meaning of Actor Output in Actor Critic Reinforcement Learning

In actor critic, The equations for calculating the loss in actor critic are an actor loss (parameterized by $\theta$) $$log[\pi_\theta(s_t,a_t)]Q_w(s_t,a_t)$$ and a critic loss (parameterized by ...
1
vote
2answers
42 views

When are Q values calculated in experince replay?

In experience replay, the update rule follows the loss: $$ L_i(\theta_i) = \mathbb{E}_{(s_t, a_t, r_t, s_{t+1}) \sim U(D)} \left[ \left(r_t + \gamma \max_{a_{t+1}} Q(s_{t+1}, a_{t+1}; \theta_i^-) - ...
2
votes
1answer
43 views

How to define the final / terminal state for Q learning?

I'm training an agent using RL and the SARSA function to update a Q function, but I'm confused how you handle the final state. In this case when the game ends and there is no S'. For example, the ...
-1
votes
1answer
47 views

What will Q-values look like in self-play tic-tac-toe?

This corresponds to Exercise 1.1 of RLBook, and a discussion followed from here. Considering two reward schemes- Win = +1, Draw = 0, Loss = -1 Win = +1, Draw or Loss = 0 Can we say something about ...
1
vote
1answer
66 views

Q-Learning tic tac toe - bad player

I tried to build an Q-learning agent which you can play tic tac toe against after training. Unfortunately the agent performs pretty poorly. He tries to win but does not try to make me 'not winning' ...
2
votes
1answer
148 views

Using the opponent's mixed strategy in estimating the state value in minimax Q learning

In the paper Markov games as a framework for multi-agent reinforcement learning (which introduces the minimax Q Learning algorithm), at the bottom left of page 3, my understanding is that the author ...
1
vote
1answer
41 views

How do I calculate $max_{a′}Q(s′,a′,w−)$ when it is represented as a neural network?

Consider the following loss function $$ L(\mathbf{w}) = [(r + \gamma max_{a'} Q(s', a', \mathbf{w^-})) - Q(s, a, \mathbf{w})]^2 $$ where $Q(s, a, \mathbf{w^-})$ and $Q(s, a, \mathbf{w})$ are ...
2
votes
2answers
99 views

Concrete Example for Q Learning

I am not sure if I understood the q learning algorithms correctly. Therefore I would give a concrete example and ask if someone can tell me how to update the q value correctly. First I initialized ...
0
votes
1answer
41 views

How do I update the Q values of a Deep Q Network when exploring?

I am trying to implement a Deep Q Network to play Asteroids. Unfortunately, I am not sure how to calculate the Q value exactly, if I am exploring. For example, the agent is exploring for 1 second (...
-1
votes
1answer
33 views

Q-Learning Algorithmus does not work

Hey I am training an initialized Neural Network with this Method ...
3
votes
1answer
86 views

How do we prove the n-step return error reduction property?

In section 7.1 (about the n-step bootstrapping) of the book Reinforcement Learning: An Introduction (2nd edition), by Andrew Barto and Richard S. Sutton, the authors write about what they call the "n-...
1
vote
0answers
13 views

Exploration rate decay and training in Q learning

I'm trying to replicate the results of the DeepMind's paper with Breakout included in OpenAI Gym. I wonder how much frames should I keep until I reach the fixed exploration rate. Actually it reaches ...
-1
votes
1answer
116 views

DQN it's not working properly

I'm trying to build a DQN to replicate the DeepMind results. I'm doing with a simple DQN for the moment, but it isn't learning properly: after +5000 episodes, it couldn't get more than 9-10 points. ...
2
votes
1answer
134 views

How to build AI bots for board games like monopoly?

I am trying to build a Q learning-based bot for board games, specifically monopoly. I am fairly new to Q-learning and currently, I have only implemented some bots that can play simple games like Tic-...
1
vote
0answers
41 views

Why does Q-learning converges to optimal policy even if I am acting suboptimally?

In Q-learning, during training, it doesn’t matter how I select actions. The algorithm always converges to optimal optimal policy. Why does this happen?
0
votes
1answer
69 views

Is it possible to use a feed-forward neural network to predict the actions in reinforcement learning?

I have done a lot of research on the internet about Reinforcement Learning and I found encountered methods of Reinforcement Learning: Q-Learning and Deep Q-Learning. And I have developed a vague idea ...
1
vote
1answer
75 views

Why are Q values updated according to the greedy policy?

Apparently, in the Q-learning algorithm, the Q values are not updated according to the "current policy", but according to a "greedy policy". Why is that the case? I think this is related to the fact ...
1
vote
1answer
82 views

Deep Q-Learning poor convergence on Stochastic Environment

I'm trying to implement a Deep Q-network in Keras/TF that learns to play Minesweeper (our stochastic environment). I have noticed that the agent learns to play the game pretty well with both small and ...
1
vote
0answers
64 views

How does Hindsight Experience Replay learn from unsuccessful trajectories

I am confused by how HER learns from unsuccessful trajectories. I understand that from failed trajectories it creates 'fake' goals that it can learn from. Ignoring HER for now, if in the case where ...
3
votes
1answer
34 views

Reason for issues with correlation in the dataset in DQN

From the paper Human level Control through DeepRL, the correlation in the data causes instability in the network and may causes the network to ...
1
vote
1answer
53 views

Some RL algorithms (especially policy gradients) initialize with random policies, which often manifests as random jitter on spot for a long time?

I am reviewing a statement on the website for ES regarding structured exploration. https://blog.openai.com/evolution-strategies/ Structured exploration. Some RL algorithms (especially policy ...
6
votes
1answer
70 views

Q-Learning the generic maze solution

After doing some exercices on Q-learning for maze solving, I wondered : my q-learning algorithms solve only ONE maze. The AI doesn't learn how to solve mazes, so how can I achieve it ? For instance ...
1
vote
3answers
252 views

Difficulty in understanding identifiability in the “Dueling Network Architectures for Deep Reinforcement Learning” paper

I have difficulty understanding the following paragraph in bracketed in red parentheses in the below excerpts from page 4 to page 5 from the paper Dueling Network Architectures for Deep Reinforcement ...
1
vote
1answer
41 views

Can Q-learning be used to find the shortest distance from each source to destination?

Is it possible to form a table that will have simply the shortest distance from each source to destination using q learning? If not, suggest any other learning algorithm.
5
votes
1answer
162 views

Is the discount not needed in a deterministic environment for Reinforcement Learning?

I'm now reading a book titled as "Deep Reinforcement Learning Hands-On" and the author said the following on the chapter about AlphaGo Zero: Self-play In AlphaGo Zero, the NN is used to ...
4
votes
1answer
94 views

Reinforcement Learning (Fitted Q): Qn on Concept & Implementation

I hope to get some clarifications on Fitted Q-Learning ('FQL'). My Research So Far I've read Sutton's book (specifically, chp 6 to 10), Ernst et al and this paper. I know that ...
2
votes
1answer
52 views

Convergence in multi-agent environment

I have a multi-agent environment where agents are trying to optimise the overall energy consumption of their group. Agents can exchange energy between themselves (actions for exchange of energy ...
9
votes
1answer
821 views

Why does DQN require two different networks?

I was going through this implementation of DQN and I see that on line 124 and 125 two different Q networks have been initialized. From my understanding, I think one network predicts the appropriate ...
3
votes
1answer
130 views

Number of Neuron in Q-Learning of Chess

So I just read about deep Q-Learning which is using a neural network for optimization instead of Q-table. I saw the example here: https://yanpanlau.github.io/2016/07/10/FlappyBird-Keras.html and he ...
1
vote
0answers
48 views

Deep Q-Network concepts and implementation

How does sequential DQN work? How would one construct the simple sequential DQN? OpenAI Baselines: DQN
1
vote
1answer
47 views

Can Q-learning working in a multi agent environment where every agent learns a behaviour independently?

I am currently exploring multi-agent reinforcement learning. I have multiple agents that communicate with each other and a central service that maintains the environment state. The central service ...
2
votes
1answer
136 views

Action Probability with Thompson Sampling in Deep Reinforcement Learning

In some implementations of off-policy Q learning we need to know the action probabilities given by the behavior policy mu(a) (e.g., if we want to use importance ...
1
vote
0answers
80 views

Training RL agent on timeseries trading data with Continous Deep Q or NAF

I am writing an MDP based agent that is supposed to learn to place bids and asks in a trading environment. The system requests 2 values (mWh energy and $, both being positive or negative). Every ...
2
votes
1answer
88 views

Should the exploration rate be reset after each trial in Q-learning?

As the title says, should I reset the exploration rate between trials? I am currently doing the Open AI pendulum task and after a number of trials my model started playing but did not take any ...
3
votes
1answer
847 views

Training AI to play NES/SNES games on NN python

I am currently getting into Deep Learning and would like to set up an environment for training an Artificial Neural Network or NEAT to play simple video games on NES (Mario etc.) and SNES ( Donkey ...
11
votes
1answer
2k views

What is the relation between Q-learning and policy gradients methods?

As far as I understand, Q-learning and policy gradients (PG) are the two major approaches used to solve RL problems. While Q-learning aims to predict the reward of a certain action taken in a certain ...
3
votes
1answer
175 views

Snake path finding variant : Algorithm choice

I am working on a project which maps to a variant of path finding problem. I am new to this area and I would be very grateful if you could give suggestions/ point to libraries for relevant algorithms. ...
3
votes
1answer
211 views

What are good learning strategies for Deep Q-Network with opponents?

I am trying to find out what are some good learning strategies for Deep Q-Network with opponents. Let's consider the well known game Tic-Tac-Toe as an example: How should an opponent be implemented ...
3
votes
1answer
318 views

How does Q-learning work when the action does not determine the next state?

I am currently building my first AI in a stochastic environment and the following question came to my mind. The Q function uses the (current and future) states to determine the action that gets the ...