Questions tagged [q-learning]

For questions related to the Q-learning algorithm, which is a model-free and temporal-difference reinforcement learning algorithm that attempts to approximate the Q function, which is a function that, given a state s and an action a, returns a real number that represents the return (or value) of state s when action a is taken from s. Q-learning was introduced in the PhD thesis "Learning from Delayed Rewards" (1989) by Watkins.

Filter by
Sorted by
Tagged with
1
vote
1answer
80 views

How can I fetch ​exploration decay rate of an iterable Q-table in Python?

I have done creating the virtual environment, creating the Q-table, initializing the q-parameters, then I made a training module and stored it in a numpy array. ...
0
votes
1answer
81 views

How can I update my Q-table in Python?

I want to implement this function on a voice searching application: $$ Q(S, A) \leftarrow Q(S, A)+\alpha\left(R+\gamma Q\left(S^{\prime}, A^{\prime}\right)-Q(S, A)\right) $$ And also restricted to use ...
2
votes
0answers
70 views

Most of state-action pairs remain unvisited in the q-table

In building my first Q-learning algorithm for OpenAI gym's CartPole problem, many of my states remain unvisited. I believe it is the reason that my agent does not learn. Can I be told of the reasons I ...
1
vote
0answers
86 views

OpenAI gym's CartPole problem system does not learn

My OpenAI CartPole-v0 problem's implementation using basic Q-learning does not learn at all. I am a beginner and have implemented my first ever Q-learning from scratch after learning from tutorials. ...
4
votes
1answer
251 views

Why do DQNs tend to forget?

Why do DQNs tend to forget? Is it because when you feed highly correlated samples, your model (function approximation) doesn't give a general solution? For example: I use level 1 experiences, my ...
2
votes
1answer
77 views

Reinforcement learning with action consisting of two discrete values

I'm new to reinforcement learning. I have a problem where an action is composed of an order (rod with a required length) and an item from a warehouse (an existing rod with a certain length, which will ...
6
votes
0answers
170 views

Is the Bellman equation that uses sampling weighted by the Q values (instead of max) a contraction?

It is proved that the Bellman update is a contraction (1). Here is the Bellman update that is used for Q-Learning: $$Q_{t+1}(s, a) = Q_{t}(s, a) + \alpha*(r(s, a, s') + \gamma \max_{a^*} (Q_{t}(s', ...
0
votes
1answer
335 views

How can I change observation states' values in OpenAI gym's cartpole environment?

I am learning with the OpenAI gym's cart pole environment. I want to make the observation states discrete (with small stepsize) and for that purpose, I need to change two of the observations from [$ -\...
4
votes
3answers
497 views

Upper limit to the maximum cumulative reward in a deep reinforcement learning problem

Is there an upper limit to the maximum cumulative reward in a deep reinforcement learning problem? For example you want to train an DQN agent in an environment and you want to know what is the highest ...
2
votes
0answers
52 views

Prioritised Remembering in Experience Replay (Q-Learning)

I'm using Experience Replay based on the original Prioritized Experience Replay (PER) paper. In the paper authors show ~ an order of magnitude increase in data efficiency from prioritized sampling. ...
2
votes
1answer
511 views

Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards?

Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards? Would it not make more sense to compute $\mathbb{E}(R \mid s, a)$ (the expected return for taking ...
1
vote
1answer
151 views

What is the purpose of a Neural Network in Reinforcement Learning when we have a Q-learning update rule?

I'm confused as to the purpose of training a neural network (NN) for reinforcement learning (RL) tasks such as Gridworld. In RL tasks, namely q-learning, we have a q-learning update rule, which is ...
1
vote
1answer
869 views

Why do we need target network in deep Q learning? [duplicate]

I already know deep RL, but to learn it deeply I want to know why do we need 2 networks in deep RL. What does the target network do? I now there is huge mathematics into this, but I want to know deep ...
2
votes
1answer
47 views

What is convergence analysis, and why is it needed in reinforcement learning?

While reading a paper about Q-learning in network energy consumption, I came across the section on convergence analysis. Does anyone know what convergence analysis is, and why is convergence analysis ...
2
votes
0answers
44 views

How can I formulate a prediction problem (given labeled data) as an RL problem and solve it with Q-learning?

One of my friends sent me a problem he was working on lately, and I couldn't help but I wonder how could it be solved using Q-learning. The statement is as follows: Given the following datasets, the ...
0
votes
1answer
111 views

How does training for DQN work if messing up in the environment in costly?

Suppose that we want to train a car to drive in the real world and decide to use Reinforcement Learning (specifically, DQN) for that. I am a bit confused about how training generally works. Is it that ...
1
vote
0answers
79 views

Tic-tac-toe: How would standard SARSA and Q-learning yield different results in the agent's behaviour?

I know this is deceptively simple. Tic tac toe is a well studied game for RL. Assume your agent is playing aggainst a strong opponent. I know you deal in after states. I know that in Q learning the ...
2
votes
1answer
126 views

Implementing SARSA for a 2-stage Markov Decision Process

I am a bit confused as to how exactly I should be implementing SARSA (or Q-learning too) on what is a simple 2-stage Markov Decision Task. The structure of the task is as follows: Basically, there ...
3
votes
1answer
109 views

q learning appears to converge but does not always win against random tic tac toe player

q learning is defined as: Here is my implementation of q learning of the tic tac toe problem: ...
2
votes
1answer
55 views

Why isn't it wise for us to completely erase our old Q value and replace it with the calculated Q value?

Why isn't it wise for us to completely erase our old Q value and replace it with the calculated Q value? Why can't we forget the learning rate and temporal difference? Here's the update formula.
3
votes
2answers
111 views

Why is it not advisable to have a 100 percent exploration rate? [duplicate]

During the learning phase, why don't we have a 100% exploration rate, to allow our agent to fully explore our environment and update the Q values, then during testing we bring in exploitation? Does ...
1
vote
1answer
62 views

Why do we update the weights of the target network in deep Q learning?

I know we keep the target network constant during training to improve stability, but why exactly are we updating the weights of our target network? In particular, if we've already reached convergence, ...
1
vote
2answers
58 views

Why do we explore after we have an accurate estimate of the value function?

Suppose we have a small space state and that, after about 2000 episodes, we've accurately explored the environment and known the accurate $Q$ values. In that case, why do we still leave a small ...
1
vote
1answer
77 views

What would happen if we sampled only one tuple from the experience replay?

The concept of experience replay is saving our experiences in our replay buffer. We select at random to break the correlation between consecutive samples, right? What would happen if we calculate our ...
1
vote
1answer
565 views

What's the best practice for Boltzmann Exploration temperature in RL?

I'm currently modeling DQN in Reinforcement Learning. My question is: what are the best practices related to Boltzmann Exploration? My current thoughts are: (1) Let the temperature decay through ...
0
votes
1answer
212 views

How to validate that my DQN hyperparameters are the optimal?

My DQN model outputs the best traffic light state in an intersection. I used different values of batch size and learning rate to find the best model. How would I know if I got the optimal ...
1
vote
0answers
39 views

Is there a way to show convergence of DQN other than by eye observation?

I made a DQN model and plot its reward curve. You can see intuitively that the curve already converged since its reward value now just oscillates. How can I show confidence that my DQN already reached ...
2
votes
1answer
116 views

Updating action-value functions in Semi-Markov Decision Process and Reinforcement Learning

Suppose that the transition time between two states is a random variable (for example, unknown exponential distribution); and between two arrivals, there is no reward. If $\tau$ (real number not an ...
3
votes
1answer
87 views

How to know if my DQN is optimized?

I made a DQN that controls a traffic light. The observation states are the number of vehicles of each lane in the intersection. I trained it for 500 episodes and saved the model every 50th episode. I ...
1
vote
2answers
81 views

Why can't we fully exploit the environment after the first episode in Q-learning?

During the first episode, it's 100% exploration, because all our Q values are 0. Suppose we have 1000 time steps, and it's terminated by meeting a reward. So, after the first episode, why can't we ...
0
votes
2answers
93 views

How can I model this problem of delivering assets by choosing a route with reinforcement learning?

I would like to build a model based on reinforcement learning (RL) for the following scenario Recommend the best route (of cities listed for a given country) that satisfies the required criteria (...
3
votes
1answer
493 views

Proof of Maximization Bias in Q-learning?

In the textbook "Reinforcement Learning: An Introduction" by Richard Sutton and Andrew Barto, the concept of Maximization Bias is introduced in section 6.7, and how Q-learning "over-estimates" action-...
1
vote
1answer
197 views

What are the differences between SARSA and Q-learning?

From Sutton and Barto's book Reinforcement Learning (Adaptive Computation and Machine Learning series), are the following definitions: To aid my learning of RL and gain an intuition, I'm focusing on ...
1
vote
2answers
73 views

Generalising performance of Q-learning agent through self-play in a two-player game (MCTS?)

I'm using Q-learning (off-policy TD-control as specified in Sutton's book on pg 131) to train an agent to play connect four. My goal is to create a strong player (superhuman performance?) purely by ...
0
votes
1answer
86 views

How do I know that the DQN has learnt an appropriate Q function?

Is there any sanity check to know whether the Q functions learnt are appropriate in deep Q networks? I know that the Q values for end states should approximate the terminal reward. However, is it ...
1
vote
1answer
124 views

Why do my rewards fall using tabular Q-learning as I perform more episodes?

Using the tutorial from: SentDex - Python Programming I added Q Learning to my script that was previously just picking random actions. His script uses the MountainCar Environment so I had to amend it ...
3
votes
1answer
266 views

Why we don't use importance sampling in tabular Q-Learning?

Why don't we use an importance sampling ratio in Q-Learning, even though Q-Learning is an off-policy method? Importance sampling is used to calculate expectation of a random variable by using data ...
0
votes
2answers
148 views

When we use a neural network to approximate the Q values, is the Q target a single value?

I have two questions When we use our network to approximate our Q values, is the Q target a single value? During backpropagation, when the weights are updated, does it automatically update the Q ...
0
votes
0answers
93 views

My Double DQN with Experience Replay produces a no-action decision most of the time. Why?

I've written a Double DQN-based stock trading bot using mainly time series stock data. The internal network of the Double DQN is a LSTM which handles the time series data. An Experience Replay buffer ...
2
votes
0answers
115 views

If the minimum Q value is decreasing and the maximum Q value increasing, is this a sign that dueling double DQN is diverging?

I'm training a dueling double DQN agent with prioritized replay buffer and notice that the min Q values are decreasing, while the max Q values are increasing. Is this a sign that it is diverging? ...
1
vote
1answer
153 views

How to implement RAM versions of Atari games

I have coded the breakout RAM version, but, unfortunately, its highest reward was 5. I trained it for about 2 hours and never reached a higher score. The code is huge, so I can't paste here, but, in ...
3
votes
0answers
83 views

How to take actions at each episode and within each step of the episode in deep Q learning?

In deep Q learning, we execute the algorithm for each episode, and for each step within an episode, we take an action and record a reward. I have a situation where my action is 2-tuple $a=(a_1,a_2)$. ...
2
votes
1answer
56 views

How to predict Q-values based on stack of frames

I decided to train deep Q-learning agent based on getting raw pixels from environment.I have one particular problem:when I input stack of frames, suppose 4 consecutive frames, if action space is 6,...
2
votes
0answers
43 views

In a DQN, can Prioritized Experience Replay actually perform worse than a regular Experience Replay?

I've written a Double DQN-based stock trading bot using mainly time series stock data. I've recently upgraded my Experience Replay(ER) code with a version of Prioritized Experience Replay (PER) ...
0
votes
2answers
241 views

Help with deep Q learning for 2048 game getting stuck

I am having trouble making a reinforcement algorithm than can win the 2048 game. I have tried with deep Q (which I think is the simplest algorithm that should be able to learn a winning strategy). ...
2
votes
0answers
79 views

How can the target rely on untrained parameters?

I'm trying to understand DQN. I understand where the loss function comes from. I'm just unsure about why the target function works in practice. Given the loss function $$ L_i(\theta_i) = [(y_i - Q(s,a;...
2
votes
0answers
52 views

Are the final states not being updated in this $n$-step Q-Learning algorithm?

I am reading this paper and in algorithm 3 they describe an $n$-step Q-Learning algorithm. Below is the pseudo-code. From this pseudo-code, it looks as though the final tuples that they would ...
1
vote
1answer
145 views

Can tabular Q-learning converge even if it doesn't explore all state-action pairs?

My understanding of tabular Q-learning is that it essentially builds a dictionary of state-action pairs, so as to maximize the Markovian (i.e., step-wise, history-agnostic?) reward. This incremental ...
2
votes
0answers
87 views

How to prevent deep Q-learning algorithms to overfit?

I have recently solved the Cartpole problem using double deep Q-learning. When I saw how the agent was doing, it used to go right every time, never left, and it did similar actions all the time. Did ...
1
vote
1answer
116 views

If deep Q-learning starts to choose only one action, is this a sign that the algorithm diverged?

I'm working on a deep q-learning model in an infinite horizon problem, with a continous state space and 3 possible actions. I'm using a neural network to approximate the action-value function. ...

1 2
3
4 5
7