What's the deal with Deno? We talk with a major contributor to find out. Listen now.

Questions tagged [q-learning]

For questions related to the Q-learning algorithm, which is a model-free and temporal-difference reinforcement learning algorithm that attempts to approximate the Q function, which is a function that, given a state s and an action a, returns a real number that represents the return (or value) of state s when action a is taken from s. Q-learning was introduced in the PhD thesis "Learning from Delayed Rewards" (1989) by Watkins.

Filter by
Sorted by
Tagged with
1
vote
1answer
38 views

Why do my rewards reduce after extensive training using D3QN?

I am running a drone simulator for collision avoidance using a slight variant of D3QN. The training is usually costly (runs for at least a week) and I have observed that reward function gradually ...
0
votes
2answers
63 views

How does one know that a problem is “model-free” in reinforcement learning?

Consider this slide from a Stanford lecture on reinforcement learning. It states that a model is the agent's representation of how the world changes in response to the agent's action. I've been ...
1
vote
0answers
26 views

Pytorch Deep q network not learning and step not stepping towards target

I am trying to create a simple deep q network for rl with conv2d layers. I can’t figure out what I am doing wrong, and the only thing I can see that doesn’t seem right is when I get the model ...
2
votes
0answers
27 views

Handling a Large Discrete Action Space in Deep Q Learning

I am attempting to solve a timetabling problem using deep Q learning. It could be thought of as a resource allocation problem to obtain some certificate of 'optimality'. However, how to define and ...
1
vote
0answers
20 views

Is it feasible to train a DQN with thousands of input ports?

I designed a DQN architecture for some problem. The problem has a parameter $m$ as the number of clients. In my situation, $m$ is large, $m\in\{100,200,\ldots,1000\}$. For this situation, the number ...
1
vote
1answer
39 views

What is the optimal exploration-exploitation trade-off in Q*bert?

I am training an RL agent with Deep Q-learning + Experience Replay on the Q*bert Atari environment. After 400,000 frames, my agent appears to have learned strategic information about the game, but ...
2
votes
2answers
52 views

How to apply Q-learning when rewards is only available at the last state?

I have a scheduling problem in which there are $n$ slots and $m$ clients. I am trying to solve the problem using Q-learning so I have made the following state-action model. A state $s_t$ is given by ...
2
votes
1answer
70 views

Why is sampling non-uniformly from the replay memory an issue? (Prioritized experience replay)

I can't seem to understand why we need importance sampling in prioritized experience replay (PER). The authors of the paper write on page 5: The estimation of the expected value with stochastic ...
3
votes
1answer
31 views

Is there a logical method of deducing an optimal batch size when training a Deep Q-learning agent with experience replay?

I am training an RL agent using Deep-Q learning with experience replay. At each frame, I am currently sampling 32 random transitions from a queue which stores a maximum of 20000 and training as ...
1
vote
1answer
22 views

How is weighted average computed in Deep Q networks

I was going through the Sutton book and they said the update formula for Q learning comes from the weighted average of the returns I.e New estimate= old estimate +alpha*[returns- old estimate] So by ...
3
votes
1answer
1k views

What are the differences between Q-Learning and A*?

Q-learning seems to be related to A*. I am wondering if there are (and what are) the differences between them.
2
votes
1answer
68 views

How to compute the target for double Q-learning update step?

I've already read the original paper about double DQN but I do not find a clear and practical explanation of how the target $y$ is computed, so here's how I interpreted the method (let's say I have 3 ...
1
vote
0answers
28 views

Customized food for persons based on their profile using Reinforcement learning

I am newbie to Reinforcement Learning, this is my idea - Agent(food provider) has to select a food based on the environment(based on the user profile). Here the reward will be given to the agent based ...
2
votes
1answer
47 views

When using experience replay in reinforcement learning, which state is used for training?

I'm slightly confused about the experience replay process. I understand why we use batch processing in reinforcement learning, and from my understanding, a batch of states is input into the neural ...
2
votes
0answers
54 views

Should I use the discounted average reward as objective in a finite-horizon problem?

I am new to reinforcement learning, but, for a finite horizon application problem, I am considering using the average reward instead of the sum of rewards as the objective. Specifically, there are a ...
3
votes
1answer
81 views

When do SARSA and Q-Learning converge to optimal Q values?

Here's another interesting multiple-choice question that puzzles me a bit. In tabular MDPs, if using a decision policy that visits all states an infinite number of times, and in each state, randomly ...
0
votes
0answers
33 views

How does DQN convergence work in reinforcement learning

In supervised learning we have an unbiased target value, but in reinforcement learning this isn’t the case The network predicts its own target value, now how exactly does it converge if the network ...
1
vote
0answers
32 views

How can deep Q-learning converge if the targets may not be correct?

In deep Q-learning, $Q(s, a)$ and $Q'(s, a)$ are predicted or estimated by the neural network itself. In supervised learning, the target value is a true unbiased value. However, this isn't the case in ...
1
vote
1answer
91 views

What exactly is the advantage of DDQN over DQN

I started looking into DDQN and apparently the difference is we use our Online network for action selection, And we use our target network for outputting the Q values, I don’t quite get how this is ...
0
votes
0answers
61 views

Strange behavior of Q-learning agent after being trained

I built a simple X*Y grid world environment to learn and then trained my agent over it. All worked fine and the agent learned as well. Let me give some detail about the environment. Environment: A ...
1
vote
1answer
73 views

How can I fetch ​exploration decay rate of an iterable Q-table in Python?

I have done creating the virtual environment, creating the Q-table, initializing the q-parameters, then I made a training module and stored it in a numpy array. ...
0
votes
1answer
80 views

How can I update my Q-table in Python?

I want to implement this function on a voice searching application: $$ Q(S, A) \leftarrow Q(S, A)+\alpha\left(R+\gamma Q\left(S^{\prime}, A^{\prime}\right)-Q(S, A)\right) $$ And also restricted to use ...
2
votes
0answers
62 views

Most of state-action pairs remain unvisited in the q-table

In building my first Q-learning algorithm for OpenAI gym's CartPole problem, many of my states remain unvisited. I believe it is the reason that my agent does not learn. Can I be told of the reasons I ...
1
vote
0answers
55 views

OpenAI gym's CartPole problem system does not learn

My OpenAI CartPole-v0 problem's implementation using basic Q-learning does not learn at all. I am a beginner and have implemented my first ever Q-learning from scratch after learning from tutorials. ...
4
votes
1answer
175 views

Why do DQNs tend to forget?

Why do DQNs tend to forget? Is it because when you feed highly correlated samples, your model (function approximation) doesn't give a general solution? For example: I use level 1 experiences, my ...
2
votes
1answer
68 views

Reinforcement learning with action consisting of two discrete values

I'm new to reinforcement learning. I have a problem where an action is composed of an order (rod with a required length) and an item from a warehouse (an existing rod with a certain length, which will ...
5
votes
0answers
139 views

Is the Bellman equation that uses sampling weighted by the Q values (instead of max) a contraction?

It is proved that the Bellman update is a contraction (1). Here is the Bellman update that is used for Q-Learning: $$Q_{t+1}(s, a) = Q_{t}(s, a) + \alpha*(r(s, a, s') + \gamma \max_{a^*} (Q_{t}(s', ...
0
votes
0answers
45 views

Predict probability of user making a conversion

My dear friends, In the past couple of years I read a lot about AI with JS and some libraries like TensorFlow. I have great interest in the subject but never used it on a serious project. However, ...
0
votes
1answer
54 views

How can I change observation states' values in OpenAI gym's cartpole environment?

I am learning with the OpenAI gym's cart pole environment. I want to make the observation states discrete (with small stepsize) and for that purpose, I need to change two of the observations from [$ -\...
4
votes
3answers
257 views

Upper limit to the maximum cumulative reward in a deep reinforcement learning problem

Is there an upper limit to the maximum cumulative reward in a deep reinforcement learning problem? For example you want to train an DQN agent in an environment and you want to know what is the highest ...
2
votes
0answers
49 views

Prioritised Remembering in Experience Replay (Q-Learning)

I'm using Experience Replay based on the original Prioritized Experience Replay (PER) paper. In the paper authors show ~ an order of magnitude increase in data efficiency from prioritized sampling. ...
2
votes
1answer
96 views

Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards?

Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards? Would it not make more sense to compute $\mathbb{E}(R \mid s, a)$ (the expected return for taking ...
0
votes
1answer
67 views

What is the purpose of a Neural Network in Reinforcement Learning when we have a Q-learning update rule?

I'm confused as to the purpose of training a neural network (NN) for reinforcement learning (RL) tasks such as Gridworld. In RL tasks, namely q-learning, we have a q-learning update rule, which is ...
1
vote
1answer
57 views

Why do we need target network in deep Q learning? [duplicate]

I already know deep RL, but to learn it deeply I want to know why do we need 2 networks in deep RL. What does the target network do? I now there is huge mathematics into this, but I want to know deep ...
2
votes
1answer
40 views

What is convergence analysis, and why is it needed in reinforcement learning?

While reading a paper about Q-learning in network energy consumption, I came across the section on convergence analysis. Does anyone know what convergence analysis is, and why is convergence analysis ...
2
votes
0answers
43 views

How can I formulate a prediction problem (given labeled data) as an RL problem and solve it with Q-learning?

One of my friends sent me a problem he was working on lately, and I couldn't help but I wonder how could it be solved using Q-learning. The statement is as follows: Given the following datasets, the ...
0
votes
1answer
46 views

How does training for DQN work if messing up in the environment in costly?

Suppose that we want to train a car to drive in the real world and decide to use Reinforcement Learning (specifically, DQN) for that. I am a bit confused about how training generally works. Is it that ...
1
vote
0answers
24 views

Tic-tac-toe: How would standard SARSA and Q-learning yield different results in the agent's behaviour?

I know this is deceptively simple. Tic tac toe is a well studied game for RL. Assume your agent is playing aggainst a strong opponent. I know you deal in after states. I know that in Q learning the ...
0
votes
0answers
33 views

Why Q-Learning and SARSA have terrible performance?

I am trying to solve a MDP problem with almost 50 states and 60 actions with Q-Learning or SARSA. However, the performance of both algorithms is terrible and cannot find the optimal policy found by ...
0
votes
0answers
33 views

Why doesn't my double deep Q network trained with the same training set give consistent performance?

I've written a Double DQN which can do either 1-step or multi-step learning. It also has a prioritised reply buffer. The internal network is an LSTM. My input data is a series of time series data and ...
2
votes
1answer
62 views

Implementing SARSA for a 2-stage Markov Decision Process

I am a bit confused as to how exactly I should be implementing SARSA (or Q-learning too) on what is a simple 2-stage Markov Decision Task. The structure of the task is as follows: Basically, there ...
2
votes
1answer
73 views

q learning appears to converge but does not always win against random tic tac toe player

q learning is defined as: Here is my implementation of q learning of the tic tac toe problem: ...
2
votes
1answer
44 views

Why isn't it wise for us to completely erase our old Q value and replace it with the calculated Q value?

Why isn't it wise for us to completely erase our old Q value and replace it with the calculated Q value? Why can't we forget the learning rate and temporal difference? Here's the update formula.
3
votes
2answers
91 views

Why is it not advisable to have a 100 percent exploration rate? [duplicate]

During the learning phase, why don't we have a 100% exploration rate, to allow our agent to fully explore our environment and update the Q values, then during testing we bring in exploitation? Does ...
0
votes
1answer
44 views

Why do we update the weights of the target network in deep Q learning?

I know we keep the target network constant during training to improve stability, but why exactly are we updating the weights of our target network? In particular, if we've already reached convergence, ...
1
vote
2answers
57 views

Why do we explore after we have an accurate estimate of the value function?

Suppose we have a small space state and that, after about 2000 episodes, we've accurately explored the environment and known the accurate $Q$ values. In that case, why do we still leave a small ...
4
votes
1answer
85 views

What happens when you select actions using softmax instead of epsilon greedy in DQN?

I understand the two major branches of RL are Q-Learning and Policy Gradient methods. From my understanding (correct me if I'm wrong), policy gradient methods have an inherent exploration built-in as ...
0
votes
0answers
31 views

Reinforcement Learning Diagnostic: Total reward doesn't converge

I'm implementing DDQN in my toy scenario. During training, I'm surprised to see that the total reward doesn't converge and have a tendency to degrade. What could be the problem? Here's the picture: ...
1
vote
1answer
35 views

What would happen if we sampled only one tuple from the experience replay?

The concept of experience replay is saving our experiences in our replay buffer. We select at random to break the correlation between consecutive samples, right? What would happen if we calculate our ...
1
vote
1answer
95 views

What's the best practice for Boltzmann Exploration temperature in RL?

I'm currently modeling DQN in Reinforcement Learning. My question is: what are the best practices related to Boltzmann Exploration? My current thoughts are: (1) Let the temperature decay through ...

1
2 3 4 5 6