# Questions tagged [q-learning]

For questions related to the Q-learning algorithm, which is a model-free and temporal-difference reinforcement learning algorithm that attempts to approximate the Q function, which is a function that, given a state s and an action a, returns a real number that represents the return (or value) of state s when action a is taken from s. Q-learning was introduced in the PhD thesis "Learning from Delayed Rewards" (1989) by Watkins.

317 questions
Filter by
Sorted by
Tagged with
107 views

302 views

### How to determine if Q-learning has converged in practice？

I am using Q-learning and SARSA to solve a problem. The agent learns to go from the start to the goal without falling in the holes. At each state, I can choose the action corresponding to the maximum ...
176 views

### Reinforcement learning simple problem: agent not learning, wrong action

I am pretty new to RL and I am trying to code a simple RL task with pytorch. The goal/task is the following: The initial state is $t_o$ and the agent takes an action $\Delta_t$: $t_o +\Delta_t = t_1$. ...
110 views

### Why is the reward function $\text{reward} = 1/{(\text{cost}+1)^2}$ better than $\text{reward} =1/(\text{cost}+1)$?

I have implemented a simple Q-learning algorithm to minimize a cost function by setting the reward to the inverse of the cost of the action taken by the agent. The algorithm converges nicely, but ...
36 views

### Pytorch and keras ddqn seem identical, only keras learns

I followed a tutorial for ddqn to beat pong, it beats it with a perfect score in keras, but trying to translate it to pytorch it doesn't learn at all. What am I missing? I pasted all the code for each ...
64 views

### Can we stop training as soon as epsilon is small?

I'm new to reinforcement learning. As it is common in RL, $\epsilon$-greedy search for the behavior/exploration is used. So, at the beginning of the training, $\epsilon$ is high, and therefore a lot ...
164 views

### Why do my rewards reduce after extensive training using D3QN?

I am running a drone simulator for collision avoidance using a slight variant of D3QN. The training is usually costly (runs for at least a week) and I have observed that reward function gradually ...
211 views

### Why can't we apply value iteration when we do not know the reward and transition functions, and how does Q-learning solve this issue?

I don't understand why we can't apply value iteration when don't know the reward and transition probabilities. In this lecture, the lecturer says it has to do with not being able to take max with ...
137 views

### How does one know that a problem is “model-free” in reinforcement learning?

Consider this slide from a Stanford lecture on reinforcement learning. It states that a model is the agent's representation of how the world changes in response to the agent's action. I've been ...
48 views

### Handling a Large Discrete Action Space in Deep Q Learning

I am attempting to solve a timetabling problem using deep Q learning. It could be thought of as a resource allocation problem to obtain some certificate of 'optimality'. However, how to define and ...
103 views

### Why is sampling non-uniformly from the replay memory an issue? (Prioritized experience replay)

I can't seem to understand why we need importance sampling in prioritized experience replay (PER). The authors of the paper write on page 5: The estimation of the expected value with stochastic ...
24 views

### Is it feasible to train a DQN with thousands of input ports?

I designed a DQN architecture for some problem. The problem has a parameter $m$ as the number of clients. In my situation, $m$ is large, $m\in\{100,200,\ldots,1000\}$. For this situation, the number ...
53 views

### What is the optimal exploration-exploitation trade-off in Q*bert?

I am training an RL agent with Deep Q-learning + Experience Replay on the Q*bert Atari environment. After 400,000 frames, my agent appears to have learned strategic information about the game, but ...
194 views

### Is there a logical method of deducing an optimal batch size when training a Deep Q-learning agent with experience replay?

I am training an RL agent using Deep-Q learning with experience replay. At each frame, I am currently sampling 32 random transitions from a queue which stores a maximum of 20000 and training as ...
28 views

### How is weighted average computed in Deep Q networks

I was going through the Sutton book and they said the update formula for Q learning comes from the weighted average of the returns I.e New estimate= old estimate +alpha*[returns- old estimate] So by ...
1k views

### What are the differences between Q-Learning and A*?

Q-learning seems to be related to A*. I am wondering if there are (and what are) the differences between them.
306 views

### What are some online courses for deep reinforcement learning?

What are some (good) online courses for deep reinforcement learning? I would like the course to be both programming and theoretical. I really liked David Silver's course, but the course dates from ...
30 views

### Customized food for persons based on their profile using Reinforcement learning

I am newbie to Reinforcement Learning, this is my idea - Agent(food provider) has to select a food based on the environment(based on the user profile). Here the reward will be given to the agent based ...
71 views

### When using experience replay in reinforcement learning, which state is used for training?

I'm slightly confused about the experience replay process. I understand why we use batch processing in reinforcement learning, and from my understanding, a batch of states is input into the neural ...
63 views

### Should I use the discounted average reward as objective in a finite-horizon problem?

I am new to reinforcement learning, but, for a finite horizon application problem, I am considering using the average reward instead of the sum of rewards as the objective. Specifically, there are a ...
279 views

### When do SARSA and Q-Learning converge to optimal Q values?

Here's another interesting multiple-choice question that puzzles me a bit. In tabular MDPs, if using a decision policy that visits all states an infinite number of times, and in each state, randomly ...
50 views

### How does DQN convergence work in reinforcement learning

In supervised learning we have an unbiased target value, but in reinforcement learning this isn’t the case The network predicts its own target value, now how exactly does it converge if the network ...
32 views

### How can deep Q-learning converge if the targets may not be correct?

In deep Q-learning, $Q(s, a)$ and $Q'(s, a)$ are predicted or estimated by the neural network itself. In supervised learning, the target value is a true unbiased value. However, this isn't the case in ...
65 views

### Strange behavior of Q-learning agent after being trained

I built a simple X*Y grid world environment to learn and then trained my agent over it. All worked fine and the agent learned as well. Let me give some detail about the environment. Environment: A ...
81 views

### How can I update my Q-table in Python?

I want to implement this function on a voice searching application: $$Q(S, A) \leftarrow Q(S, A)+\alpha\left(R+\gamma Q\left(S^{\prime}, A^{\prime}\right)-Q(S, A)\right)$$ And also restricted to use ...
79 views

### How can I fetch ​exploration decay rate of an iterable Q-table in Python?

I have done creating the virtual environment, creating the Q-table, initializing the q-parameters, then I made a training module and stored it in a numpy array. ...
70 views

### Most of state-action pairs remain unvisited in the q-table

In building my first Q-learning algorithm for OpenAI gym's CartPole problem, many of my states remain unvisited. I believe it is the reason that my agent does not learn. Can I be told of the reasons I ...
80 views

### OpenAI gym's CartPole problem system does not learn

My OpenAI CartPole-v0 problem's implementation using basic Q-learning does not learn at all. I am a beginner and have implemented my first ever Q-learning from scratch after learning from tutorials. ...
242 views

### Why do DQNs tend to forget?

Why do DQNs tend to forget? Is it because when you feed highly correlated samples, your model (function approximation) doesn't give a general solution? For example: I use level 1 experiences, my ...
72 views

### Reinforcement learning with action consisting of two discrete values

I'm new to reinforcement learning. I have a problem where an action is composed of an order (rod with a required length) and an item from a warehouse (an existing rod with a certain length, which will ...
165 views

### Is the Bellman equation that uses sampling weighted by the Q values (instead of max) a contraction?

It is proved that the Bellman update is a contraction (1). Here is the Bellman update that is used for Q-Learning: Q_{t+1}(s, a) = Q_{t}(s, a) + \alpha*(r(s, a, s') + \gamma \max_{a^*} (Q_{t}(s', ...
307 views

52 views

### Prioritised Remembering in Experience Replay (Q-Learning)

I'm using Experience Replay based on the original Prioritized Experience Replay (PER) paper. In the paper authors show ~ an order of magnitude increase in data efficiency from prioritized sampling. ...
141 views

### What is the purpose of a Neural Network in Reinforcement Learning when we have a Q-learning update rule?

I'm confused as to the purpose of training a neural network (NN) for reinforcement learning (RL) tasks such as Gridworld. In RL tasks, namely q-learning, we have a q-learning update rule, which is ...
46 views

### What is convergence analysis, and why is it needed in reinforcement learning?

While reading a paper about Q-learning in network energy consumption, I came across the section on convergence analysis. Does anyone know what convergence analysis is, and why is convergence analysis ...
699 views

### Why do we need target network in deep Q learning? [duplicate]

I already know deep RL, but to learn it deeply I want to know why do we need 2 networks in deep RL. What does the target network do? I now there is huge mathematics into this, but I want to know deep ...
43 views

### How can I formulate a prediction problem (given labeled data) as an RL problem and solve it with Q-learning?

One of my friends sent me a problem he was working on lately, and I couldn't help but I wonder how could it be solved using Q-learning. The statement is as follows: Given the following datasets, the ...
97 views

### How does training for DQN work if messing up in the environment in costly?

Suppose that we want to train a car to drive in the real world and decide to use Reinforcement Learning (specifically, DQN) for that. I am a bit confused about how training generally works. Is it that ...
56 views

### How to predict Q-values based on stack of frames

I decided to train deep Q-learning agent based on getting raw pixels from environment.I have one particular problem:when I input stack of frames, suppose 4 consecutive frames, if action space is 6,...
77 views

### Tic-tac-toe: How would standard SARSA and Q-learning yield different results in the agent's behaviour?

I know this is deceptively simple. Tic tac toe is a well studied game for RL. Assume your agent is playing aggainst a strong opponent. I know you deal in after states. I know that in Q learning the ...