Questions tagged [q-learning]

For questions related to the Q-learning algorithm, which is a model-free and temporal-difference reinforcement learning algorithm that attempts to approximate the Q function, which is a function that, given a state s and an action a, returns a real number that represents the return (or value) of state s when action a is taken from s. Q-learning was introduced in the PhD thesis "Learning from Delayed Rewards" (1989) by Watkins.

Filter by
Sorted by
Tagged with
0
votes
1answer
39 views

Is a genetic algorithm efficient for a snake game?

I am working on a DIY project in which I want to be able to train a neural network to play Snake. Is a genetic algorithm an efficient way of training a network for this application? For a GA, what ...
0
votes
0answers
14 views

Reward firstly increase, but after more episodes, start decrease, and weights diverges

I'm making a simple deep Q learning algorithm, with cartpole-v1 env. Like you can see in chart, after many episodes the reward decrease, some possible reasons? The exploration vs axplotation algorithm ...
0
votes
0answers
7 views

DQN + HER, TD Error spiked then success rate plummets. What went wrong?

TL;DR: I trained a DQN + HER model using stable-baselines library for a custom environment. I noticed that in most runs, sometimes the TD-Error will spike and then the success rate of my model ...
0
votes
0answers
45 views

Reinforcement learning for rearranging the mobile home screen icon layout: what inputs/states do I need to pass into the algorithm?

I have a problem where I need to rearrange a particular user's mobile home screen icon layout. Let's say that the social media app usage of a user is high compared to other app usage. So I need the ...
3
votes
1answer
77 views

In DQN, would it be cheaper to have $N$ neural networks with a single real-valued output, one for each of the $N$ actions?

In the classical examples of deep q-learning, I often see neural networks in which the input represents the state of the agent, while the output is a tuple with all the values of $Q(s, a)$ predicted ...
1
vote
1answer
42 views

How to scale all positive continuous reward?

My RL project has all positive continuous rewards for every step and the goal is to have the maximum cumulative reward (episodic reward). The problem is that the rewards are too close and all between ...
0
votes
0answers
16 views

Training labels: integers or vectors?

I'm trying to implement Deep Q Learning using Tensorflow. The input is a vectorized representation of the state, and the output is a vector whose length is the number of possible actions. I've already ...
-1
votes
0answers
38 views

Can I create a Q-table in iterated procedure?

Let's say I have a time series vector (hourly price data), and want to obtain an optimal trading policy. To do so, I need a Q-table. My question is about creating such table from a raw time-series. ...
1
vote
1answer
37 views

How to properly resume training of deep Q-learning network?

I'm currently training a deep q-learning network. Due to resource limitations, I am not able to train the model to the desired performance in one go. So what I'm doing now is training the model for a ...
1
vote
1answer
23 views

Is is not possible to achieve average reward of more than 20-40 with simple Q-Learning

I have implemented the simple Q-Learning based solution for AI-gym's Cartpole-v0. However, despite changing hyper-parameters, and rechecking my code, I cannot get an average reward (N-running reward) ...
0
votes
1answer
41 views

Given a sequence of states followed by the agent, is it possible to find the Q-value for a state-action pair not in this sequence?

Assume you are given a sequence of states followed by the agent, generated by a random policy, $[s_0, s_1, s_2, \dots, s_n]$. Furthermore, assume the MDP is fully observable and time is discrete. Is ...
0
votes
0answers
12 views

Basic RL code for Production Line Optimization ( Need help )

I am working on how to optimize a Production line given 4 Workstations , 4 Employees(MA) . There are 5 products whose Workpackages need to be completed ( in order ) and these workpackages need to be ...
0
votes
0answers
22 views

MicroPython MicroMLP: How do I reward the program based on state?

I have been trying to use MicroMLP to teach a small neural network to converge to correct results. Ultimately, I want to have three outputs, one which is high priority (...
0
votes
1answer
73 views

Reinforcement Learning for an environment that is non-markovian

I am a beginner in the field of Reinforcement Learning with only a couple of months of experience being in the field. Soon, I will start working on a project where we want to optimize the production ...
1
vote
1answer
54 views

How to avoid being stuck local optima in q-learning and q-network

When using Bellman equation to update q-table or train q-network to fit to greedy max values, the q-values very often get to the local optima and get stuck although randomisation rate ($\epsilon$) ...
1
vote
1answer
45 views

Q-learning in gridworld with random board

I'm trying to use Q-learning in order to solve Wumpus world environment. Wumpus world is a toy problem on 4x4 gridworld. The agent starts in entry position of the cave, looks for gold (agent can sense ...
1
vote
1answer
28 views

Compute state space from variables in Q-learning (RL)

I'm trying to use Q-learning, but I'm stuck because I don't know how to compute the state. Let's say, in my problem, there are the following variables, which I'm using to compute state: ...
2
votes
0answers
22 views

Are there any known disadvantages of implementing vanilla Q-learning on a discretized-state space environment?

For an RL problem on a continuous state space, the states could be discretized into buckets and these buckets used in implementing the Q-table. I see that is what is done here. However, according to ...
2
votes
1answer
67 views

What is a good convergence criterion for Q-learning in a stochastic environment?

I have a stochastic environment and I'm implementing a Q-table for the learning that happens on the environment. The code is shown below. In short, there are ten states (0, 1, 2,...,9), and three ...
4
votes
1answer
70 views

How would I compute the optimal state-action value for a certain state and action?

I am currently trying to learn reinforcement learning and I started with the basic gridworld application. I tried Q-learning with the following parameters: Learning rate = 0.1 Discount factor = 0.95 ...
4
votes
1answer
173 views

Why does Q-learning converge under 100% exploration rate?

I am working on this assignment where I made the agent learn state-action values (Q-values) with Q-learning and 100% exploration rate. The environment is the classic gridworld as shown in the ...
0
votes
0answers
5 views

What is an appropriate stop criteria for training on a non-stationary environment in reinforcement learning?

I'm currently studying reinforcement learning (RL) and would like to understand non-stationary environments better. So for stationary environments, the Q-values of all state-action pairs converge ...
3
votes
1answer
70 views

Why can we take the action $a$ from the next state $s'$ in the max part of the Q-learning update rule, if that action doesn't lead to any reward?

I'm using OpenAI's cartpole environment. First of all, is this environment not Markov? Knowing that, my main question concerns Q-learning and off-policy methods: For me, there is something weird in ...
0
votes
0answers
22 views

How DynaQ behaves in stochastic world in comparison with other reinforcement learning algorithms?

I came across of implementations of a bunch of algorithms on stochastic windy gridworld. This is the graph comparing their performance: So clearly, it seems that DynaQ performs better than all other ...
2
votes
2answers
111 views

What is the target output for updating a Deep Q Network

I'm trying to implement Deep Q-Learning for a pet problem having a continuous state space and discretized action space. The algorithm for table-based Q-Learning updates a single entry of the Q table - ...
0
votes
1answer
56 views

What are the popular approaches to estimating the Q-function?

I need the q-value for my RL training, there are some approaches: Brute-force the action sequence (this won't work for long sequence) Use a classic algorithm to optimise and estimate (this ain't much ...
5
votes
3answers
462 views

In Q-learning, wouldn't it be better to simply iterate through all possible states?

In Q-learning, all resources I've found seem to say that the algorithm to update the Q-table should start at some initial state, and pick actions (which are sometimes random) to explore the state ...
1
vote
1answer
173 views

What is a “learned policy” in Q-learning?

I am completing an assignment at the moment. One of the assignment questions asks how you identified the learned policy and how you obtained it. The question is a reinforcement learning question, and ...
0
votes
1answer
56 views

Relationship between Rewards and Q Value (Graph between Q(s, a) vs episodes)

I'm employing the Actor-Critic algorithm. The critic network approximates the action-value function, i.e. $Q(s, a)$, which determines how good a particular state is, when provided with an action. $Q(s,...
9
votes
2answers
1k views

What is the difference between Q-learning, Deep Q-learning and Deep Q-network?

Q-learning uses a table to store all state-action pairs. Q-learning is a model-free RL algorithm, so how could there be the one called Deep Q-learning, as deep means using DNN; or maybe the state-...
2
votes
0answers
27 views

Is better to reward short- or long-term progress in Q-learning?

I have been training some kind of agent to reach a target using a Q-learning based approach, and I have tried two different types of rewards: Long-term reward: $\mathrm{reward} = - \mathrm{distance}(\...
0
votes
1answer
64 views

When calculating the cost in deep Q-learning, do we use both the input and target states?

I just finished Andrew Ngs's deep learning specialization, but RL was not covered, so I don't know the basics of RL. So, I have been having trouble understanding the cost function in deep Q-learning. ...
2
votes
0answers
48 views

Offline/Batch Reinforcement Learning: when to stop training and what agent to select

Context: My team and I are working on a RL problem for a specific application. We have data collected from user interactions (states, actions, rewards, etc.). It is too costly for us to emulate agents....
3
votes
1answer
198 views

Why does regular Q-learning (and DQN) overestimate the Q values?

The motivation for the introduction of double DQN (and double Q-learning) is that the regular Q-learning (or DQN) can overestimate the Q value, but is there a brief explanation as to why it is ...
4
votes
1answer
108 views

Research into social behavior in Prisoner's Dilemma

I've been working on research into reproducing social behavior using multi-agent reinforcement learning. My focus has been on a GridWorld-style game, but I was thinking that maybe a simpler Prisoner's ...
0
votes
2answers
97 views

Simple DQN too slow to train [closed]

I have been trying to solve the OpenAI lunar lander game with a DQN taken from this paper https://arxiv.org/pdf/2006.04938v2.pdf The issue is that it takes 12 hours to train 50 episodes so something ...
0
votes
0answers
55 views

Is there any toy example that can exemplify the performance of double Q-learning?

I recently tried to reproduce the results of double Q-learning. However, the results are not satisfying. I have also tried to compare double Q learning with Q-learning in Taxi-v3, FrozenLake without ...
1
vote
1answer
57 views

How to build a Neural Network to approximate the Q-function?

I am learning reinforcement learning with Q-learning using online resources, like blog posts, youtube videos, and books. At this point, I have learned the underpinning concepts of reinforcement ...
0
votes
0answers
39 views

Why convergence is not guaranteed when using approximation? [duplicate]

I am doing self study of Reinforcement Learning with Q-learning using online resources like blog posts, youtube videos and books and at this point, I have learned the underpinning concepts of ...
6
votes
1answer
127 views

How is the DQN loss derived from (or theoretically motivated by) the Bellman equation, and how is it related to the Q-learning update?

I'm doing a project on Reinforcement Learning. I programmed an agent that uses DDQN. There are a lot of tutorials on that, so the code implementation was not that hard. However, I have problems ...
1
vote
1answer
66 views

If $\alpha$ decreases over time, why is Q-learning guaranteed to converge?

Q-Learning is guaranteed to converge if $\alpha$ decreases over time. On page 161 of the RL book by Sutton and Barto, 2nd edition, section 8.1, they write that Dyna-Q is guaranteed to converge if each ...
2
votes
2answers
266 views

What happens when the agent faces a state that never before encountered?

I have a network with nodes and links, each of them with a certain amount of resources (that can take discrete values) at the initial state. At random time steps, a service is generated, and, based on ...
0
votes
1answer
51 views

Why is the policy implied by Q-learning deterministic, when it always chooses the action with highest probability?

Q-learning uses the maximizing value at each step, which implies that there is a probability distribution and it happens to choose the one with the highest probability. There is no direct mapping ...
0
votes
0answers
36 views
2
votes
1answer
66 views

Do I need to know in advance all possible number of states in Q-Learning?

In Q-learning, is it mandatory to know all possible states that can the agent may end up in? I have a network with 4 source nodes, 3 sink nodes, and 4 main links. The initial state is the status ...
2
votes
1answer
70 views

What constitutes a large space state (in Q-learning)?

I know this might be specific to different problems, but does anyone know if there is any rule of thumb or references on what constitutes a large state space? I know that, according to multiple papers,...
1
vote
0answers
29 views

How to compute the Retrace target for multi-step off-policy Reinforcement Learning?

I am implementing the A3C algorithm and I want to add off-policy training using Retrace but I am having some trouble understanding how to compute the retrace target. Retrace is used in combination ...
3
votes
1answer
93 views

Why does off-policy learning outperform on-policy learning?

I am self-studying about Reinforcement Learning using different online resources. I now have a basic understanding of how RL works. I saw this in a book: Q-learning is an off-policy learner. An off-...
2
votes
1answer
69 views

What should the value of epsilon be in the Q-learning?

I am trying to understand Reinforcement Learning and already explored different Youtube videos, blog posts, and Wikipedia articles. What I don't understand is the impact of $\epsilon$. What value ...
2
votes
1answer
40 views

Can we use Q-learning update for policy evaluation (not control)?

For policy evaluation purposes, can we use the Q-learning algorithm even though, technically, it is meant for control? Maybe like this: Have the policy to be evaluated as the behaviour policy. Update ...

1
2 3 4 5
7