Questions tagged [q-learning]

For questions related to the Q-learning algorithm, which is a model-free and temporal-difference reinforcement learning algorithm that attempts to approximate the Q function, which is a function that, given a state s and an action a, returns a real number that represents the return (or value) of state s when action a is taken from s. Q-learning was introduced in the PhD thesis "Learning from Delayed Rewards" (1989) by Watkins.

323 questions
Filter by
Sorted by
Tagged with
45 views

What is the derivative of equation 1 in the paper "Conservative Q-Learning for Offline Reinforcement Learning"?

I am looking at the paper Conservative Q-Learning for Offline Reinforcement Learning, but I'm not sure how they proved theorem 3.1. Here is a screenshot of theorem 3.1. In the proof of theorem 3.1 ...
70 views
+100

How to approach a blackjack-like card game with the possibility of cards being counted?

Consider a single-player card game which shares many characteristics to "unprofessional" (not being played in casino, refer point 2) Blackjack, i.e.: You're playing against a dealer with ...
18 views

Why does Q-function training not query the Q-function value at unobserved states?

It says here in the Conservative Q-Learning paper that "standard Q-function training does not query the Q-function value at unobserved states, but queries the Q-function at unseen actions" (...
78 views

Is it really hard to learn in a stochastic environment?

I understand that a stochastic environment is one that does not always lead you to the desired state by giving a particular action A (But the probability to change to a not desire state is fixed right?...
48 views

When to activate batch normalization and dropout in deep Q-learning?

In the vanilla version of deep Q-learning, there are three places where the Q-network is queried: When exploring. When training: a. When calculating the optimal value of the state reached by an ...
32 views

In deep reinforcement learning, what is this model with state as input and value as output?

I was looking at this implementation for creating an agent for playing Tetris using DeepRL. This model uses "a state based on the statistics of the board after a potential action. All predictions ...
34 views

Defining states and possible actions in Q learning

I am trying to define the number of states and possible actions for a reinforcement learning problem that I want to solve with Q-learning, but I am a bit confused, as I'm totally new to reinforcement ...
28 views

Where can I find the original conference paper that introduced Q-learning and Deep Q-Learning?

I tried searching a lot, but I could neither find the paper that introduced Q-Learning nor the paper that introduced Deep Q Learning. If anyone knows anything about it please do tell me.
54 views

Why do I get bad results no matter my neural network function approximator for parametrized Q-learning implementation for Contextual Bandits?

I'd like to ask you why, no matter my neural network function approximator for parametrized Q-learning implementation for a Contextual Bandits environment, I'm getting bad results. I don't know if it'...
54 views

Which policy has to be followed by a player while construction of its own Q-table?

Consider the scenario, where there are two players. One of the players perform the action randomly, whereas I want second player as a Q-player. I mean, the player selects a best action from the Q-...
63 views

Which policy do I need to use in updating Q function?

Policy function can be of two types: deterministic policy and stochastic policy. Deterministic policy is of the form $\pi : S \rightarrow A$ Stochastic policy is defined using conditional probability ...
66 views

In addition to the reward function, which other functions do I need to implement Q-learning?

In general, $Q$ function is defined as $$Q : S \times A \rightarrow \mathbb{R}$$ $$Q(s_t,a_t) = Q(s_t,a_t) + \alpha[r_{t+1} + \gamma \max\limits_{a} Q(s_{t+1},a) - Q(s_t,a_t)]$$ $\alpha$ and $\gamma$...
21 views

Is there any work that applies the approach in "Finite-Sample Convergence Rates for Q-Learning and Indirect Algorithms" to standard Q-learning?

I am trying to mathematically characterize the finite sample convergence rates for Q-learning. To this end, I have read the following papers Learning rates for Q-learning, by Eyal Even-Dar et al.; ...
65 views

How is it possible that Q-learning can learn a state-action value without taking into account the policy followed thereafter?

From my readings, I have been taught that the state-action value depends on the policy being followed. That seems logical because the expected return from actual actions will be different depending on ...
109 views

Deep Q-Learning "catastrophic drop" reasons?

I am implementing some "classical" papers in Model Free RL like DQN, Double DQN, and Double DQN with Prioritized Replay. Through the various models im running on ...
33 views

CartPoleV0 model is not getting trained in even after 1500+ episodes using deep Q-learning

I am new to deep Q learning and trying to train the open AI cartpole_V0 game using deep Q learning. Here is my code: ...
113 views

Update Rule with Deep Q-Learning (DQN) for 2-player games

I am wondering how to correctly implement the DQN algorithm for two-player games such as Tic Tac Toe and Connect 4. While my algorithm is mastering Tic Tac Toe relatively quickly, I cannot get great ...
57 views

Is a genetic algorithm efficient for a snake game?

I am working on a DIY project in which I want to be able to train a neural network to play Snake. Is a genetic algorithm an efficient way of training a network for this application? For a GA, what ...
24 views

Reward firstly increase, but after more episodes, start decrease, and weights diverges

I'm making a simple deep Q learning algorithm, with cartpole-v1 env. Like you can see in chart, after many episodes the reward decrease, some possible reasons? The exploration vs axplotation algorithm ...
16 views

DQN + HER, TD Error spiked then success rate plummets. What went wrong?

TL;DR: I trained a DQN + HER model using stable-baselines library for a custom environment. I noticed that in most runs, sometimes the TD-Error will spike and then the success rate of my model ...
48 views

Reinforcement learning for rearranging the mobile home screen icon layout: what inputs/states do I need to pass into the algorithm?

I have a problem where I need to rearrange a particular user's mobile home screen icon layout. Let's say that the social media app usage of a user is high compared to other app usage. So I need the ...
81 views

In DQN, would it be cheaper to have $N$ neural networks with a single real-valued output, one for each of the $N$ actions?

In the classical examples of deep q-learning, I often see neural networks in which the input represents the state of the agent, while the output is a tuple with all the values of $Q(s, a)$ predicted ...
55 views

How to scale all positive continuous reward?

My RL project has all positive continuous rewards for every step and the goal is to have the maximum cumulative reward (episodic reward). The problem is that the rewards are too close and all between ...
18 views

Training labels: integers or vectors?

I'm trying to implement Deep Q Learning using Tensorflow. The input is a vectorized representation of the state, and the output is a vector whose length is the number of possible actions. I've already ...
44 views

How to properly resume training of deep Q-learning network?

I'm currently training a deep q-learning network. Due to resource limitations, I am not able to train the model to the desired performance in one go. So what I'm doing now is training the model for a ...
33 views

Is is not possible to achieve average reward of more than 20-40 with simple Q-Learning

I have implemented the simple Q-Learning based solution for AI-gym's Cartpole-v0. However, despite changing hyper-parameters, and rechecking my code, I cannot get an average reward (N-running reward) ...
43 views

Given a sequence of states followed by the agent, is it possible to find the Q-value for a state-action pair not in this sequence?

Assume you are given a sequence of states followed by the agent, generated by a random policy, $[s_0, s_1, s_2, \dots, s_n]$. Furthermore, assume the MDP is fully observable and time is discrete. Is ...
14 views

Basic RL code for Production Line Optimization ( Need help )

I am working on how to optimize a Production line given 4 Workstations , 4 Employees(MA) . There are 5 products whose Workpackages need to be completed ( in order ) and these workpackages need to be ...
44 views

MicroPython MicroMLP: How do I reward the program based on state?

I have been trying to use MicroMLP to teach a small neural network to converge to correct results. Ultimately, I want to have three outputs, one which is high priority (...
84 views

Reinforcement Learning for an environment that is non-markovian

I am a beginner in the field of Reinforcement Learning with only a couple of months of experience being in the field. Soon, I will start working on a project where we want to optimize the production ...
94 views

How to avoid being stuck local optima in q-learning and q-network

When using Bellman equation to update q-table or train q-network to fit to greedy max values, the q-values very often get to the local optima and get stuck although randomisation rate ($\epsilon$) ...
58 views

Q-learning in gridworld with random board

I'm trying to use Q-learning in order to solve Wumpus world environment. Wumpus world is a toy problem on 4x4 gridworld. The agent starts in entry position of the cave, looks for gold (agent can sense ...
31 views

Compute state space from variables in Q-learning (RL)

I'm trying to use Q-learning, but I'm stuck because I don't know how to compute the state. Let's say, in my problem, there are the following variables, which I'm using to compute state: ...
24 views

Are there any known disadvantages of implementing vanilla Q-learning on a discretized-state space environment?

For an RL problem on a continuous state space, the states could be discretized into buckets and these buckets used in implementing the Q-table. I see that is what is done here. However, according to ...
131 views

What is a good convergence criterion for Q-learning in a stochastic environment?

I have a stochastic environment and I'm implementing a Q-table for the learning that happens on the environment. The code is shown below. In short, there are ten states (0, 1, 2,...,9), and three ...
86 views

How would I compute the optimal state-action value for a certain state and action?

I am currently trying to learn reinforcement learning and I started with the basic gridworld application. I tried Q-learning with the following parameters: Learning rate = 0.1 Discount factor = 0.95 ...
199 views

Why does Q-learning converge under 100% exploration rate?

I am working on this assignment where I made the agent learn state-action values (Q-values) with Q-learning and 100% exploration rate. The environment is the classic gridworld as shown in the ...
15 views

What is an appropriate stop criteria for training on a non-stationary environment in reinforcement learning?

I'm currently studying reinforcement learning (RL) and would like to understand non-stationary environments better. So for stationary environments, the Q-values of all state-action pairs converge ...
77 views

Why can we take the action $a$ from the next state $s'$ in the max part of the Q-learning update rule, if that action doesn't lead to any reward?

I'm using OpenAI's cartpole environment. First of all, is this environment not Markov? Knowing that, my main question concerns Q-learning and off-policy methods: For me, there is something weird in ...
26 views

How DynaQ behaves in stochastic world in comparison with other reinforcement learning algorithms?

I came across of implementations of a bunch of algorithms on stochastic windy gridworld. This is the graph comparing their performance: So clearly, it seems that DynaQ performs better than all other ...
129 views

What is the target output for updating a Deep Q Network

I'm trying to implement Deep Q-Learning for a pet problem having a continuous state space and discretized action space. The algorithm for table-based Q-Learning updates a single entry of the Q table - ...
57 views

What are the popular approaches to estimating the Q-function?

I need the q-value for my RL training, there are some approaches: Brute-force the action sequence (this won't work for long sequence) Use a classic algorithm to optimise and estimate (this ain't much ...
470 views

In Q-learning, wouldn't it be better to simply iterate through all possible states?

In Q-learning, all resources I've found seem to say that the algorithm to update the Q-table should start at some initial state, and pick actions (which are sometimes random) to explore the state ...
191 views

What is a "learned policy" in Q-learning?

I am completing an assignment at the moment. One of the assignment questions asks how you identified the learned policy and how you obtained it. The question is a reinforcement learning question, and ...
59 views

104 views

When calculating the cost in deep Q-learning, do we use both the input and target states?

I just finished Andrew Ngs's deep learning specialization, but RL was not covered, so I don't know the basics of RL. So, I have been having trouble understanding the cost function in deep Q-learning. ...