Questions tagged [q-learning]

For questions related to the Q-learning algorithm, which is a model-free and temporal-difference reinforcement learning algorithm that attempts to approximate the Q function, which is a function that, given a state s and an action a, returns a real number that represents the return (or value) of state s when action a is taken from s. Q-learning was introduced in the PhD thesis "Learning from Delayed Rewards" (1989) by Watkins.

Filter by
Sorted by
Tagged with
0
votes
0answers
31 views

Python libraries for solving reinforcement learning problems implemented in OpenAI gym [closed]

I have implemented a reinforcement problem with Open AI gym in Python and I would like to solve it using different algorithms like Deep-Q-Learning and SARSA (any maybe others). My question is which ...
2
votes
2answers
159 views

Is the optimal policy the one with the highest accumulative reward (Q-Learning vs SARSA)?

I was looking at the following diagram, The reward obtained with SARSA is higher. However, the path that Q learning chooses is eventually the optimal one, isn't it? Why is the SARSA reward higher if ...
1
vote
0answers
29 views

How to use Actor-Critic RL with a categorical, state-dependent action space?

I have a problem where the agent is given an embedding vector to represent the state. Then it is also given a set of possible actions in the environment, let's say that the actions are each ...
1
vote
0answers
27 views

What is the difference between the $Q_a$ calculated to update delta and those to select next action in the exploitation phase?

As the title suggests, I have a doubt about the computation of the $Q_a$ used to update the delta and the $Q_a$ used to select the next action in the exploitation phase, as shown below (source of ...
2
votes
0answers
44 views

Watkins' Q(λ) with function approximation: why is gradient not considered when updating eligibility traces for the exploitation phase?

I'm implementing the Watkins' Q(λ) algorithm with function approximation (in 2nd edition of Sutton & Barto). I am very confused about updating the eligibility traces because, at the beginning of ...
0
votes
1answer
52 views

How is $Q(s', a')$ calculated in SARSA and Q-Learning?

I have a question about how to update the Q-function in Q-learning and SARSA. Here (What are the differences between SARSA and Q-learning?) the following updating formulas are given: Q-Learning $$Q(s,...
0
votes
0answers
25 views

Should you clip Q values if they start to grow indefinitely?

I am training the SAC algorithm for an environment where the rewards are small as shown below and the episode length is 84. I have a problem with the Q values that grow with each step. The following ...
1
vote
1answer
72 views

What is meant by "two action selections" in SARSA?

I have some difficulties understanding the difference between Q-learning and SARSA. Here (What are the differences between SARSA and Q-learning?) the following updating formulas are given: Q-Learning $...
2
votes
0answers
26 views

Reduction of state space of the game Connect Four to apply RL algorithms SARSA and Q-Learning

I would like to implement the reinforcement learning algorithms SARSA and Q-Learning for the board game Connect Four. I am familiar with the algorithms and know about their limitations regarding large ...
1
vote
0answers
57 views

Can Q-learning be used for my scenario, and how might I do so?

I have already asked 2-3 general questions w.r.t Q learning and now I am asking a scenario specific one. I will try to be concise and understandable. I really really need help. Scenario: I have a ...
0
votes
0answers
26 views

Alternatives to neural networks for function approximation in Q learning?

I want to know if there is anything other than neural networks (or Deep NNs) that I can effectively use to perform function approximation? I am asking this w.r.t to the use of approximators in Q ...
0
votes
0answers
58 views

What do we actually 'approximate' when dealing with large state spaces in Q-learning?

I realized that my state space is very large in size. I had planned to use tabular Q-learning (Bellman equation to update the $Q(s, a)$ after each action taken). But this 'large space' realization has ...
0
votes
0answers
32 views

How to deal with Q-learning having low variance in predicted Q-values?

I have a neural network that takes the state (which contains a lot of data), and the possible action (which is very little data), and predicts the Q-value of the action. I am double Q-learning. I've ...
1
vote
1answer
54 views

Why would SARSA diverge (but not Expected SARSA or Q-learning)?

In figure 6.3 (shown below) from Reinforcement Learning: An Introduction (second edition) by Sutton and Barto, SARSA is shown to perform worse asymptotically (after 100k episodes) than in the interim (...
1
vote
0answers
34 views

Multi-armed Bandit in optimization on graph edges selection

I have the problem, which I described below. I wonder if there exists a class of multi-armed bandit approaches that is related to it. I am working on computer networking optimization. In the simplest ...
1
vote
0answers
20 views

Transferring a Q-learning policy to larger instances

How do I best transfer and fine-tune a Q-learning policy that was trained on small instances to large instances? Some more details on the problem: I am currently trying to derive a decision policy for ...
3
votes
2answers
340 views

When to use Value Iteration vs. Policy Iteration

Both value iteration and policy iteration are General Policy Iteration (GPI) algorithms. However, they differ in the mechanics of their updates. Policy Iteration seeks to first find a completed ...
2
votes
1answer
55 views

How to encourage the reinforcement-learning agent to reach the goal as quickly as possible, and what's the effect of discount factor?

I am trying to use reinforcement learning to solve a task and compare its performance to humans. The task is to find a single target in a fixed number of locations. At each step, the agent will pick ...
0
votes
0answers
29 views

How do I solve a minimization problem with Q-learning learning?

I am trying to learn reinforcement learning by myself and so I have a lot of doubts. In particular, I am investigating how to use Q-learning in order to solve minimization problems. For example, ...
1
vote
1answer
60 views

What is the difference between gradient decent in neural networks and temporal difference in reinforcement learning?

I am studying Q-learning in reinforcement learning. My question is about the Bellman equation. In Q-learning, the Bellman equation is often introduced as follows. \begin{align} Q_{new}(s,a) &= Q_{...
1
vote
1answer
57 views

How to handle invalid actions for next state in Q-learning loss

I am implementing an RL application in an environment with illegal moves. For handling the illegal moves, I am currently just picking an action as the maximum Q-value from the set of legal Q-values. ...
1
vote
1answer
76 views

What is the derivative of equation 1 in the paper "Conservative Q-Learning for Offline Reinforcement Learning"?

I am looking at the paper Conservative Q-Learning for Offline Reinforcement Learning, but I'm not sure how they proved theorem 3.1. Here is a screenshot of theorem 3.1. In the proof of theorem 3.1 ...
4
votes
1answer
84 views

How to approach a blackjack-like card game with the possibility of cards being counted?

Consider a single-player card game which shares many characteristics to "unprofessional" (not being played in casino, refer point 2) Blackjack, i.e.: You're playing against a dealer with ...
1
vote
1answer
33 views

Why does Q-function training not query the Q-function value at unobserved states?

In the paper Conservative Q-Learning for Offline Reinforcement Learning, it is stated (section 3.1, page 3) that standard Q-function training does not query the Q-function value at unobserved states, ...
3
votes
1answer
116 views

Is it really hard to learn in a stochastic environment?

I understand that a stochastic environment is one that does not always lead you to the desired state by giving a particular action $a$ (But the probability to change to a not desire state is fixed, ...
1
vote
1answer
65 views

When to activate batch normalization and dropout in deep Q-learning?

In the vanilla version of deep Q-learning, there are three places where the Q-network is queried: When exploring. When training: a. When calculating the optimal value of the state reached by an ...
1
vote
1answer
36 views

In deep reinforcement learning, what is this model with state as input and value as output?

I was looking at this implementation for creating an agent for playing Tetris using DeepRL. This model uses "a state based on the statistics of the board after a potential action. All predictions ...
0
votes
0answers
39 views

Defining states and possible actions in Q learning

I am trying to define the number of states and possible actions for a reinforcement learning problem that I want to solve with Q-learning, but I am a bit confused, as I'm totally new to reinforcement ...
0
votes
1answer
36 views

Where can I find the original conference paper that introduced Q-learning and Deep Q-Learning?

I tried searching a lot, but I could neither find the paper that introduced Q-Learning nor the paper that introduced Deep Q Learning. If anyone knows anything about it please do tell me.
1
vote
1answer
60 views

Why do I get bad results no matter my neural network function approximator for parametrized Q-learning implementation for Contextual Bandits?

I'd like to ask you why, no matter my neural network function approximator for parametrized Q-learning implementation for a Contextual Bandits environment, I'm getting bad results. I don't know if it'...
1
vote
1answer
61 views

Which policy has to be followed by a player while construction of its own Q-table?

Consider the scenario, where there are two players. One of the players perform the action randomly, whereas I want second player as a Q-player. I mean, the player selects a best action from the Q-...
0
votes
1answer
78 views

Which policy do I need to use in updating Q function?

Policy function can be of two types: deterministic policy and stochastic policy. Deterministic policy is of the form $\pi : S \rightarrow A$ Stochastic policy is defined using conditional probability ...
1
vote
1answer
69 views

In addition to the reward function, which other functions do I need to implement Q-learning?

In general, $Q$ function is defined as $$Q : S \times A \rightarrow \mathbb{R}$$ $$Q(s_t,a_t) = Q(s_t,a_t) + \alpha[r_{t+1} + \gamma \max\limits_{a} Q(s_{t+1},a) - Q(s_t,a_t)] $$ $\alpha$ and $\gamma$...
1
vote
0answers
22 views

Is there any work that applies the approach in "Finite-Sample Convergence Rates for Q-Learning and Indirect Algorithms" to standard Q-learning?

I am trying to mathematically characterize the finite sample convergence rates for Q-learning. To this end, I have read the following papers Learning rates for Q-learning, by Eyal Even-Dar et al.; ...
2
votes
2answers
69 views

How is it possible that Q-learning can learn a state-action value without taking into account the policy followed thereafter?

From my readings, I have been taught that the state-action value depends on the policy being followed. That seems logical because the expected return from actual actions will be different depending on ...
4
votes
1answer
269 views

Deep Q-Learning "catastrophic drop" reasons?

I am implementing some "classical" papers in Model Free RL like DQN, Double DQN, and Double DQN with Prioritized Replay. Through the various models im running on ...
0
votes
0answers
39 views

CartPoleV0 model is not getting trained in even after 1500+ episodes using deep Q-learning

I am new to deep Q learning and trying to train the open AI cartpole_V0 game using deep Q learning. Here is my code: ...
2
votes
0answers
150 views

Update Rule with Deep Q-Learning (DQN) for 2-player games

I am wondering how to correctly implement the DQN algorithm for two-player games such as Tic Tac Toe and Connect 4. While my algorithm is mastering Tic Tac Toe relatively quickly, I cannot get great ...
0
votes
1answer
68 views

Is a genetic algorithm efficient for a snake game?

I am working on a DIY project in which I want to be able to train a neural network to play Snake. Is a genetic algorithm an efficient way of training a network for this application? For a GA, what ...
0
votes
0answers
39 views

Reward firstly increase, but after more episodes, start decrease, and weights diverges

I'm making a simple deep Q learning algorithm, with cartpole-v1 env. Like you can see in chart, after many episodes the reward decrease, some possible reasons? The exploration vs axplotation algorithm ...
0
votes
0answers
29 views

DQN + HER, TD Error spiked then success rate plummets. What went wrong?

TL;DR: I trained a DQN + HER model using stable-baselines library for a custom environment. I noticed that in most runs, sometimes the TD-Error will spike and then the success rate of my model ...
0
votes
0answers
52 views

Reinforcement learning for rearranging the mobile home screen icon layout: what inputs/states do I need to pass into the algorithm?

I have a problem where I need to rearrange a particular user's mobile home screen icon layout. Let's say that the social media app usage of a user is high compared to other app usage. So I need the ...
4
votes
1answer
86 views

In DQN, would it be cheaper to have $N$ neural networks with a single real-valued output, one for each of the $N$ actions?

In the classical examples of deep q-learning, I often see neural networks in which the input represents the state of the agent, while the output is a tuple with all the values of $Q(s, a)$ predicted ...
1
vote
1answer
84 views

How to scale all positive continuous reward?

My RL project has all positive continuous rewards for every step and the goal is to have the maximum cumulative reward (episodic reward). The problem is that the rewards are too close and all between ...
0
votes
0answers
29 views

What is it called when every entry of the output vector has a meaning?

I'm trying to implement Deep Q Learning using Tensorflow. The input is a vectorized representation of the state, and the output is a vector whose length is the number of possible actions. I've already ...
1
vote
1answer
50 views

How to properly resume training of deep Q-learning network?

I'm currently training a deep q-learning network. Due to resource limitations, I am not able to train the model to the desired performance in one go. So what I'm doing now is training the model for a ...
0
votes
1answer
39 views

Is is not possible to achieve average reward of more than 20-40 with simple Q-Learning

I have implemented the simple Q-Learning based solution for AI-gym's Cartpole-v0. However, despite changing hyper-parameters, and rechecking my code, I cannot get an average reward (N-running reward) ...
0
votes
1answer
65 views

Given a sequence of states followed by the agent, is it possible to find the Q-value for a state-action pair not in this sequence?

Assume you are given a sequence of states followed by the agent, generated by a random policy, $[s_0, s_1, s_2, \dots, s_n]$. Furthermore, assume the MDP is fully observable and time is discrete. Is ...
0
votes
0answers
15 views

Basic RL code for Production Line Optimization ( Need help )

I am working on how to optimize a Production line given 4 Workstations , 4 Employees(MA) . There are 5 products whose Workpackages need to be completed ( in order ) and these workpackages need to be ...
0
votes
0answers
60 views

MicroPython MicroMLP: How do I reward the program based on state?

I have been trying to use MicroMLP to teach a small neural network to converge to correct results. Ultimately, I want to have three outputs, one which is high priority (...

1
2 3 4 5
7