Questions tagged [q-learning]

For questions related to the Q-learning algorithm, which is a model-free and temporal-difference reinforcement learning algorithm that attempts to approximate the Q function, which is a function that, given a state s and an action a, returns a real number that represents the return (or value) of state s when action a is taken from s. Q-learning was introduced in the PhD thesis "Learning from Delayed Rewards" (1989) by Watkins.

Filter by
Sorted by
Tagged with
0 votes
0 answers
8 views

Q-learning: In tic-tac-toe, how to choose rewards values?

| I'm experimenting for first times in neural networks and Q-learning. In tic-tac-toe game, each move has 5 possible esits invalid move valid move, match is still open draft you lose you win ...
1 vote
2 answers
66 views

Does maximizing the value function and maximizing the state-action value function generate the same optimal policy?

In reinforcement learning, we define the optimal policy $\pi^*$ as the policy that maximizes the value of the state: $$ \pi_v^*=\underset{\pi}{\operatorname{argmax}} {V_{\pi}(s)} $$ In Q-learning, we ...
1 vote
1 answer
61 views

Why does Q-function training not query the Q-function value at unobserved states?

In the paper Conservative Q-Learning for Offline Reinforcement Learning, it is stated (section 3.1, page 3) that standard Q-function training does not query the Q-function value at unobserved states, ...
0 votes
2 answers
109 views

How can I model this problem of delivering assets by choosing a route with reinforcement learning?

I would like to build a model based on reinforcement learning (RL) for the following scenario Recommend the best route (of cities listed for a given country) that satisfies the required criteria (...
2 votes
1 answer
205 views

Why would SARSA diverge (but not Expected SARSA or Q-learning)?

In figure 6.3 (shown below) from Reinforcement Learning: An Introduction (second edition) by Sutton and Barto, SARSA is shown to perform worse asymptotically (after 100k episodes) than in the interim (...
2 votes
1 answer
207 views

How to implement RAM versions of Atari games

I have coded the breakout RAM version, but, unfortunately, its highest reward was 5. I trained it for about 2 hours and never reached a higher score. The code is huge, so I can't paste here, but, in ...
1 vote
1 answer
619 views

Reinforcement learning simple problem: agent not learning, wrong action

I am pretty new to RL and I am trying to code a simple RL task with pytorch. The goal/task is the following: The initial state is $t_o$ and the agent takes an action $\Delta_t$: $t_o +\Delta_t = t_1$. ...
1 vote
1 answer
240 views

How to create a Q-Learning agent when we have a matrix as an action space?

I have a 2-dimentional matrix as an action space, the rows being a resource to be allocated, and the columns are the users that we will allocate the resources to. (I built my own RL environment) The ...
3 votes
1 answer
576 views

Which Reinforcement Learning algorithms are efficient for episodic problems?

I have some episodic datasets extracted from a turn-based RTS game in which the current actions leading to the next state doesn’t determine the final solution/outcome of the episode. The learning is ...
0 votes
1 answer
100 views

How do I create an AI controller for Pacman?

How do I create an AI controller, which can play pacman - by taking in pixel values (or some other data by represents the state) which perhaps runs on a separate thread, which can control the game? It ...
0 votes
1 answer
75 views

Is the case of a big state space, should we use a softmax exploration policy rather than $\epsilon$-greedy for Q-Learning?

In Reinforcement Learning, epsilon-greedy policies are the most used exploration policies, but in case there is a big state space with impossible actions, wouldn't it be better to use soft-max ...
0 votes
1 answer
66 views

How to compare memory requirements for tabular Q-learning vs deep neural network?

I want to compare the space complexity/memory requirement of tabular Q-learning v.s. deep neural Q-network (DQN). I think DQN would be faster and Q-table has a disadvantage at large table sizes but ...
1 vote
1 answer
31 views

Is the described Q-table considered large?

I never saw any rule of thumb as to what size is said as large for a q-table but I have a Q-table with like 2500 entries. Is it considered large for a tabular approach? Anyone from experience can ...
0 votes
0 answers
30 views

What if we modify some Q-values while taking the action?

Just a passing thought about Q-learning. In the tabular Q-learning, what if I play around and modify any Q-values as I am using them to take actions? Would it be a violation of any (1) theoretical ...
0 votes
1 answer
45 views

Is it possible to add states to the Q-table after the game has started?

I would like to implement Q-learning in a game. Here is the board: It's a 2 player game. At each turn, each player can put a pawn on a line of their choice. They can't choose the column. The right ...
0 votes
1 answer
42 views

How to deal with changing rewards in Q-learning? DQN?

I read the working of Q-learning through a grid-based taxi routing wherein a taxi has to pick and drop off a passenger from source to destination. Likewise, I have a routing problem and hence, I tried ...
0 votes
0 answers
22 views

DDQN for Connect 4: Sudden explosion of Loss

I am trying to solve Connect 4 with DDQN through the self-play regime that was used for AlphaZero. That means, I let a student version play against a teacher version of itself and replace the teacher ...
4 votes
1 answer
758 views

When do SARSA and Q-Learning converge to optimal Q values?

Here's another interesting multiple-choice question that puzzles me a bit. In tabular MDPs, if using a decision policy that visits all states an infinite number of times, and in each state, randomly ...
2 votes
1 answer
84 views

Is it possible learning convergence is lost in Reinforcement Learning as the state space grows?

I am new in the AI field and I am trying to use Reinforcement Learning. Specifically, I am using tabular Q-Learning and SARSA algorithms to solve a sequential decision making problem. (I am using <...
0 votes
1 answer
53 views

Could Softmax Action Selection be useful to solve an episodic task with more than 100000 possible states and 2000 actions?

I am new in the field of RL. I am trying to use tabular methods, Q-Learning for solving a problem that takes a lot of time for computation, so I would like to know if there are more efficient methods ...
2 votes
1 answer
439 views

Is Q-learning only capable of learning a deterministic policy?

I was following a reinforcement learning course on coursera and in this video at 2:57 the instructor says Expected SARSA and SARSA both allow us to learn an optimal $\epsilon$-soft policy, but, Q-...
1 vote
1 answer
234 views

How to scale all positive continuous reward?

My RL project has all positive continuous rewards for every step and the goal is to have the maximum cumulative reward (episodic reward). The problem is that the rewards are too close and all between ...
0 votes
1 answer
25 views

Using reinforcement learning for human-robot interaction [closed]

I have a scenario where a user is wanting to exercise and improve over time. They attend around 10 exercise sessions, doing 20 repititions of an exercise each session. I want to develop a ...
0 votes
1 answer
47 views

Does $S_{t+1}$ denote the future information in Q-learning?

In Q-learning, $Q(S_t,a)$ is updated by the Bellman equation. $Q(S_t,a) = r + \max_{a'}(Q(S_{t+1},a'))$ where $S_{t+1}$ is the future state. Let's say $S$ denotes the stock price, does it mean we are ...
1 vote
1 answer
164 views

How to represent a state in a card game environment? (Wizard)

We are attempting to build an AI that manages to play the cardgame Wizard. So far er have a working network (based on the YOLO object-detection) that is abled to detect which cards are played. When ...
2 votes
1 answer
85 views

Adversarial Q Learning should use the same Q Table?

I'm creating a RF Q-Learning agent for a two player fully-observable board game and wondered, if I was to train the Q Table using adversarial training, should I let both 'players' use, and update, the ...
0 votes
1 answer
49 views

How to manage impossible actions? [closed]

I am using Q-learning in julia language. Because of the solver’s configuration, actions have to be defined as the whole action space and impossible actions have to be also considered. It means that I ...
2 votes
1 answer
110 views

What is the derivative of equation 1 in the paper "Conservative Q-Learning for Offline Reinforcement Learning"?

I am looking at the paper Conservative Q-Learning for Offline Reinforcement Learning, but I'm not sure how they proved theorem 3.1. Here is a screenshot of theorem 3.1. In the proof of theorem 3.1 ...
5 votes
1 answer
164 views

Why Q2 is a more or less independant estimate in Twin Delayed DDPG (TD3)?

Twin Delayed Deep Deterministic (TD3) policy gradient is inspired by both double Q-learning and double DQN. In double Q-learning, I understand that Q1 and Q2 are independent because they are trained ...
0 votes
1 answer
706 views

Problem over DQN Algorithm not converging on snake

I'm using a DQN Algorithm to play Snake. The input of the neural network is a stack of 4 images taken from the games 80x80. The output is an array of 4 values, one for every direction. The problem ...
1 vote
1 answer
43 views

Can directly using expert policy in epsilon-greedy speed-up Q-learning?

In deep Q-learning we typically use epsilon-greedy policy during training. We choose a random action for a certain probability $\epsilon$, and choose the action that maximize the current Q-value ...
1 vote
1 answer
39 views

Is using Monte-Carlo estimate of returns in Deep Q Learning possible?

In all the tutorials of deep Q-learning (using neural networks) I have read so far, the state-action value function $Q(s,a)$ is learned by temporal difference learning. However, in policy gradient ...
0 votes
1 answer
160 views

When calculating the max in DQN, do I have to calculate the Q for every possible action for a particular state?

I'm trying to implement the DQN paper using python/pytorch for my needs (https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf). I'm studying the main algorithm: I am a bit confused about the $\gamma* \max ...
4 votes
3 answers
699 views

Is there an upper limit to the maximum cumulative reward in a deep reinforcement learning problem?

Is there an upper limit to the maximum cumulative reward in a deep reinforcement learning problem? For example, you want to train a DQN agent in an environment, and you want to know what the highest ...
4 votes
1 answer
309 views

How do updates in SARSA and Q-learning differ in code?

The update rules for Q-learning and SARSA each are as follows: Q Learning: $$Q(s_t,a_t)←Q(s_t,a_t)+α[r_{t+1}+γ\max_{a'}Q(s_{t+1},a')−Q(s_t,a_t)]$$ SARSA: $$Q(s_t,a_t)←Q(s_t,a_t)+α[r_{t+1}+γQ(s_{t+...
3 votes
1 answer
192 views

Given these two reward functions, what can we say about the optimal Q-values, in self-play tic-tac-toe?

This corresponds to Exercise 1.1 of Sutton & Barto's book (2nd edition), and a discussion followed from this answer. Consider the following two reward functions Win = +1, Draw = 0, Loss = -1 Win =...
2 votes
1 answer
1k views

Why can I still easily beat my Q-learning agent that was trained against another Q-learning agent to play tic tac toe?

I implemented the Q-learning algorithm to play tic-tac-toe. The AI plays against the same algorithm, but they don't share the same Q matrix. After 200,000 games, I still beat the AI very easily and it'...
0 votes
1 answer
1k views

Why isn't my DQN implementation working properly?

I'm trying to build a DQN to replicate the DeepMind results. I'm doing with a simple DQN for the moment, but it isn't learning properly: after +5000 episodes, it couldn't get more than 9-10 points. ...
2 votes
1 answer
81 views

Deep Q-Learning Model Effectiveness Improves then Crashes

I am implementing a Deep Q-Learning Algorithm. The model appears to improve but after awhile it just crashes and does just as well as if an agent was making random decisions. Shouldn't the behavior ...
-1 votes
1 answer
266 views

what does the OpenAI ALE/Breakout-RAM-V5 observation return [closed]

I haven't been able to understand the output that OpenAI gym return for observation from this snippet ...
0 votes
0 answers
24 views

Are temperature schedules used in Boltzmann Q-learning?

In Q-learning with the epsilon greedy method, I have seen implementations for training an agent that provide the possibility to pass epsilon schedules. Basically you can keep the epsilon constant or ...
0 votes
2 answers
335 views

Reinforcement Learning for an environment that is non-markovian [closed]

I will start working on a project where we want to optimize the production of a chemical unit through reinforcement learning approach. From the SME's, we already obtained a simulator code that can ...
5 votes
2 answers
4k views

How should I model all available actions of a chess game in deep Q-learning?

I just read about deep Q-learning, which is using a neural network for the value function instead of a table. I saw the example here: Using Keras and Deep Q-Network to Play FlappyBird and he used a ...
1 vote
1 answer
138 views

Q table not converging for an arbitrary experiment

This is an experiment in order to understand the working of Q table and Q learning. I have the states as states = [0,1,2,3] I have an arbitrary value for each ...
3 votes
1 answer
2k views

Is there an advantage in decaying $\epsilon$ during Q-Learning?

If the agent is following an $\epsilon$-greedy policy derived from Q, is there any advantage to decaying $\epsilon$ even though $\epsilon$ decay is not required for convergence?
1 vote
1 answer
157 views

In the definition of the state-action value function, what is the random variable we take the expectation of?

I know that $$\mathbb{E}[g(X) \mid A] = \sum\limits_{x} g(x) p_{X \mid A}(x)$$ for any random variable $X$. Now, consider the following expression. $$\mathbb{E}_{\pi} \left[ \sum \limits_{k=0}^{\infty}...
7 votes
2 answers
406 views

What are some online courses for deep reinforcement learning?

What are some (good) online courses for deep reinforcement learning? I would like the course to be both programming and theoretical. I really liked David Silver's course, but the course dates from ...
1 vote
0 answers
43 views

Can multiple reinforcement algorithms be applied to the same system?

Can a system, for instance, a robotic vehicle, be controlled by more than one reinforcement learning algorithm. I intend to use one to address collision avoidance whereas the other to tackle ...
4 votes
2 answers
258 views

Is the optimal policy the one with the highest accumulative reward (Q-Learning vs SARSA)?

I was looking at the following diagram, The reward obtained with SARSA is higher. However, the path that Q learning chooses is eventually the optimal one, isn't it? Why is the SARSA reward higher if ...
4 votes
1 answer
179 views

Can Google's patented ML algorithms be used commercially?

I just find that Google patents some of the widely used machine learning algorithms. For example: System and method for addressing overfitting in a neural network (Dropout?) Processing images using ...

1
2 3 4 5
7