Questions tagged [q-learning]

For questions related to the Q-learning algorithm, which is a model-free and temporal-difference reinforcement learning algorithm that attempts to approximate the Q function, which is a function that, given a state s and an action a, returns a real number that represents the return (or value) of state s when action a is taken from s. Q-learning was introduced in the PhD thesis "Learning from Delayed Rewards" (1989) by Watkins.

Filter by
Sorted by
Tagged with
0
votes
0answers
38 views

Understanding V- and Q-functions

Assume the existence of a Markov Decision Process consisting of: State space $S$ Action space $A$ Transition model $T: S \times A \times S \to [0,1]$ Reward function $R: S \times A \times S \to \...
2
votes
0answers
24 views

Evaluation a policy learned using Q - learning

I have been reading literature on reinforcement learning in healthcare. I am slightly confused between the policy evaluation for both SARSA and Q-learning. To my knowledge, I believe that SARSA is ...
2
votes
1answer
69 views

Taxi-v3 help. What is meant exactly by convergence of the algo, the highest reward and optimal action for every state?

I started learning about Q table from this blog post Introduction to reinforcement learning and OpenAI Gym, by Justin Francis, which has a line as below - After so many episodes, the algorithm will ...
1
vote
0answers
39 views

How to represent a state in a card game environment? (Wizard)

We are attempting to build an AI that manages to play the cardgame Wizard. So far er have a working network (based on the YOLO object-detection) that is abled to detect which cards are played. When ...
1
vote
1answer
30 views

Q-learning problem wrong policy

I am coding out a simple 4x4 grid game whereby the agent starts at a particular state and his aim is to reach the terminal state. The agent is supposed to avoid traps along the way and reach the end ...
2
votes
1answer
38 views

Do we need an explicit policy to sample $A'$ in order to compute the target in SARSA or Q-learning?

I would much appreciate if you could point me in the right direction regarding this question about targets for SARSA and Q-learning (notation: $S$ is the current state, $A$ is the current action, $R$ ...
1
vote
1answer
494 views

What is the difference between the epsilon greedy and softmax policies?

Could someone explain to me which is the key difference between the epsilon greedy policy and the softmax policy? In particular in the contest of SARSA and Q-Learning algorithms. I understood the main ...
1
vote
0answers
91 views

Expected SARSA, SARSA and Q-learning

I would much appreciate if you could point me in the right direction regarding this question about targets for approximate ...
3
votes
1answer
74 views

Why Monte Carlo epsilon-soft approach cannot compute $\max Q(s,a)$?

I am new to Reinforcement learning and am currently reading up on the estimation of Q $\pi(s, a)$ values using MC epsilon-soft approach and chanced upon this algorithm. The link to the algorithm is ...
4
votes
1answer
51 views

How can a DQN backpropagate its loss?

I'm currently trying to take the next step in deep learning. I managed so far to write my own basic feed-forward network in python without any frameworks (just numpy and pandas), so I think I ...
3
votes
1answer
55 views

Is the Q value updated at every episode?

I trying to understand the Bellman equation for updating the Q table values. The concept of initially updating the value is clear to me. What is unclear is the subsequent updates to the value. Is the ...
1
vote
0answers
22 views

N-tuple based tic tac toe diverges in temporal difference learning

I have n-tuple based tic tac toe. I already have perfect minimax player and perfectly trained table-based player. My n-tuple network consists of 8 different rows of 3 of the board as triplets having ...
2
votes
0answers
51 views

Q-learning: How to include a terminal state in updating rule?

I use Q-learning in order to determine the optimal path of an agent. I know in advance that my path is composed of exactly 3 states (so after 3 states I reach a terminal state). I would like to know ...
1
vote
0answers
47 views

Reinforcement learning for a 2D game involving two players

I'd like to create an AI for a 2D game involving two players fighting against each other. The map look something like this (The map is a NxN array somehow randomly generated): Basically the players ...
1
vote
1answer
31 views

Is the following the correct implementation of the Q learning algorithm for a neural network?

I just wanted to confirm that my understanding of Q learning was correct (with respect to a neural network). ...
1
vote
0answers
72 views

Why isn't this deep Q network agent for the snake game learning?

I wanted to combine this snake game with this DQN implementation I found in this article. First, I tried to change the NN's input layer to a 400 input. The game has a field of 20 times 20, so I ...
3
votes
1answer
266 views

Does using the softmax function in Q learning not defeat the purpose of Q learning?

It is my understanding that, in Q-learning, you are trying to mimic the optimal $Q$ function $Q*$, where $Q*$ is a measure of the predicted reward received from taking action $a$ at state $s$ so that ...
2
votes
0answers
34 views

How would one develop an action space for a game that is proprietary?

I'm currently trying to develop an RL that will teach itself to play the popular fighting game "Tekken 7". I initially had the idea of teaching it to play generally- against actual opponents with ...
3
votes
1answer
64 views

Can Google's patented ML algorithms be used commercially?

I just find that Google patents some of the widely used machine learning algorithms. For example: System and method for addressing overfitting in a neural network (Dropout?) Processing images using ...
2
votes
0answers
29 views

What is the complexity of policy gradient algorithms compared to discrete action space algorithms?

I am using a policy gradient algorithm (actor-critic) for wireless networks. The policy gradient-based algorithm helps because it considers continuous action space. But how much does a policy ...
2
votes
1answer
31 views

Why feed actions in later layer in Q network?

I read the DDPG paper, in which the authors state that the actions are fed only later to their Q network: Actions were not included until the 2nd hidden layer of Q. (Sec 7, Experiment Details) So ...
1
vote
0answers
40 views

Did I understand deep Q leaning right? (Implementation)

Gday guys, so I tried to implement my own enviroment and agent in order to fully understand DQNs. The enviroment is a dungeon with five states. actionspace = 2 statespace = 5 !!!Action a0 is ...
2
votes
1answer
247 views

Q Learning for FrozenLake environment not converging to V* values from Value Iteration

I am trying to learn tabular Q learning, value iteration using the classical algorithms (no neural networks) by using a table of states and actions. I was trying it out on FrozenLake environment in ...
0
votes
0answers
78 views

Model-based Reinforcement Learning algorithm for real-time robotics task

I'm quite a newbie when it comes to practically working with Deep Learning techniques, although I studied them quite a lot theoretically in the last months. However, now I'm facing my first practical ...
1
vote
0answers
15 views

TD losses are descreasing, but also rewards are decreasing, increasing sigma

I'm using Q-learning with some extensions such as noisy linear layers, n-steps and double DQN. The training, however, isn't that successful, my rewards are descreasing over time after a steep ...
2
votes
1answer
164 views

What happens to the optimal value function if the reward is multiplied by a constant?

What happens to the optimal action-value function, $q_*$ if the reward is multiplied by a constant $c$? Is the optimal action-value function also multiplied by such a constant?
6
votes
2answers
194 views

What does the symbol $\mathbb E$ mean in these equations?

I came across some papers that use $\mathbb E$ in equations, in particular, this paper: https://arxiv.org/pdf/1511.06581.pdf. Here is some equations from the paper that uses it: $Q^\pi \left(s,a \...
5
votes
1answer
69 views

Can exogenous variables be state features in reinforcement learning?

I have a question about state representation of Q-learning or DQN algorithm. I'm still a beginner of RL, so I'm not sure that is it suitable to take exogenous variables as state features. For example,...
2
votes
3answers
151 views

Reinforcement learning: How to deal with illegal actions?

I'm a beginner of RL and currently trying to make DQN agent that can act optimally in a simple situation. In the situation agent should decide at what rate to charge or discharge the electrical ...
4
votes
1answer
124 views

Is tabular Q-learning considered interpretable?

I am working on a research project in a domain where other related works have always resorted to deep Q-learning. The motivation of my research stems from the fact that the domain has an inherent ...
1
vote
1answer
81 views

Deep Q Learning for Simple Game Not Effective

This is a follow-up question about one I asked earlier. The first question is here. Basically, I have a game where a paddle moves left and right to catch as much "food" as possible. Some food is good (...
1
vote
1answer
97 views

Deep Q Learning Algorithm for Simple Python Game makes player stuck

I made a simple Python game. A screenshot is below: Basically, a paddle moves left and right catching particles. Some make you lose points while others make you gains points. This is my first Deep Q ...
1
vote
0answers
42 views

Unique game problem (ML, DP, PP etc)

Looking for a solution to my below game problem. I believe it to require some sort of reinforcement learning, dynamic programming, or probabilistic programming solution, but am unsure... This is my ...
2
votes
1answer
37 views

Will the target network, which is less trained than the normal network, output inferior estimates?

I'm having some trouble understanding some parts of the usage of target networks. I get that having the same network predict the state/action/advantage values for both the current networks can lead ...
3
votes
0answers
85 views

How can I use Q-learning for inventory decision making?

I am trying to model operational decisions in inventory control. The control policy is base stock with a fixed stock level of $S$. That is replenishment orders are placed for every demand arrival to ...
1
vote
1answer
56 views

Probabilistic action selection in pursuit algorithm

In the Pursuit algorithm (to balance exploration and exploitation), the greedy action has a probability say $p_1$ (updated every episode) of being selected, while the rest have a probability $p_2$ (...
3
votes
1answer
105 views

Is it possible to have a dynamic $Q$-function?

I am trying to use Q-learning for energy optimization. I only wish to have states that will be visited by the learning agent, and, for each state, I have a function that generates possible actions, so ...
1
vote
0answers
33 views

Can multiple reinforcement algorithms be applied to the same system?

Can a system, for instance robotic vehicle, be controlled by more than one reinforcement learning algorithm. I intend to use one to address collision avoidance whereas the other to tackle autonomous ...
2
votes
1answer
42 views

If deep Q learning involves adjusting the value function for a specific policy, then how do I choose the right policy?

I wrote a simple implementation of Flappy Bird in Python, and now I'm trying to train an agent to play it at a reasonable skill level using TFLearn. I feed the network an input vector of size 4: ...
2
votes
0answers
80 views

How does Friend-or-Foe Q-learning intuitively work?

I read about Q-Learning and was reading about multi-agent environments. I tried to read the paper Friend-or-Foe Q-learning, but could not understand anything, except for a very vague idea. What does ...
1
vote
0answers
126 views

Why epsilon-greedy hyperparameter is annealed smoothly?

Regarding of DQN, or DQRNN, (reinforcement learning) To me, RL is a process that can be divided into 2 stages: Exploring wide range of paths (acting randomly) Refining the current optimal paths (...
6
votes
1answer
261 views

Does AlphaZero use Q-Learning?

I was reading the AlphaZero paper Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm, and it seems they don't mention Q-Learning anywhere. So does AZ use Q-...
3
votes
2answers
97 views

What is the difference between return and expected return?

At a time step $t$, for a state $S_{t}$, the return is defined as the discounted cumulative reward from that time step $t$. If an agent is following a policy (which in itself is a probability ...
2
votes
2answers
93 views

How to stop evaluation phase in reinforcement learning with epsilon-greedy Monte Carlo agent?

I have implemented an epsilon-greedy Monte Carlo reinforcement learning agent like suggested in Sutton and Barto's RL book (page 101). As far as I understood epsilon-greedy agents so far, the ...
-2
votes
1answer
279 views

Problem over DQN Algorithm not converging on snake

I'm using a DQN Algorithm to play Snake. The input of the neural network is a stack of 4 images taken from the games 80x80. The output is an array of 4 values, one for every direction. The problem ...
2
votes
1answer
159 views

Deep Reinforcement Learning: Rewards suddenly dip down

I am working on a deep reinforcement learning problem. The policy network has the same architecture as the one Deepmind published in 'Playing Atari with Deep Reinforcement Learning'. I am also using ...
3
votes
1answer
39 views

How do I represent a multi-dimensional state using a neural network?

I have a set of 15 unique playing cards from a deck of 52 playing cards. A given state is represented by the respective card values in the set of 15 cards, where the card value is a prime number ...
4
votes
1answer
107 views

Q-learning, am I interpreting correctly $Q(s,a) = r + \gamma \max_{a'} Q(s',a')$?

Ok, due to previous question I was pointed to use reinfrocement learning. So far what I understood from random websites is the following: there is a Q(s,a) function involved I can assume my neural ...
1
vote
1answer
307 views

Choice of inputs features for Snake game

I am designing a neural network using Deep Q-Learning, which teaches an agent how to play Snake (The classic Nokia game from the 90'ies). The goal of the game is to navigate the snake on a playing ...
1
vote
0answers
30 views

High variance in performance of q-learning agents trained with same parameters

I am training an agent to play a simple game using double deep q learning. However, the variance in agent performance is very high, even for agents trained with same model parameters. For example, I ...