Questions tagged [q-learning]

For questions related to the Q-learning algorithm, which is a model-free and temporal-difference reinforcement learning algorithm that attempts to approximate the Q function, which is a function that, given a state s and an action a, returns a real number that represents the return (or value) of state s when action a is taken from s. Q-learning was introduced in the PhD thesis "Learning from Delayed Rewards" (1989) by Watkins.

Filter by
Sorted by
Tagged with
1
vote
0answers
35 views

Expected SARSA, SARSA and Q-learning

I would much appreciate if you could point me in the right direction regarding this question about targets for approximate ...
3
votes
1answer
55 views

Why Monte Carlo epsilon-soft approach cannot compute $\max Q(s,a)$?

I am new to Reinforcement learning and am currently reading up on the estimation of Q $\pi(s, a)$ values using MC epsilon-soft approach and chanced upon this algorithm. The link to the algorithm is ...
3
votes
1answer
33 views

How can a DQN backpropagate its loss?

I'm currently trying to take the next step in deep learning. I managed so far to write my own basic feed-forward network in python without any frameworks (just numpy and pandas), so I think I ...
3
votes
1answer
45 views

Is the Q value updated at every episode?

I trying to understand the Bellman equation for updating the Q table values. The concept of initially updating the value is clear to me. What is unclear is the subsequent updates to the value. Is the ...
1
vote
0answers
13 views

N-tuple based tic tac toe diverges in temporal difference learning

I have n-tuple based tic tac toe. I already have perfect minimax player and perfectly trained table-based player. My n-tuple network consists of 8 different rows of 3 of the board as triplets having ...
2
votes
0answers
33 views

Q-learning: How to include a terminal state in updating rule?

I use Q-learning in order to determine the optimal path of an agent. I know in advance that my path is composed of exactly 3 states (so after 3 states I reach a terminal state). I would like to know ...
1
vote
0answers
45 views

Reinforcement learning for a 2D game involving two players

I'd like to create an AI for a 2D game involving two players fighting against each other. The map look something like this (The map is a NxN array somehow randomly generated): Basically the players ...
1
vote
1answer
29 views

Is the following the correct implementation of the Q learning algorithm for a neural network?

I just wanted to confirm that my understanding of Q learning was correct (with respect to a neural network). ...
1
vote
0answers
37 views

Why isn't this deep Q network agent for the snake game learning?

I wanted to combine this snake game with this DQN implementation I found in this article. First, I tried to change the NN's input layer to a 400 input. The game has a field of 20 times 20, so I ...
3
votes
1answer
41 views

Does using the softmax function in Q learning not defeat the purpose of Q learning?

It is my understanding that, in Q-learning, you are trying to mimic the optimal $Q$ function $Q*$, where $Q*$ is a measure of the predicted reward received from taking action $a$ at state $s$ so that ...
2
votes
0answers
25 views

How would one develop an action space for a game that is proprietary?

I'm currently trying to develop an RL that will teach itself to play the popular fighting game "Tekken 7". I initially had the idea of teaching it to play generally- against actual opponents with ...
3
votes
1answer
61 views

Can Google's patented ML algorithms be used commercially?

I just find that Google patents some of the widely used machine learning algorithms. For example: System and method for addressing overfitting in a neural network (Dropout?) Processing images using ...
1
vote
0answers
25 views

What is the complexity of policy gradient algorithms compared to discrete action space algorithms?

I am using a policy gradient algorithm (actor-critic) for wireless networks. The policy gradient-based algorithm helps because it considers continuous action space. But how much does a policy ...
2
votes
1answer
27 views

Why feed actions in later layer in Q network?

I read the DDPG paper, in which the authors state that the actions are fed only later to their Q network: Actions were not included until the 2nd hidden layer of Q. (Sec 7, Experiment Details) So ...
1
vote
0answers
37 views

Did I understand deep Q leaning right? (Implementation)

Gday guys, so I tried to implement my own enviroment and agent in order to fully understand DQNs. The enviroment is a dungeon with five states. actionspace = 2 statespace = 5 !!!Action a0 is ...
2
votes
1answer
55 views

Q Learning for FrozenLake environment not converging to V* values from Value Iteration

I am trying to learn tabular Q learning, value iteration using the classical algorithms (no neural networks) by using a table of states and actions. I was trying it out on FrozenLake environment in ...
0
votes
1answer
63 views

Model-based Reinforcement Learning algorithm for real-time robotics task

I'm quite a newbie when it comes to practically working with Deep Learning techniques, although I studied them quite a lot theoretically in the last months. However, now I'm facing my first practical ...
1
vote
0answers
13 views

TD losses are descreasing, but also rewards are decreasing, increasing sigma

I'm using Q-learning with some extensions such as noisy linear layers, n-steps and double DQN. The training, however, isn't that successful, my rewards are descreasing over time after a steep ...
2
votes
1answer
82 views

What happens to the optimal value function if the reward is multiplied by a constant?

What happens to the optimal action-value function, $q_*$ if the reward is multiplied by a constant $c$? Is the optimal action-value function also multiplied by such a constant?
1
vote
0answers
58 views

How is q-learning related to game trees?

At a first look, q-learning is a revolutionary strategy in realizing Artificial Intelligence. It has to do with finding a policy, a reward structure, neural networks for storing the q-table and a ...
6
votes
2answers
174 views

What does the symbol $\mathbb E$ mean in these equations?

I came across some papers that use $\mathbb E$ in equations, in particular, this paper: https://arxiv.org/pdf/1511.06581.pdf. Here is some equations from the paper that uses it: $Q^\pi \left(s,a \...
5
votes
1answer
50 views

Can exogenous variables be state features in reinforcement learning?

I have a question about state representation of Q-learning or DQN algorithm. I'm still a beginner of RL, so I'm not sure that is it suitable to take exogenous variables as state features. For example,...
2
votes
3answers
105 views

Reinforcement learning: How to deal with illegal actions?

I'm a beginner of RL and currently trying to make DQN agent that can act optimally in a simple situation. In the situation agent should decide at what rate to charge or discharge the electrical ...
4
votes
1answer
77 views

Is tabular Q-learning considered interpretable?

I am working on a research project in a domain where other related works have always resorted to deep Q-learning. The motivation of my research stems from the fact that the domain has an inherent ...
1
vote
1answer
78 views

Deep Q Learning for Simple Game Not Effective

This is a follow-up question about one I asked earlier. The first question is here. Basically, I have a game where a paddle moves left and right to catch as much "food" as possible. Some food is good (...
1
vote
1answer
81 views

Deep Q Learning Algorithm for Simple Python Game makes player stuck

I made a simple Python game. A screenshot is below: Basically, a paddle moves left and right catching particles. Some make you lose points while others make you gains points. This is my first Deep Q ...
1
vote
0answers
38 views

Unique game problem (ML, DP, PP etc)

Looking for a solution to my below game problem. I believe it to require some sort of reinforcement learning, dynamic programming, or probabilistic programming solution, but am unsure... This is my ...
2
votes
1answer
35 views

Will the target network, which is less trained than the normal network, output inferior estimates?

I'm having some trouble understanding some parts of the usage of target networks. I get that having the same network predict the state/action/advantage values for both the current networks can lead ...
3
votes
0answers
81 views

How can I use Q-learning for inventory decision making?

I am trying to model operational decisions in inventory control. The control policy is base stock with a fixed stock level of $S$. That is replenishment orders are placed for every demand arrival to ...
0
votes
1answer
50 views

Probabilistic action selection in pursuit algorithm

In the Pursuit algorithm (to balance exploration and exploitation), the greedy action has a probability say $p_1$ (updated every episode) of being selected, while the rest have a probability $p_2$ (...
3
votes
1answer
66 views

Is it possible to have a dynamic $Q$-function?

I am trying to use Q-learning for energy optimization. I only wish to have states that will be visited by the learning agent, and, for each state, I have a function that generates possible actions, so ...
1
vote
0answers
33 views

Can multiple reinforcement algorithms be applied to the same system?

Can a system, for instance robotic vehicle, be controlled by more than one reinforcement learning algorithm. I intend to use one to address collision avoidance whereas the other to tackle autonomous ...
2
votes
1answer
41 views

If deep Q learning involves adjusting the value function for a specific policy, then how do I choose the right policy?

I wrote a simple implementation of Flappy Bird in Python, and now I'm trying to train an agent to play it at a reasonable skill level using TFLearn. I feed the network an input vector of size 4: ...
2
votes
0answers
75 views

How does Friend-or-Foe Q-learning intuitively work?

I read about Q-Learning and was reading about multi-agent environments. I tried to read the paper Friend-or-Foe Q-learning, but could not understand anything, except for a very vague idea. What does ...
0
votes
0answers
51 views

Why epsilon-greedy hyperparameter is annealed smoothly?

Regarding of DQN, or DQRNN, (reinforcement learning) To me, RL is a process that can be divided into 2 stages: Exploring wide range of paths (acting randomly) Refining the current optimal paths (...
6
votes
1answer
181 views

Does AlphaZero use Q-Learning?

I was reading the AlphaZero paper Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm, and it seems they don't mention Q-Learning anywhere. So does AZ use Q-...
3
votes
2answers
72 views

What is the difference between return and expected return?

At a time step $t$, for a state $S_{t}$, the return is defined as the discounted cumulative reward from that time step $t$. If an agent is following a policy (which in itself is a probability ...
2
votes
2answers
77 views

How to stop evaluation phase in reinforcement learning with epsilon-greedy Monte Carlo agent?

I have implemented an epsilon-greedy Monte Carlo reinforcement learning agent like suggested in Sutton and Barto's RL book (page 101). As far as I understood epsilon-greedy agents so far, the ...
-2
votes
1answer
169 views

Problem over DQN Algorithm not converging on snake

I'm using a DQN Algorithm to play Snake. The input of the neural network is a stack of 4 images taken from the games 80x80. The output is an array of 4 values, one for every direction. The problem ...
1
vote
1answer
100 views

Deep Reinforcement Learning: Rewards suddenly dip down

I am working on a deep reinforcement learning problem. The policy network has the same architecture as the one Deepmind published in 'Playing Atari with Deep Reinforcement Learning'. I am also using ...
3
votes
1answer
36 views

How do I represent a multi-dimensional state using a neural network?

I have a set of 15 unique playing cards from a deck of 52 playing cards. A given state is represented by the respective card values in the set of 15 cards, where the card value is a prime number ...
4
votes
1answer
103 views

Q-learning, am I interpreting correctly $Q(s,a) = r + \gamma \max_{a'} Q(s',a')$?

Ok, due to previous question I was pointed to use reinfrocement learning. So far what I understood from random websites is the following: there is a Q(s,a) function involved I can assume my neural ...
0
votes
0answers
18 views

Problems while training a DQN Agent on DSTC dataset

I am trying to create a dialogue policy model on DSTC data. This model takes in a state of the conversation and outputs an act the machine must take. I am creating this model using reinforcement ...
1
vote
1answer
213 views

Choice of inputs features for Snake game

I am designing a neural network using Deep Q-Learning, which teaches an agent how to play Snake (The classic Nokia game from the 90'ies). The goal of the game is to navigate the snake on a playing ...
1
vote
0answers
28 views

High variance in performance of q-learning agents trained with same parameters

I am training an agent to play a simple game using double deep q learning. However, the variance in agent performance is very high, even for agents trained with same model parameters. For example, I ...
1
vote
1answer
67 views

Static or dynamic learning rate (Q-learning)

I have the following code (below), where an agent uses Q-learning (RL) to play a simple game. What appears to be questionable for me in that code is the fixed learning rate. When it's set low, it's ...
5
votes
2answers
185 views

Deciding on a reward per each action in a given state (Q-learning)

I looked for existing posts on Stack Exchange, which kind of answer the questions about the reward system and reward function, but not specifically what I want to ask here, which is how do you ...
5
votes
2answers
322 views

Is Q-Learning suitable for continous (state or action) spaces?

Many examples work with a table based method for Q-Llearning. This may be suitable for discrete state(observation) or actions like a robot in a grid world but is there a way to use Q-Learning for ...
1
vote
0answers
24 views

Measure grid-world environments difference for reinforcement learning

I'd like to measure the difference between 2 grid-worlds to determine the generalization capacity of my agent using tabular Q-learning. Example (OpenAI Frozen Lake) : SFFF FHFH FFFH HFFG and : ...
3
votes
1answer
76 views

Experience Replay Not Always Giving Better Results

I have recently started working on a control problem using a Deep Q Network as proposed by DeepMind (https://arxiv.org/abs/1312.5602). Initially, I implemented it without Experience Replay. The ...