Questions tagged [q-learning]

For questions related to the Q-learning algorithm, which is a model-free and temporal-difference reinforcement learning algorithm that attempts to approximate the Q function, which is a function that, given a state s and an action a, returns a real number that represents the return (or value) of state s when action a is taken from s. Q-learning was introduced in the PhD thesis "Learning from Delayed Rewards" (1989) by Watkins.

Filter by
Sorted by
Tagged with
4
votes
1answer
408 views

How can a DQN backpropagate its loss?

I'm currently trying to take the next step in deep learning. I managed so far to write my own basic feed-forward network in python without any frameworks (just numpy and pandas), so I think I ...
3
votes
1answer
121 views

Is the Q value updated at every episode?

I trying to understand the Bellman equation for updating the Q table values. The concept of initially updating the value is clear to me. What is unclear is the subsequent updates to the value. Is the ...
1
vote
0answers
40 views

N-tuple based tic tac toe diverges in temporal difference learning

I have n-tuple based tic tac toe. I already have perfect minimax player and perfectly trained table-based player. My n-tuple network consists of 8 different rows of 3 of the board as triplets having ...
2
votes
0answers
82 views

Q-learning: How to include a terminal state in updating rule? [duplicate]

I use Q-learning in order to determine the optimal path of an agent. I know in advance that my path is composed of exactly 3 states (so after 3 states I reach a terminal state). I would like to know ...
1
vote
0answers
81 views

Reinforcement learning for a 2D game involving two players

I'd like to create an AI for a 2D game involving two players fighting against each other. The map look something like this (The map is a NxN array somehow randomly generated): Basically the players ...
1
vote
1answer
46 views

Is the following the correct implementation of the Q learning algorithm for a neural network?

I just wanted to confirm that my understanding of Q learning was correct (with respect to a neural network). ...
6
votes
2answers
1k views

Does using the softmax function in Q learning not defeat the purpose of Q learning?

It is my understanding that, in Q-learning, you are trying to mimic the optimal $Q$ function $Q*$, where $Q*$ is a measure of the predicted reward received from taking action $a$ at state $s$ so that ...
2
votes
0answers
75 views

How would one develop an action space for a game that is proprietary?

I'm currently trying to develop an RL that will teach itself to play the popular fighting game "Tekken 7". I initially had the idea of teaching it to play generally- against actual opponents with ...
3
votes
1answer
126 views

Can Google's patented ML algorithms be used commercially?

I just find that Google patents some of the widely used machine learning algorithms. For example: System and method for addressing overfitting in a neural network (Dropout?) Processing images using ...
2
votes
0answers
38 views

What is the complexity of policy gradient algorithms compared to discrete action space algorithms?

I am using a policy gradient algorithm (actor-critic) for wireless networks. The policy gradient-based algorithm helps because it considers continuous action space. But how much does a policy ...
2
votes
1answer
68 views

Why feed actions in later layer in Q network?

I read the DDPG paper, in which the authors state that the actions are fed only later to their Q network: Actions were not included until the 2nd hidden layer of Q. (Sec 7, Experiment Details) So ...
1
vote
0answers
43 views

Did I understand deep Q leaning right? (Implementation)

Gday guys, so I tried to implement my own enviroment and agent in order to fully understand DQNs. The enviroment is a dungeon with five states. actionspace = 2 statespace = 5 !!!Action a0 is ...
4
votes
1answer
464 views

How do we compute the target value when the agent ends up in the terminal state?

I am working on a deep reinforcement learning problem. Throughout the episode, there is a small positive and negative reward for good or bad decisions. In the end, there is a huge reward for the ...
3
votes
1answer
864 views

Why is my implementation of Q-learning not converging to the right values in the FrozenLake environment?

I am trying to learn tabular Q learning by using a table of states and actions (i.e. no neural networks). I was trying it out on the FrozenLake environment. It's a very simple environment, where the ...
1
vote
0answers
26 views

TD losses are descreasing, but also rewards are decreasing, increasing sigma

I'm using Q-learning with some extensions such as noisy linear layers, n-steps and double DQN. The training, however, isn't that successful, my rewards are descreasing over time after a steep ...
2
votes
1answer
135 views

When training a DQN, how should we update the value of actions that were not taken?

Let's say that we have three actions. The highest-valued action of the three choices is the first. When training the DQN, what do we do with the other two, as we don't have a target for them, since ...
7
votes
2answers
961 views

What does the symbol $\mathbb E$ mean in these equations?

I came across some papers that use $\mathbb E$ in equations, in particular, this paper: https://arxiv.org/pdf/1511.06581.pdf. Here is some equations from the paper that uses it: $Q^\pi \left(s,a \...
5
votes
1answer
146 views

Can exogenous variables be state features in reinforcement learning?

I have a question about state representation of Q-learning or DQN algorithm. I'm still a beginner of RL, so I'm not sure that is it suitable to take exogenous variables as state features. For example,...
2
votes
3answers
686 views

Reinforcement learning: How to deal with illegal actions? [duplicate]

I'm a beginner of RL and currently trying to make DQN agent that can act optimally in a simple situation. In the situation agent should decide at what rate to charge or discharge the electrical ...
4
votes
1answer
196 views

Is tabular Q-learning considered interpretable?

I am working on a research project in a domain where other related works have always resorted to deep Q-learning. The motivation of my research stems from the fact that the domain has an inherent ...
1
vote
1answer
90 views

Deep Q Learning for Simple Game Not Effective

This is a follow-up question about one I asked earlier. The first question is here. Basically, I have a game where a paddle moves left and right to catch as much "food" as possible. Some food is good (...
1
vote
1answer
115 views

Deep Q Learning Algorithm for Simple Python Game makes player stuck

I made a simple Python game. A screenshot is below: Basically, a paddle moves left and right catching particles. Some make you lose points while others make you gains points. This is my first Deep Q ...
1
vote
0answers
48 views

Unique game problem (ML, DP, PP etc)

Looking for a solution to my below game problem. I believe it to require some sort of reinforcement learning, dynamic programming, or probabilistic programming solution, but am unsure... This is my ...
2
votes
1answer
60 views

Will the target network, which is less trained than the normal network, output inferior estimates?

I'm having some trouble understanding some parts of the usage of target networks. I get that having the same network predict the state/action/advantage values for both the current networks can lead ...
3
votes
0answers
90 views

How can I use Q-learning for inventory decision making?

I am trying to model operational decisions in inventory control. The control policy is base stock with a fixed stock level of $S$. That is replenishment orders are placed for every demand arrival to ...
1
vote
1answer
129 views

Probabilistic action selection in pursuit algorithm

In the Pursuit algorithm (to balance exploration and exploitation), the greedy action has a probability say $p_1$ (updated every episode) of being selected, while the rest have a probability $p_2$ (...
3
votes
1answer
139 views

Is it possible to have a dynamic $Q$-function?

I am trying to use Q-learning for energy optimization. I only wish to have states that will be visited by the learning agent, and, for each state, I have a function that generates possible actions, so ...
1
vote
0answers
35 views

Can multiple reinforcement algorithms be applied to the same system?

Can a system, for instance robotic vehicle, be controlled by more than one reinforcement learning algorithm. I intend to use one to address collision avoidance whereas the other to tackle autonomous ...
2
votes
1answer
48 views

If deep Q learning involves adjusting the value function for a specific policy, then how do I choose the right policy?

I wrote a simple implementation of Flappy Bird in Python, and now I'm trying to train an agent to play it at a reasonable skill level using TFLearn. I feed the network an input vector of size 4: ...
2
votes
0answers
92 views

How does Friend-or-Foe Q-learning intuitively work?

I read about Q-Learning and was reading about multi-agent environments. I tried to read the paper Friend-or-Foe Q-learning, but could not understand anything, except for a very vague idea. What does ...
8
votes
1answer
678 views

Does AlphaZero use Q-Learning?

I was reading the AlphaZero paper Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm, and it seems they don't mention Q-Learning anywhere. So does AZ use Q-...
4
votes
2answers
221 views

What is the difference between return and expected return?

At a time step $t$, for a state $S_{t}$, the return is defined as the discounted cumulative reward from that time step $t$. If an agent is following a policy (which in itself is a probability ...
2
votes
2answers
120 views

How to stop evaluation phase in reinforcement learning with epsilon-greedy Monte Carlo agent?

I have implemented an epsilon-greedy Monte Carlo reinforcement learning agent like suggested in Sutton and Barto's RL book (page 101). As far as I understood epsilon-greedy agents so far, the ...
0
votes
1answer
569 views

Problem over DQN Algorithm not converging on snake

I'm using a DQN Algorithm to play Snake. The input of the neural network is a stack of 4 images taken from the games 80x80. The output is an array of 4 values, one for every direction. The problem ...
3
votes
1answer
506 views

Deep Reinforcement Learning: Rewards suddenly dip down

I am working on a deep reinforcement learning problem. The policy network has the same architecture as the one Deepmind published in 'Playing Atari with Deep Reinforcement Learning'. I am also using ...
3
votes
1answer
49 views

How do I represent a multi-dimensional state using a neural network?

I have a set of 15 unique playing cards from a deck of 52 playing cards. A given state is represented by the respective card values in the set of 15 cards, where the card value is a prime number ...
4
votes
1answer
111 views

Q-learning, am I interpreting correctly $Q(s,a) = r + \gamma \max_{a'} Q(s',a')$?

Ok, due to previous question I was pointed to use reinfrocement learning. So far what I understood from random websites is the following: there is a Q(s,a) function involved I can assume my neural ...
1
vote
1answer
668 views

Choice of inputs features for Snake game

I am designing a neural network using Deep Q-Learning, which teaches an agent how to play Snake (The classic Nokia game from the 90'ies). The goal of the game is to navigate the snake on a playing ...
1
vote
0answers
33 views

High variance in performance of q-learning agents trained with same parameters

I am training an agent to play a simple game using double deep q learning. However, the variance in agent performance is very high, even for agents trained with same model parameters. For example, I ...
1
vote
1answer
856 views

In Q-learning, shouldn't the learning rate change dynamically during the learning phase?

I have the following code (below), where an agent uses Q-learning (RL) to play a simple game. What appears to be questionable for me in that code is the fixed learning rate. When it's set low, it's ...
11
votes
2answers
6k views

Can Q-learning be used for continuous (state or action) spaces?

Many examples work with a table-based method for Q-learning. This may be suitable for a discrete state (observation) or action space, like a robot in a grid world, but is there a way to use Q-learning ...
1
vote
0answers
34 views

Measure grid-world environments difference for reinforcement learning

I'd like to measure the difference between 2 grid-worlds to determine the generalization capacity of my agent using tabular Q-learning. Example (OpenAI Frozen Lake) : SFFF FHFH FFFH HFFG and : ...
3
votes
1answer
128 views

Experience Replay Not Always Giving Better Results

I have recently started working on a control problem using a Deep Q Network as proposed by DeepMind (https://arxiv.org/abs/1312.5602). Initially, I implemented it without Experience Replay. The ...
4
votes
0answers
75 views

Why is there inconsistency in the definitions of the retrace?

In Section 4.3 of paper Learning by Playing - Solving Sparse Reward Tasks from Scratch, the authors define Retrace as $$ Q^{ret}=\sum_{j=i}^\infty\left(\gamma^{j-i}\prod_{k=i}^jc_k\right)[r(s_j,a_j)+\...
1
vote
1answer
97 views

Picking a random move in exploitation in Q-Learning

I've been unsure about a principle of Q-Learning, I was hoping someone could clear it up. When a new state is encountered, and thus there are no existing Q values, and that the algorithm decides to ...
5
votes
4answers
846 views

How to stop DQN Q function from increasing during learning?

Following the DQN algorithm with experience replay: Store transition $\left(\phi_{t}, a_{t}, r_{t}, \phi_{t+1}\right)$ in $D$ Sample random minibatch of transitions $\left(\phi_{j}, a_{j}, r_{j}, \...
0
votes
1answer
60 views

Encourage Deep Q to seek short-term reward

I understand that gamma is an important factor in determining the rewards for a deep Q agent, however during testing of my network I am noticing that the agent is outputting more actions to "do ...
2
votes
1answer
68 views

Maximum Q value for new state in Q-Learning never exists

I'm working on implementing a Q-Learning algorithm for a 2 player board game. I encountered what I think may be a problem. When it comes time to update the Q value with the Bellman equation (above), ...
3
votes
0answers
52 views

Deep Q-Learning agent poor performing actions. Need help optimizing

I'm trying to make deep q-learning agent from https://keon.io/deep-q-learning My environment looks like this: https://imgur.com/a/OnbiCtV As you can see my agent is a circle and there is one gray ...
6
votes
2answers
722 views

Can DQN perform better than Double DQN?

I'm training both DQN and double DQN in the same environment, but DQN performs significantly better than double DQN. As I've seen in the double DQN paper, double DQN should perform better than DQN. Am ...

1
3 4
5
6 7