Questions tagged [reinforcement-learning]
For questions related to learning controlled by external positive reinforcement or negative feedback signal or both, where learning and use of what has been thus far learned occur concurrently.
0
votes
0answers
9 views
How to properly optimize shared network between actor and critic?
I'm building an actor-critic reinforcment learning algorithm to solve environments. I want to use a single encoder to find representation of my environment.
When I share the encoder with the actor ...
0
votes
1answer
22 views
Picking a random move in exploitation in Q-Learning
I've been unsure about a principle of Q-Learning, I was hoping someone could clear it up.
When a new state is encountered, and thus there are no existing Q values, and that the algorithm decides to ...
0
votes
0answers
27 views
$\epsilon$-greedy policies for huge state space
I'm currently building an agent that learns to play Kalah through reinforcement learning. I've gotten quite far along. With an $\epsilon$ of 0, meaning no exploration and only exploitation, it is able ...
0
votes
2answers
27 views
How to stop DQN Q function from increasing during learning?
Following the DQN algorithm with experience replay:
We calculate the $loss=(Q(s,a)-(r+Q(s+1,a)))^2$.
Assume I have positive but changing rewards. Meaning, $r>0$.
Thus, since the rewards are ...
0
votes
1answer
21 views
Alphazero policy head loss not decreasing
I am now working on training an alphazero player for a board game. The implementation of board game is mine, MCTS for alphazero was taken elsewhere. Due to complexity of the game, it takes a much ...
1
vote
1answer
26 views
Is there any example of using Q-learning with big data?
Could we even use reinforcement learning with big datasets?
Or in RL does the agent built its own dataset ?
1
vote
0answers
32 views
Difficulty in balancing Pendulum using Deep Reinforcement Learning Algorithm
I am using OpenAI Gym framework for reinforcement learning where I am trying solve classic control problem of balancing an Inverted Pendulum, which is similar to the "Pendulum-v0" with some changes in ...
2
votes
0answers
19 views
Why experience reply memory in DQN instead of a RNN memory?
I was trying to implement a DQN without experience reply memory, and the agent is not learning anything at all. I know from readings that experience reply is used for stabilizing gradients. But how ...
1
vote
1answer
22 views
AlphaGo neural network inputs
I have two questions:
1)
I have been reading an article on AlphaGo and one sentence confused me a little bit, because I'm not sure what it exactly means. The article says:
AlphaGo Zero only uses ...
4
votes
1answer
39 views
How is the policy gradient calculated in REINFORCE?
Reading Sutton and Barto, I see the following in describing policy gradients:
How is the gradient calculated with respect to an action (taken at time t)? I've read implementations of the algorithm, ...
1
vote
0answers
32 views
Policy gradient loss for neural network training
Say i want to train a neural network with 10 classes as outputs and use categorical_cross_entropy as a loss function in keras. This will try
to fit the training ...
0
votes
0answers
23 views
A few questions regarding the on-policy $\epsilon$-greedy Monte Carlo control algorithm [closed]
Below is the $\epsilon$-greedy Monte Carlo algorithm.
I do not fully understand it. I have a few questions.
$A^*$ probably just means the action that was returned by the $\arg max$, i.e. it will be ...
2
votes
0answers
44 views
Intuition behind $\gamma$-discounted state frequency
At the appendix A of paper "near-optimal representation learning for hierarchical reinforcement learning", the authors express the $\gamma$-discounted state visitation frequency $d$ of policy $\pi$ as
...
0
votes
0answers
18 views
Turn a NES ROM into object/tile representation
So i have a rom of a hacked super mario game (it has 2 players: Mario and Luigi). Feeding in the raw pixel data of this results in very poor rewards. I was wondering if there was a way to transform ...
3
votes
1answer
28 views
Are successive actions independent?
The proof of the consistency of the per-decision importance sampling estimator assumes the independence of
$$\frac{\pi(A_t|S_t)}{b(A_t|S_t)}R_{t+1}\quad\text{ and }\quad \prod_{k=t+1}^{T-1}\frac{\pi(...
7
votes
1answer
83 views
Are there reinforcement learning algorithms that scale to large problems?
Given a large problem, value iteration and other table based approaches seem to require too many iterations before they start to converge. Are there other reinforcement learning approaches that better ...
1
vote
0answers
13 views
Deciding the rewards for different actions in Pong for a DQN agent
I am attempting to implement an agent that learns to play in the Pong environment, the environment was created in PyGame and I return the pixel data and score at each frame. I use a CNN to take a ...
1
vote
0answers
31 views
Difficulty understanding Monte Carlo policy evaluation (state-value) for gridworld
I've been trying to read Sutton & Barto book chapter 5.1, but I'm still a bit confused about the procedure of using Monte Carlo policy evaluation (p.92), and now I just cant proceed anymore coding ...
3
votes
1answer
38 views
How can alpha zero learn if the tree search stops and restarts before finishing a game?
I am trying to understand how alpha zero works, but there is one point that I have problems understanding, even after reading several different explanations. As I understand it (see for example https:/...
0
votes
1answer
22 views
How do map providers like Google calculate the distance between two coordinates and find turn by turn directions?
I have searched on how Google or any map provider calculates distance between two coordinates. The closest I could find is Haversine formula.
If I draw a straight line between two points, then ...
1
vote
1answer
53 views
How can the $\lambda$-return be defined recursively?
The $\lambda$-return is defined as
$$G_t^\lambda = (1-\lambda)\sum_{n=1}^\infty \lambda^{n-1}G_{t:t+n}$$
where
$$G_{t:t+n} = R_{t+1}+\gamma R_{t+2}+\dots +\gamma^{n-1}R_{t+n} + \gamma^n\hat{v}(S_{t+n})...
2
votes
0answers
14 views
Why don't people use projected bellman error with deep nets?
Projected Bellman error has shown to be stable with linear function approximation. The technique is not at all new. I can only wonder why this technique is not adopted to use with non-linear function ...
2
votes
0answers
25 views
Deep Q-Learning agent poor performing actions. Need help optimizing
I'm trying to make deep q-learning agent from https://keon.io/deep-q-learning
My environment looks like this:
https://imgur.com/a/OnbiCtV
As you can see my agent is a circle and there is one gray ...
0
votes
0answers
30 views
Actor-critic algorithm using gaussian Radial Basis Function, Local Linear Regression and shallow Neural Network
I'm attempting to implement the actor-critic algorithm on Matlab using Radial Basis Function, Local Linear Regression, and shallow Neural Network for inverted pendulum system.
the state space and the ...
3
votes
0answers
30 views
Feature Selection using Monte Carlo Tree Search
I'm trying to tackle the problem of feature selection as an RL problem, inspired by the paper Feature Selection as a One-Player Game. I know Monte-Carlo tree search (MCTS) is hardly RL.
So, I used ...
0
votes
1answer
77 views
Bachelor thesis in reinforcement learning
I've decided to make my bachelor thesis in RL.
I am currently struggling in finding a proper problematic. I am interested in multi-agent RL with the dilemma between selfishness and cooperation.
I ...
-2
votes
0answers
19 views
Huber loss going up performs better
I'm training an Agent with the Huber loss, and as I'm getting bigger rewards, the loss only goes up. Shouldn't it go down as it is a loss function?
2
votes
0answers
43 views
What can be considered a deep recurrent neural network?
In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the DRQN is described as DQN with the first post-convolutional fully-connected layer replaced by a recurrent LSTM.
I have DQN ...
4
votes
2answers
58 views
Can DQN perform better than Double DQN?
I'm training both kind of agents against an environment but DQN performs significantly better than Double DQN. As I've saw here, Double DQN use to perform better than DQN. Am I doing something wrong ...
4
votes
0answers
18 views
What research has been done on learning non-Markovian reward functions?
Recently, some work has been done planning and learning in Non-Markovian Decision Processes, that is, decision-making with temporally extended rewards. In these settings, a particular reward is ...
3
votes
0answers
117 views
How can I convert the problem formulation to multi-agent reinforcement learning?
I'm trying to minimize the power consumption in wireless networks and I have some constraints such as that the SINR should not pass the threshold and the power should be between the 0 and maximum ...
8
votes
2answers
90 views
Why doesn't Q-learning converge when using function approximation?
The tabular Q-learning algorithm is guaranteed to find the optimal $Q$ function, $Q^*$, provided the following conditions regarding the learning rate are satisfied
$\sum_{t} \alpha_t(s, a) = \infty$
$...
-1
votes
0answers
31 views
Actor-critic model returns nan values
I want to use the actor critic model in order to makes a player of the famous Mountain Car game.
All the environment code is here: https://github.com/nbrosson/Actor-critic-MountainCar/
Everything ...
3
votes
1answer
33 views
Understanding the n-step off-policy SARSA update
In Sutton & Barto's book (2nd ed) page 149, there is the equation 7.11
I am having a hard time understanding this equation.
I would have thought that we should be moving $Q$ towards $G$, where $...
1
vote
1answer
41 views
What is the motivation behind using a deterministic policy?
What is the motivation behind using a deterministic policy? Given that the environment is uncertain, it seems stochastic policy makes more sense.
0
votes
1answer
26 views
How large should the replay buffer be?
I'm learning DDPG algorithm by following the following link: Open AI Spinning Up document on DDPG, where it is written
In order for the algorithm to have stable behavior, the replay buffer should ...
2
votes
1answer
23 views
Inform policy learning of environment constants
Policy learning refers to mapping an agent state onto an action to maximize reward. A linear policy, such as the one used in the Augmented Random Search paper, refers to learning a linear mapping ...
1
vote
1answer
28 views
IQN bellman target: using Z vs using Q
IQN paper (https://arxiv.org/abs/1806.06923) uses distributional bellman target:
$$ \delta^{\tau,\tau'}_t = r_t + \gamma Z_{\tau'}(x_{t+1}, \pi_{\beta}(x_{t+1})) - Z_{\tau}(x_t, a_t) $$
And optimizes: ...
1
vote
0answers
30 views
Does everyone still use discount rates?
In Section 10.4 of Sutton and Barto's RL book, they argue that the discount rate $\gamma$ has no effect in continuing settings. They show (at least for one objective function) that the average of the ...
3
votes
2answers
52 views
Can Q-learning be used in a POMDP?
Can Q-learning (and SARSA) be directly used in a Partially Observable Markov Decision Process (POMDP)? If not, why not? My intuition is that the policies learned will be terrible because of partial ...
1
vote
1answer
36 views
DQN Q-values are static
I am working on a DDQN with 5 LSTM layers and 3 actions as output and state space of 21 features. I am dividing the dataset into episodes of 720 timesteps, for each episode the agent acts greedily for ...
1
vote
0answers
24 views
What is the internal state of a Simple Neural Attentive Meta-Learner(SNAIL)?
In the paper A Simple Neural Attentive Meta-Learner, the authors mentioned right before Section 3.1:
we preserve the internal state of a SNAIL across episode boundaries, which allows it to have ...
1
vote
1answer
38 views
What is the difference between A2C and running an agent in an environment vector?
I've implemented A2C, but I'm now wondering why we have multiple actors walk around the environment and gather rewards, why not just have a single agent run in an environment vector? I personally ...
3
votes
2answers
61 views
Is reinforcement learning using shallow neural networks still deep reinforcement learning?
Often times I see the term deep reinforcement learning to refer to RL algorithms that use neural networks, regardless of whether or not the networks are deep.
For example, PPO (https://arxiv.org/pdf/...
0
votes
0answers
9 views
Can a DQNAgent with a PrioritizedMemory overfit? [migrated]
I'm using a DQNAgent and a PrioritizedMemory to train against an environment and by the rewards it could be overfitting but, can this really happen when the environment only shows new states or is it ...
0
votes
0answers
34 views
Policy gradient in keras predicts only one action
I have trouble with the REINFORCE algorithm in keras with Atari games. After round about 30 episodes the network converges to one action. But the same algorithm is working with CartPole-v1 and ...
0
votes
0answers
15 views
How to limit actions based on a state [duplicate]
I'm trying to implement DQN using tf-agents for simple environment.
So far I have
...
2
votes
1answer
35 views
Can the normalization factor for the belief state update be zero?
In order to update the belief state in a POMDP, the following formula is used:
$$b'(s')=\frac{O(a, s', z) \sum_{s\in S} b(s)T(s, a, s')}{\mathbb{P}(z \mid b, a)}$$
where
$s$ is a specific state in ...
0
votes
1answer
35 views
DQN Q-mean values converge negatively
I'm trying to implement my own DQN. So far I think my code is good, but my Q-values (I'm getting the mean of all the values for every episode) tends to converge near-zero but negatively. It is normal? ...
2
votes
1answer
47 views
Is REINFORCE the same as 'vanilla policy gradient'?
I don't know what people mean by 'vanilla policy gradient', but what comes to mind is REINFORCE, which is the simplest policy gradient algorithm I can think of. Is this an accurate statement?
By ...
