Questions tagged [reinforcement-learning]

For questions related to learning controlled by external positive reinforcement or negative feedback signal or both, where learning and use of what has been thus far learned occur concurrently.

Filter by
Sorted by
Tagged with
1
vote
1answer
72 views

A3C fails to solve MountainCar-v0 enviroment (implementation by OpenAi gym)

While I've been able to solve MountainCar-v0 using Deep Q learning, no matter what I try I can't solve this enviroment using policy-gradient approaches. As far as I learnt searching the web, this is a ...
2
votes
0answers
10 views

How is GARB implemented in PGRD-DL to calculate gradients w.r.t. internal rewards?

In section 3 of this paper the author outlines how GARB was adapted to reduce the variance in updating parameters to an internal reward function estimator. I have read it a number of times and ...
3
votes
1answer
82 views

Which kind of prioritized experience replay should I use?

The Prioritized Experience Replay paper gives two different ways of sampling from the replay buffer. One, called "proportional prioritization", assigns each transition a priority proportional to its ...
0
votes
0answers
24 views

Shared actor-critic using only local rules

I was wondering if the following ‘shared actor-critic’ principal using local rules has been established ?.. Take an actor network, which can form abstract (ie hidden layer) representations using a ...
5
votes
2answers
49 views

Reinforcement learning with uniformly random dynamics

Suppose I have an MDP $(S, A, p, R)$ where the $p(s_j|s_i,a_i)$ is uniform, i.e given an state $s_i$ and an action $a_i$ all states $s_j$ are equally probable. Now I want to find an optimal policy ...
2
votes
0answers
33 views

How does the TRPO surrogate loss account for the error in the policy?

In the Trust Region Policy Optimization (TRPO) paper, on page 10, it is stated An informal overview is as follows. Our proof relies on the notion of coupling, where we jointly define the ...
1
vote
0answers
53 views

Do we need to use the experience replay buffer with the A3C algorithm?

I have skimmed through a bunch of deep learning books, but I have not yet understood whether we must use the experience replay buffer with the A3C algorithm. The approached I used is the following: ...
1
vote
0answers
18 views

What are a list of board game environments for RL practice?

Recently OpenAI removes their board game environments. (It may be possible to install an older version to get access to them, but I haven’t downgraded). Is there a list of repositories or resources ...
3
votes
0answers
41 views

Should noise (such as OU) be decreased over time in actor / critic algorithms?

In most of RL algorithms I saw, there is a coefficient that reduces actions exploration over time, to help convergence. But in Actor-Critic, or other algorithms (A3C, DDPG, ...) used in continuous ...
2
votes
0answers
95 views

Why overfitting is bad in DQN?

It is mentioned by Fu 2019 that overfitting might have a negative effect on training DQN. They showed that with either early stopping or experience replay this effect could be reduced. The first is ...
5
votes
2answers
957 views

What algorithms are considered reinforcement learning algorithms?

What are the areas that belong to the Reinforcement Learning? TD(0), Q-Learning and SARSA are all temporal-difference algorithms, which belong to the reinforcement learning area, but is there more to ...
0
votes
0answers
20 views

Measure grid-world environments difference for reinforcement learning

I'd like to measure the difference between 2 grid-worlds to determine the generalization capacity of my agent using tabular Q-learning. Example (OpenAI Frozen Lake) : SFFF FHFH FFFH HFFG and : ...
1
vote
0answers
27 views

How is Parallelism Implemented in RL algorithms like PPO?

There are multiple ways to implement parallelism in reinforcement learning. One is to use parallel workers running in their own environments to collect data in parallel, instead of using replay memory ...
3
votes
1answer
69 views

Experience Replay Not Always Giving Better Results

I have recently started working on a control problem using a Deep Q Network as proposed by DeepMind (https://arxiv.org/abs/1312.5602). Initially, I implemented it without Experience Replay. The ...
4
votes
0answers
51 views

Inconsistent definitions of the retrace

In Section 4.3 of paper Learning by Playing - Solving Sparse Reward Tasks from Scratch, the authors define Retrace as $$ Q^{ret}=\sum_{j=i}^\infty\left(\gamma^{j-i}\prod_{k=i}^jc_k\right)[r(s_j,a_j)+\...
3
votes
0answers
27 views

Code examples of controlling multiple units with RL

Anyone knows a resources (papers, articles and especially repositories) regarding controlling multiple units with RL. The controlled units should not be fixed, for example in Real Time Strategy the ...
1
vote
0answers
27 views

How to properly optimize shared network between actor and critic?

I'm building an actor-critic reinforcment learning algorithm to solve environments. I want to use a single encoder to find representation of my environment. When I share the encoder with the actor ...
1
vote
1answer
49 views

Picking a random move in exploitation in Q-Learning

I've been unsure about a principle of Q-Learning, I was hoping someone could clear it up. When a new state is encountered, and thus there are no existing Q values, and that the algorithm decides to ...
0
votes
0answers
39 views

$\epsilon$-greedy policies for huge state space

I'm currently building an agent that learns to play Kalah through reinforcement learning. I've gotten quite far along. With an $\epsilon$ of 0, meaning no exploration and only exploitation, it is able ...
1
vote
3answers
52 views

How to stop DQN Q function from increasing during learning?

Following the DQN algorithm with experience replay: We calculate the $loss=(Q(s,a)-(r+Q(s+1,a)))^2$. Assume I have positive but changing rewards. Meaning, $r>0$. Thus, since the rewards are ...
2
votes
1answer
101 views

Alphazero policy head loss not decreasing

I am now working on training an alphazero player for a board game. The implementation of board game is mine, MCTS for alphazero was taken elsewhere. Due to complexity of the game, it takes a much ...
1
vote
1answer
34 views

Is there any example of using Q-learning with big data?

Could we even use reinforcement learning with big datasets? Or in RL does the agent built its own dataset ?
1
vote
0answers
61 views

Difficulty in balancing Pendulum using Deep Reinforcement Learning Algorithm

I am using OpenAI Gym framework for reinforcement learning where I am trying solve classic control problem of balancing an Inverted Pendulum, which is similar to the "Pendulum-v0" with some changes in ...
2
votes
0answers
34 views

Why experience reply memory in DQN instead of a RNN memory?

I was trying to implement a DQN without experience reply memory, and the agent is not learning anything at all. I know from readings that experience reply is used for stabilizing gradients. But how ...
1
vote
1answer
37 views

AlphaGo neural network inputs

I have two questions: 1) I have been reading an article on AlphaGo and one sentence confused me a little bit, because I'm not sure what it exactly means. The article says: AlphaGo Zero only uses ...
4
votes
1answer
124 views

How is the policy gradient calculated in REINFORCE?

Reading Sutton and Barto, I see the following in describing policy gradients: How is the gradient calculated with respect to an action (taken at time t)? I've read implementations of the algorithm, ...
1
vote
0answers
69 views

Policy gradient loss for neural network training

Say i want to train a neural network with 10 classes as outputs and use categorical_cross_entropy as a loss function in keras. This will try to fit the training ...
2
votes
1answer
99 views

Intuition behind $\gamma$-discounted state frequency

At the appendix A of paper "near-optimal representation learning for hierarchical reinforcement learning", the authors express the $\gamma$-discounted state visitation frequency $d$ of policy $\pi$ as ...
0
votes
0answers
19 views

Turn a NES ROM into object/tile representation

So i have a rom of a hacked super mario game (it has 2 players: Mario and Luigi). Feeding in the raw pixel data of this results in very poor rewards. I was wondering if there was a way to transform ...
2
votes
1answer
33 views

Are successive actions independent?

The proof of the consistency of the per-decision importance sampling estimator assumes the independence of $$\frac{\pi(A_t|S_t)}{b(A_t|S_t)}R_{t+1}\quad\text{ and }\quad \prod_{k=t+1}^{T-1}\frac{\pi(...
7
votes
1answer
110 views

Are there reinforcement learning algorithms that scale to large problems?

Given a large problem, value iteration and other table based approaches seem to require too many iterations before they start to converge. Are there other reinforcement learning approaches that better ...
1
vote
0answers
21 views

Deciding the rewards for different actions in Pong for a DQN agent

I am attempting to implement an agent that learns to play in the Pong environment, the environment was created in PyGame and I return the pixel data and score at each frame. I use a CNN to take a ...
1
vote
0answers
68 views

Difficulty understanding Monte Carlo policy evaluation (state-value) for gridworld

I've been trying to read Sutton & Barto book chapter 5.1, but I'm still a bit confused about the procedure of using Monte Carlo policy evaluation (p.92), and now I just cant proceed anymore coding ...
4
votes
2answers
71 views

How can alpha zero learn if the tree search stops and restarts before finishing a game?

I am trying to understand how alpha zero works, but there is one point that I have problems understanding, even after reading several different explanations. As I understand it (see for example https:/...
1
vote
2answers
56 views

How do map providers like Google calculate the distance between two coordinates and find turn by turn directions?

I have searched on how Google or any map provider calculates distance between two coordinates. The closest I could find is Haversine formula. If I draw a straight line between two points, then ...
1
vote
1answer
75 views

How can the $\lambda$-return be defined recursively?

The $\lambda$-return is defined as $$G_t^\lambda = (1-\lambda)\sum_{n=1}^\infty \lambda^{n-1}G_{t:t+n}$$ where $$G_{t:t+n} = R_{t+1}+\gamma R_{t+2}+\dots +\gamma^{n-1}R_{t+n} + \gamma^n\hat{v}(S_{t+n})...
5
votes
2answers
97 views

Why don't people use projected Bellman error with deep neural networks?

Projected Bellman error has shown to be stable with linear function approximation. The technique is not at all new. I can only wonder why this technique is not adopted to use with non-linear function ...
3
votes
0answers
39 views

Deep Q-Learning agent poor performing actions. Need help optimizing

I'm trying to make deep q-learning agent from https://keon.io/deep-q-learning My environment looks like this: https://imgur.com/a/OnbiCtV As you can see my agent is a circle and there is one gray ...
1
vote
0answers
49 views

Actor-critic algorithm using gaussian Radial Basis Function, Local Linear Regression and shallow Neural Network

I'm attempting to implement the actor-critic algorithm on Matlab using Radial Basis Function, Local Linear Regression, and shallow Neural Network for inverted pendulum system. the state space and the ...
3
votes
0answers
44 views

Feature Selection using Monte Carlo Tree Search

I'm trying to tackle the problem of feature selection as an RL problem, inspired by the paper Feature Selection as a One-Player Game. I know Monte-Carlo tree search (MCTS) is hardly RL. So, I used ...
0
votes
1answer
109 views

Bachelor thesis in reinforcement learning

I've decided to make my bachelor thesis in RL. I am currently struggling in finding a proper problematic. I am interested in multi-agent RL with the dilemma between selfishness and cooperation. I ...
3
votes
0answers
95 views

What can be considered a deep recurrent neural network?

In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the DRQN is described as DQN with the first post-convolutional fully-connected layer replaced by a recurrent LSTM. I have DQN ...
4
votes
2answers
85 views

Can DQN perform better than Double DQN?

I'm training both kind of agents against an environment but DQN performs significantly better than Double DQN. As I've saw here, Double DQN use to perform better than DQN. Am I doing something wrong ...
4
votes
0answers
33 views

What research has been done on learning non-Markovian reward functions?

Recently, some work has been done planning and learning in Non-Markovian Decision Processes, that is, decision-making with temporally extended rewards. In these settings, a particular reward is ...
3
votes
0answers
132 views

How can I convert the problem formulation to multi-agent reinforcement learning?

I'm trying to minimize the power consumption in wireless networks and I have some constraints such as that the SINR should not pass the threshold and the power should be between the 0 and maximum ...
9
votes
2answers
228 views

Why doesn't Q-learning converge when using function approximation?

The tabular Q-learning algorithm is guaranteed to find the optimal $Q$ function, $Q^*$, provided the following conditions regarding the learning rate are satisfied $\sum_{t} \alpha_t(s, a) = \infty$ $...
3
votes
1answer
72 views

Understanding the n-step off-policy SARSA update

In Sutton & Barto's book (2nd ed) page 149, there is the equation 7.11 I am having a hard time understanding this equation. I would have thought that we should be moving $Q$ towards $G$, where $...
1
vote
1answer
49 views

What is the motivation behind using a deterministic policy?

What is the motivation behind using a deterministic policy? Given that the environment is uncertain, it seems stochastic policy makes more sense.
2
votes
2answers
371 views

How large should the replay buffer be?

I'm learning DDPG algorithm by following the following link: Open AI Spinning Up document on DDPG, where it is written In order for the algorithm to have stable behavior, the replay buffer should ...
2
votes
1answer
24 views

Inform policy learning of environment constants

Policy learning refers to mapping an agent state onto an action to maximize reward. A linear policy, such as the one used in the Augmented Random Search paper, refers to learning a linear mapping ...