Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

594 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
2
votes
0answers
50 views

Are there reinforcement learning algorithms that ensure convergence for continuous state space problems?

The Q-learning does not guarantee convergence for continuous state space problems (Why doesn't Q-learning converge when using function approximation?). In that case, is there an algorithm which ...
2
votes
0answers
17 views

How is GARB implemented in PGRD-DL to calculate gradients w.r.t. internal rewards?

In section 3 of this paper the author outlines how GARB was adapted to reduce the variance in updating parameters to an internal reward function estimator. I have read it a number of times and ...
2
votes
0answers
53 views

How does the TRPO surrogate loss account for the error in the policy?

In the Trust Region Policy Optimization (TRPO) paper, on page 10, it is stated An informal overview is as follows. Our proof relies on the notion of coupling, where we jointly define the ...
2
votes
0answers
229 views

Why overfitting is bad in DQN?

It is mentioned by Fu 2019 that overfitting might have a negative effect on training DQN. They showed that with either early stopping or experience replay this effect could be reduced. The first is ...
2
votes
0answers
141 views

Why experience reply memory in DQN instead of a RNN memory?

I was trying to implement a DQN without experience reply memory, and the agent is not learning anything at all. I know from readings that experience reply is used for stabilizing gradients. But how ...
2
votes
0answers
385 views

Difficulty understanding Monte Carlo policy evaluation (state-value) for gridworld

I've been trying to read Sutton & Barto book chapter 5.1, but I'm still a bit confused about the procedure of using Monte Carlo policy evaluation (p.92), and now I just cant proceed anymore coding ...
2
votes
0answers
44 views

How does Hindsight Experience Replay cope with multiple goals?

What if there are multiple goals? For example, let's consider the bit-flipping environment as described in the paper HER with one small change: now, the goal is not some specific configuration, but ...
2
votes
0answers
87 views

Reinforcement learning for inventory management with dynamic changes to available products

Consider a shop owner who has to deal with having to buy for one week from a different supplier with several different brands. Another week a brand is removed or added from the market. Yet another ...
2
votes
0answers
157 views

Is Reinforcement Learning the future of Natural Language Processing?

I was reading about the grounding problem after seeing it mentioned in another answer today. The article states that, in order to avoid the "infinite regress" of defining all words with other words, ...
2
votes
0answers
69 views

Eligibility trace In Model-based Reinforcement Learning

In model-based reinforcement learning algorithms, the model of the environment is constructed to efficiently use samples, models such as Dyna, and Prioritize Sweeping. Moreover, eligibility trace ...
2
votes
0answers
38 views

How to include exploration in Gaussian policy

When dealing with continuous action spaces, a common choice when designing a policy in policy gradient methods is to learn mean and variance of actions for a specific state and then simply sample from ...
2
votes
0answers
154 views

Reward discounting in reinforcement learning for a Pong game

I am trying to understand how to train a neural network to win a Pong game using reinforcement learning, by following the blog post Spinning up a Pong AI with deep reinforcement learning. The ...
2
votes
0answers
59 views

Understanding the proof of theorem 2.1 from the paper “Efficient reductions for imitation learning”

I am trying to understand the proof of theorem 2.1 from this paper: Ross, Stéphane, and Drew Bagnell. "Efficient reductions for imitation learning." Proceedings of the thirteenth international ...
2
votes
0answers
373 views

How many episodes does it take for a vanilla one-step actor-critic agent to master the OpenAI BipedalWalker-v2 problem?

I'm trying to solve the OpenAI BipedalWalker-v2 by using a one-step actor-critic agent. I'm implementing the solution using python and tensorflow. I'm following this pseudo-code taken from the book ...
2
votes
0answers
67 views

Why do we have to solve MDP in each iteration of Maximum Entropy Inverse Reinforcement Learning?

Gradient in Maximum Entropy IRL requires to find the probability of expert trajectories given the reward function weights. This is done in the paper by calculating state visitation probabilities but I ...
2
votes
0answers
69 views

Algorithms for multiple agents problems

Can anyone recommend a reinforcement learning algorithm for a multi-agent environment? In my simplified example, I'm implementing a Q-Learning system with different 10 agents. The agents compete for ...
2
votes
0answers
76 views

When can we say an RL algorithm learns an Atari game?

If an Atari game's rewards can be between $-100$ and $100$, when can we say an agent learned to play this game? Should it get the reward very close to $100$ for each instance of the game? Or it is ...
2
votes
0answers
259 views

Extend the loss function from the single action to the n-action case per time step

My question concerns a side question (which was not answered) asked here: Policy gradients for multiple continuous actions I am trying to implement a simple policy gradient algorithm for a discrete ...
2
votes
0answers
88 views

Which features and algorithm could optimize this air-conditioner problem?

Imagine we have 2 air conditioner systems (AA) and 2 "free cooling" systems which mix external and internal air (FC) in a closed box which always tends to warm up. For each system, we have to find ...
2
votes
0answers
172 views

Is iLQG a good algorithm for model-based planning with simple environments?

In their work Continuous Deep Q-Learning with Model-based Acceleration, the author demonstrate great results of applying Imagination Rollouts for model-based acceleration of learning. They test their ...
2
votes
1answer
217 views

Is it expected that during self-play reinforcement learning that player 1 or player 2 wins the majority of games?

I'm testing various learning rates and neural network configurations. I'm testing over 10000 games, with the first 2000 having random starting moves and general randomness throughout of about 20%, i.e....
1
vote
0answers
15 views

Is there any research on the application of policy gradients to problems where the selection of an action requires the selection of another one?

I am working on a problem and want to explore if it can be solved with PPO (or other policy gradient methods). The problem is that the action space is a bit special, compared to classic RL ...
1
vote
0answers
10 views

Understanding advantage estimator in proximal policy optimization

I was reading Proximal Policy Optimization paper. It states following: The advantage estimator used is: $\hat{A}_t=-V(s_t)+r_t+\gamma r_{t+1}+...+\gamma^{T-t+1}r_{T-1}+\color{blue}{\gamma^{T-t}}V(s_T)...
1
vote
0answers
14 views

How would I design a finite budget, cascaded multi agent deep reinforcement learning model?

In most of the multi-agent reinforcement learning models I've found, it seems to generate the observations for each of the agents simultaneously and then uses a centralized critic to assess all of the ...
1
vote
0answers
43 views

What's the best way to take a list of lists as DQN input?

I have my own environment for the DQN algorithm. In my environment, the state space is represented by a list of lists, where each sublist can be of different lengths. In my case, the length of the ...
1
vote
0answers
47 views

Which RL algorithm would be suitable for this multi-dimensional and continuous action space?

Is there an RL approach/algorithm that would be suited for the following kind of problem? There is a continuous action space with an action value $A_{a,t}$ for each action dimension $a$. The ...
1
vote
0answers
46 views

Why is Openai's PPO2 implementation differentiable?

I'm trying to understand the concept behind the implementation of the OpenAI PPO2 algorithm. The loss function that is minimized is as follows: ...
1
vote
0answers
47 views

How should I simulate this Markov Decision Process?

I am working on solving a problem on nodes in a graph communicating with each other. They try to estimate a central state using Kalman consensus filter, with the connections described by the graph's ...
1
vote
0answers
29 views

PPO2: Intuition behind Gumbel Softmax and Exploration?

I'm trying to understand the logic behind the magic of using the gumbel distribution for action sampling inside the PPO2 algorithm. This code snippet implements the action sampling, taken from here: <...
1
vote
0answers
22 views

Is (log-)standard deviation learned in TRPO and PPO or fixed instead?

After having read Williams (1992), where it was suggested that actually both the mean and standard deviation can be learned while training a REINFORCE algorithm on generating continuous output values, ...
1
vote
0answers
31 views

Confusion about computing policy gradient with automatic differentiation ( material from Berkeley CS285)

I am taking Berkeley’s CS285 via self-study. On this particular lecture regarding Policy Gradient, I am very confused about the inconsistency between the concept explanation and the demonstration of ...
1
vote
0answers
29 views

Understanding loss function gradient in asynchronous advantage actor-critic (A3C) algorithm

This is a question I posted here. I am asking it on this StackExchange branch as well, so that more people who could potentially answer get to see the question. In the A3C algorithm from the original ...
1
vote
0answers
23 views

Gradual decrease in performance of a DDPG agent

I'm trying to solve the OpenAI's CarRacing-v0 environment with the DDPG algorithm. I've observed that after a period of learning, the agent's performance starts to deteriorate slowly. For some ...
1
vote
0answers
47 views
1
vote
0answers
37 views

Understanding policy gradient derivation with importance sampling

I was watching a lecture on Policy Gradient and had difficulty following it when importance sampling was introduced. It was shown that the gradient of the objective can be written as $$\nabla_\theta U(...
1
vote
0answers
29 views

Difference in average rewards between taking random actions and following random policies

I wrote two programs that simulated 10000 episodes in gym environment CartPole-v0. The first program takes random moves in every steps in each episode. The average reward over 10000 episodes is 22....
1
vote
0answers
68 views

Is object-based representation of the observation space feasible?

I just started working on a DRL project from scratch. The state of each episode can be expressed as a state set $S=(S^A, S^B, S^C, S^D)$. Each subset is a feature set of a constituent component of the ...
1
vote
0answers
70 views

How do I learn the value function for a POMDP with a single-step horizon (bandit)?

Consider a POMDP with a finite number of environment states, $|\mathcal{S}| = N$, but the number of belief states is uncountably infinite. The belief state space is the convex hull of an $N$ simplex. ...
1
vote
0answers
40 views

Are actions deterministic during testing in continuous action space PPO?

In a continuous action space (for instance, in PPO, TRPO, REINFORCE, etc.), during training, an action is sampled from the random distribution with $\mu$ and $\sigma$. This results in an inherent ...
1
vote
0answers
20 views

Why do we use big batch/epoch size in policy gradient methods (vpg specifically)?

I am re-implementing vpg and using Spinning Up as reference implementation. I noticed that the default epoch size is 4000. I also see cues in papers that big batch size is quite standard. My ...
1
vote
0answers
39 views

Is training on single game each time appropriate for an agent to learn to play checkers

I was facing a problem I mentioned in a previous question but after a while, I realize that maybe the problem in the dataset not in the learning rate. I build the dataset from white positions only i.e ...
1
vote
0answers
77 views

learning rate and credit assignment problem in checkers

I have implemented an AI agent to play checkers based on the design written in the first chapter of Machine Learning, Tom Mitchell, McGraw Hill, 1997. We train the agent by letting it plays against ...
1
vote
1answer
72 views

Model-based RL for time series data

I have time-series data. When I take an action, it impacts the next state, because my action directly determines the next state, but it is not known what the impact is. To be concrete: I have $X(t)$ ...
1
vote
0answers
66 views

Designing Policy-Network for Deep-RL with Large, Variable Action Space

I am attempting a project involving training an agent to play a game using deep reinforcement learning. This project has a few features that complicate the design of the neural network: The action ...
1
vote
0answers
23 views

Understanding example for Improved Policy Iteration for POMDPs

I was going through this paper by Hansen. This paper proposes policy improvement by first converting set of $\alpha$ vectors into finite state controller and then comparing them to obtain improved ...
1
vote
0answers
39 views

DDQN Agent in Othello (Reversi) game struggle to learn

This is my first question on this forum and I would like to welcome everyone. I am trying to implement DDQN Agent playing Othello (Reversi) game. I have tried multiple things but the agent which seems ...
1
vote
0answers
63 views

What kind of reinforcement learning method does AlphaGo Deepmind use to beat the best human Go player?

In reinforcement learning, there are model-based versus model-free methods. Within model-based ones, there are policy-based and value-based methods. AlphaGo Deepmind RL model has beaten the best Go ...
1
vote
0answers
58 views

How to deal with KerasRL DDPG algorithm getting stuck in a local optima?

I am using KerasRL DDPG to try to learn a policy on my own custom environment, but the agent is stuck in a local optima although I am adding the OrnsteinUhlenbeck randomization process. I used the ...
1
vote
0answers
59 views

Stack of Planes as the Action Space Representation for AlphaZero (Chess)

I have a question regarding the action space of the policy network used in AlphaZero. From the paper: We represent the policy π(a|s) by a 8 × 8 × 73 stack of planes encoding a probability ...
1
vote
1answer
70 views

How to deal with the time delay in reinforcement learning?

I have a question regarding the time delay in reinforcement learning (RL). In the RL, one has state, reward and action. It is usually assumed that (as far as I understand it) when the action is ...

1
3 4
5
6 7
12