Questions tagged [reinforcement-learning]
For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.
2,142
questions
0
votes
0
answers
7
views
How can we approximate infinite horizon MDP with finite horizon MDP in the context of reinforcement learning?
For a given value of "discount factor" (and reward values' range) in fixed finite horizon markov decision process (MDP), upto how many episodes we have to extend this MDP so that we can ...
1
vote
0
answers
18
views
off-policy Monte Carlo learning: Why is Probability of Sampling a Trajectory the same as Having a return?
In Sutton and Barto's RL book, in the section for off-policy learning, we would like to find the expected value of the random variable $G_t$, given $S_t = s$ under our target policy: $$E_{\pi}[G_t|S_t ...
- 11
0
votes
0
answers
11
views
PPO continous action space working in a complex scenario but failing to work in a simple scenario
I tried solving supply chain optimization problem using RL discrete and continuous actipn space. For some reason, with simplified version of problem (i.e. if customer order is always equal to 1), how ...
- 111
0
votes
1
answer
18
views
Detecting object position given the relative position of another object
I know that the title might be redundant but I'm trying to understand if there is way to predict where a specific object will be if I provide a certain object as a reference.
See as an example the ...
- 1
0
votes
0
answers
35
views
How does having zero advantage help with identifiability?
I am reading the D3QN paper and they have the following paragraph -
Equation (7) is unidentifiable in the sense that given Q we cannot
recover V and A uniquely. To see this, add a constant to V (s; θ,...
- 341
0
votes
0
answers
31
views
Implementation of DQN
Good day
I attempted to implement DQN from scratch to solve the cartpole problem, the Tested my neural network class on the XOR table and it worked so I'm assuming the issue isn't with the neural ...
0
votes
0
answers
6
views
Frozen baseline for policy gradient rewards
I have a continuous reinforcement learning problem for which I use policy gradients and I use a baseline to decrease the variance of the gradients.
The baseline that I used is the moving average of ...
- 1
0
votes
0
answers
10
views
Reinforcement Learning with sparse/delayed reward - should intermediate rewards be decayed over time/training?
I'm thinking of a situation like a game (say, chess) where the real objective/reward is actually determined at the very end.
I understand that it's important/helpful to do reward shaping with ...
- 302
0
votes
0
answers
29
views
Is it possible to compare Machine Learning algorithms on an abstract level without a specific use case?
I've got the task of comparing some ML algorithms at an abstract level on an argumentative basis. I wonder if this is possible in general without a specific use case (it is derived from ML in ...
- 1
0
votes
0
answers
36
views
How do I improve the reward of policy gradient network when multiple states and actions exist per time step?
I am working on a project, in which I'm using a policy gradient algorithm (REINFORCE) to select the best cleaning method/methods for erroneous samples in tabular datasets.
The details are as follows.
...
- 1
2
votes
2
answers
46
views
+50
Are batches useful for REINFORCE without strong episode cutoffs?
I'm following along with PyTorch's example implementations (found here) of reinforcement learning algorithms that happen to be largely REINFORCE (vanilla policy gradient) based, and I notice they don'...
- 79
1
vote
0
answers
13
views
In the policy gradient method, state dependent baseline does not affect gradient of the objective function. Then how this is better approach?
In policy gradient theory subtracting state dependent baseline from Q(s,a) does not affect gradient of the objective function. I understand the proof shown below. One things is, if it's not affecting ...
0
votes
0
answers
20
views
Reinforcement Learning with PPO - entropy loss dropping, but so is performance. Why?
I'm using PPO with an action-mask and I'm encountering a weird phenomenon.
At first during training, the entropy loss is decreasing (I interpret this as less exploration, more exploitation, more "...
- 302
0
votes
0
answers
9
views
overcoming the original policy with offline RL
I am doing a comparison between RL and a metaheuristic algorithm. What I have found is that online RL does a good work but does not overcome the latter algorithm. In this case generating the samples ...
- 49
0
votes
0
answers
25
views
Can action space be part of the state space?
I'm working on a project where I have access to position coordinates and velocity components of multiple agents in an environment. Assuming that one agent is controllable while others are not ...
- 133
0
votes
0
answers
12
views
What is the meaning about the $\alpha$ in TD3 algorithm
I am study the paper with TD3 algorithm.
I am curious about the meaning of $\alpha$ while the paper prove that overestimation will be happened in a critical situation.
The contents about mathematical ...
- 1
0
votes
0
answers
18
views
How to design rewards in RL?
I am a bit confused regarding rewards in reinforcement learning. In my quite simple environment, where the agent has to find it's way to a target and kill it, the agent has control over heading ...
- 31
0
votes
0
answers
12
views
Reward Function for Reinforcement Learning model
I am trying to create a reinforcement learning model to control the acceleration of a car. I am designing the model such that initially the acceleration is provided and then deceleration is provided ...
1
vote
1
answer
34
views
How to correctly evaluate the state value of this simple markov decision process?
For some contexts, I'm working on a c# library for reinforcement learning. I implemented two methods to evaluate a state value function, namely the TD(0) method and the Monte Carlo first visit method.
...
- 113
0
votes
0
answers
22
views
Bouding the state using the reward in RL
I'm wondering what the common approaches are bounding out state $s\in\mathbf{R}$ to some values $\in[s_0,s_1]$ is required. So in my case, for example, the state represents an angle of rotation, that ...
- 227
1
vote
1
answer
32
views
How can rewards and loss calculation be extended to multiple agents in a vanilla policy gradient RL setting?
Say I have a simple multi-agent reinforcement learning problem using vanilla policy gradient methods (i.e. REINFORCE) that is currently running with one network per agent. If I can say that each of my ...
- 79
0
votes
1
answer
31
views
How to deal with delay in reinforcement learning, an unclear case
According to the question in How to deal with the time delay in reinforcement learning?, we can tell the delay in the reinforcement learning can be observation delay, action delay and reward delay.
I ...
- 1
0
votes
1
answer
48
views
What modifications can maximize the efficacy of the REINFORCE algorithm for a policy gradient task?
I am straying out of my domain knowledge to attempt a basic reinforcement learning task in a toy environment and have become fairly familiar with the REINFORCE algorithm for policy gradient agents, ...
- 79
3
votes
2
answers
62
views
How do I derive Sutton and Barto Equation 3.14?
I'm having trouble going to the 2nd to last line of (3.14),
http://incompleteideas.net/book/RLbook2020.pdf#page=81
$$
\require{enclose}
\begin{aligned}
v_{\pi}(s) & \doteq \mathbb{E}_{\pi}\left[...
- 133
0
votes
0
answers
13
views
What are the set of actions in reinforcement learning?
I'm reading https://ai.stanford.edu/~ang/papers/icml04-apprentice.pdf
In 2. Preliminaries, they claim that the reward function takes a state in S to an action
$$R : S \rightarrow A$$
But in the next ...
- 133
0
votes
1
answer
69
views
How is reinforcement learning applied in the real industry?
I'm a newbie to reinforcement learning.
While studying reinforcement learning, a question arose about how to apply reinforcement learning in the real world.
Assuming that a reinforcement learning ...
2
votes
0
answers
58
views
Deep reinforcement learning the board game "Battle Sheep" - too large action space?
I was recently introduced to this simple board game called "Battle Sheep". In this game, two to four players try to acquire as many hex tiles from a hex grid as possible.
You can find the ...
0
votes
0
answers
12
views
Can TRPO use replay buffers
I understand that TRPO is a on-policy RL method and that it optimizes an expectation of the advantage or accumulated returns function over actions taken according to policy pi. Is it possible to use a ...
- 1
1
vote
0
answers
16
views
How many parameters of a RL environment?
I’m working at a Reinforcement Learning model, using PPO algorithm, in which the agent has 4 possible actions, acting in a stochastic environment defined by 3 parameters.
Given its stochasticity, I ...
-1
votes
0
answers
10
views
What parameters are used during deployment of an RL model
In academia RL algorithms are run from multiple random seeds and the results from these seeds are plotted with the mean and std. However, in real life applications how do we deploy RL models, train ...
- 299
0
votes
1
answer
36
views
Should the concept of discounted rewards result in multiple arrays per episode in RL?
Note that I'm coming from mostly only working with the REINFORCE algorithm, but I've typically seen discounted rewards calculated in a way that looks like below:
Say you have a reward array of length <...
- 79
0
votes
0
answers
34
views
Shaping reward so that it maximizes multiple components together
I am fairly new to RL and I compete in AWS DeepRacer student league. The main task there is to create a reward function. All the hyperparameters and action space are fixed. So far, I know how to shape ...
2
votes
1
answer
35
views
Why might my policy gradient agent appear to maximize the absolute value of rewards?
I have a toy policy gradient RL algorithm using REINFORCE (aka monte carlo policy gradients) that involves bots moving on a grid attempting to "acquire" targets in Pytorch. The bots receive +...
- 79
0
votes
0
answers
16
views
What are some approaches for specifying goals for deep-RL agents?
I'm wondering what are the approaches for specifying goals for a trained deep-RL in deployment? E.g. how to tell a car drive agent to go to location $y$?
To elaborate, I understand that, for example, ...
0
votes
1
answer
14
views
How to use RL on a robotic moving arm?
I'm working on a simulation of a motor that is attached to a wing (Later, this will also have a real-life counterpart once I'll assemble all the components in our lab), and I can control the forces/...
- 227
0
votes
0
answers
28
views
Define possible?
In Reinforcement Learning, policies are defined in terms of possible actions (see for instance page 58 of the book by Sutton et al.). So, is any action that an agent has in its repertoire always "...
0
votes
0
answers
61
views
Reward shaping for an autonomous driving car (AWS DeepRacer)
From past 2 months, I am competing in AWS Deepracer student league. I was new to RL but had some knowledge in supervised and unsupervised learning. In the league the hyperparameters and action space ...
0
votes
0
answers
14
views
Learning the optimal force to apply using RL
I have a robotic system with a motor that is attached to a wing that rotates along the motor's main axis $\hat{z}$. I can formulate the torque's equation on the wing as $I\ddot\phi=\tau_z-\tau_{drag}$,...
- 227
2
votes
0
answers
33
views
For REINFORCE, why do different books give different algorithms?
The discount rate appears twice in the REINFORCE algorithm in Sutton and Barto (2018). However, in three major books (Graesser and Keng (2020); Morales (2020); Ravichandiran (2020)) on reinforcement ...
-1
votes
1
answer
21
views
What are "feature planes" in neural networks? (current context is deep reinforcement learning)
The input contains 14 feature planes, each of shape 11x11
What does this mean?
- 9
0
votes
0
answers
23
views
When use positive or negative rewards in reinforcement learning? Is there anything in literature?
Let's say I can design a reward as function of a distance $d>0$ from the target in 2 ways: $r=\frac{1}{1+d}$ or $r=-d$. The first is defined in $(0,1]$ the second in $(-\infty,0]$. I would expect ...
- 41
0
votes
0
answers
13
views
Problem with continual learning in RL
I've got a vague question, I trained model in continual learning manner on 3 environments (using both SAC and TD3), taking the last model from the previous environment and performing additional ...
0
votes
0
answers
17
views
Actor-Critic Algorithm - how to design and train $V$
Background
I've been watching the lecture of Standford's CS231n course, and one of the lectures (No. 14) is about RL. After talking about Q-Learning and Policy Gradients (REINFORCE algorithm), they ...
- 141
1
vote
0
answers
18
views
How to compare different trajecories in a Markov Decision Process
I realize that my question is a bit fuzzy and I am sorry for that. If needed, I will try to make it more rigorous and precice.
Let $\mathcal{M}$ be a Markov Decision Process, with state space $\...
- 173
0
votes
0
answers
21
views
how to appy action mask
I'm trying to figure out how action masking works and the closest workaround i get is following the hanabi environment example and then write my own custom version ...
1
vote
0
answers
17
views
Proof that the Policy Iteration Converges?
Let $\mathcal{X}=:\{x_1, x_2, x_3,...,x_n\}$ be the state space. Let $\mathcal{U}:=\{u_1, u_2, u_3,...,u_m\}$ be the set of actions. Let $A^{u_1}, A^{u_2}, A^{u_3},...,A^{u_m}$ be the state transition ...
0
votes
1
answer
36
views
Are there limitations on network output architecture and action mapping in reinforcement learning?
I'm easing my way into a toy reinforcement learning problem where my objects can move and take different actions on a simple grid, but I'm having trouble understanding what constraints might exist in ...
- 79
0
votes
0
answers
13
views
Why does a quantile regression estimator underestimate the variance when using the quantile huber loss?
I have a question to quantile regression which is related to distributional Reinforcement Learning. Let the quantile loss (QL) be defined as
\begin{align*}
\mathcal{L}^{\tau}_{\text{QR}}(\...
- 35
3
votes
1
answer
76
views
Does the AlphaZero algorithm keep the subtree statistics after each move during MCTS?
This question is regarding the Monte Carlo Tree Search (MCTS) algorithm presented in the AlphaZero paper (arXiv). As described in the paper, each MCTS used 800 simulations to determine the next action....
- 33
0
votes
0
answers
24
views
Is it possible to train street fighter agents in gym retro environments?
Is it possible to train street fighter 2 champion edition agents to play against CPU in gym retro. I used the stable baseline package and after training the model, it seems like there is no difference ...
- 1