Questions tagged [reinforcement-learning]
For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.
1,986
questions
0
votes
0
answers
7
views
Tensorflow Reinforcement Learning Framework for External Environment
Hi everyone and sorry for the long question, I am trying to solve an environment that has no simulation and cannot be interacted with, but it can consume web service. so my current design for the RL ...
- 3
1
vote
0
answers
19
views
Why is training longer not better in reinforcement learning?
I have trained an RL agent (PPO) for 6 million steps to solve the OpenAI gym LunarLander-v2. Surprisingly, the agent performs best already after 320K steps and is getting worse after that.
In the ...
- 203
0
votes
0
answers
17
views
Why should the learned pseudometric be meaningful?
I'm going through the paper 'Offline Reinforcement Learning with Pseudometric Learning' by Dadashi et al. The authors define the following operator to learn a distance function:
$$\mathcal{F}(d)(s_{1},...
- 35
0
votes
0
answers
9
views
Are there assumptions made about Self-Play that don't hold up in regular MA competition?
I read about this paper Efficient Competitive Self-Play Policy Optimization which proposes an algorithm for training a population of agents with self-play using a perturbation based matchmaking ...
- 85
3
votes
1
answer
44
views
What should the discount factor for the non-slippery version of the FrozenLake environment be?
I was working with FrozenLake 4x4 from open AI gym. In the slippery case, using a discounting factor of 1, my value iteration implementation was giving a success rate of around 75 percent. It was much ...
2
votes
1
answer
26
views
Can we also estimate $V_{\pi}$ with SARSA?
For SARSA, I know we can estimate the action value $Q(s,a)$, and the relationship between $V(s)$ and $Q(s,a)$ is $V_{\pi}(s) = \sum_{a \in \mathcal{A}} \pi(a|s)Q_{\pi}
(s,a)$.
So my question is, can ...
- 21
0
votes
1
answer
36
views
How to manage impossible actions? [closed]
I am using Q-learning in julia language.
Because of the solver’s configuration, actions have to be defined as the whole action space and impossible actions have to be also considered. It means that I ...
- 53
0
votes
1
answer
33
views
Could Softmax Action Selection be useful to solve an episodic task with more than 100000 possible states and 2000 actions?
I am new in the field of RL. I am trying to use tabular methods, Q-Learning for solving a problem with more than 100000 possible states and 2000 actions. It is an episodic task.
It takes a lot of time ...
- 53
0
votes
1
answer
27
views
Is it possible to successively train an RL agent on the same environment with different data
I have a scheduling problem that I am trying to solve with RL (if you are interested in more details you can read about it here Reinforcement learning applicable to a scheduling problem?).
I have ...
- 213
2
votes
0
answers
19
views
How should $\epsilon$ decay be done?
I am new to reinforcement learning and I am trying to understand the relationship between its hyperparameters.
When I use tabular methods, like Q-learning and SARSA, I know that for SARSA the agent ...
- 53
0
votes
0
answers
20
views
How to find probability and value functions of blackjack states [closed]
Consider a game of blackjack:
A player is in the middle of a game, with a hand of cards that sum to 18, no usable ace and the
dealer’s card is a 10.
If the player decides to hit, what are the possible ...
1
vote
1
answer
39
views
Can you make a Neural Network drunk or high?
We know that the human brain can become sozzled by various substances that are released into the brain, but can you make an artificial neural network drunk or high? For example on a RL Agent that ...
- 19
2
votes
1
answer
45
views
Is it possible learning convergence is lost in Reinforcement Learning as the state space grows? What about catastrophic forgetting?
I am new in the AI field and I am trying to use Reinforcement Learning, Q-Learning and Sarsa algorithms to solve a sequential decision making problem. When I apply those algorithms to a problem with 6....
- 53
0
votes
0
answers
19
views
If we have a working reward function, would adding another action have a significant effect on the agent performance if task remains the same?
If we have a working reward function, providing the desired behavior and optimal policy in a continuous action/state-space problem, would adding another action significantly affect the possible ...
- 1
2
votes
1
answer
147
views
Is it appropriate to represent 'total failure' as an absorbing state?
My understanding is that, in Markov decision processes, absorbing state are states which can transition only to themselves and that these transitions generate rewards of 0. I know that absorbing ...
- 121
3
votes
1
answer
40
views
What is the total number of actions and rewards count
Reading this two articles about Reinforcement Learning:
Deep Reinforcement Learning with Double Q-learning by Hado van Hasselt et al.
Human-level control through deep reinforcement learning by ...
- 133
1
vote
2
answers
49
views
Given a set of trajectories produced by a fixed policy, what is the the standard approach to estimate Q?
Let's say that I have a set of trajectories $\mathcal{D} = \{\tau_1, \dots, \tau_n\}$ produced by an agent acting in a (episodic) MDP with a fixed policy $\pi$. I would like to estimate the $Q$ ...
- 163
-1
votes
1
answer
48
views
What is so special about the Bellman Optimality Principle?
In the context of Decision Making and Game Theory, "Bellman's Equations and Bellman's Conditions of Optimality" are said to be some of the most important mathematical principles in this ...
- 287
0
votes
0
answers
7
views
Is there a way to retain the environment order/sequence of generated observations in the driver and replay buffer
I am using the tf_agents for contextual bandit algorithm. My data is at userlevel and hence very important to make sure that actions and rewards (and trajectories in order to train on them) are ...
- 111
1
vote
1
answer
35
views
How to handled delayed rewards in contextual bandits [closed]
All the examples I see in the tf_Agents for contextual bandits, involves a reward function we generated the reward instantly after an observation has been generated.
But, in my real world usecase (say ...
- 111
1
vote
1
answer
38
views
Are there any guidelines on how to map the state space to integers in the case tabular RL algorithms?
Let's say that you want to solve a problem with a tabular reinforcement learning algorithm, for example, Q-learning. You can represent the value function $Q(s, a)$ as a $|\mathcal{S}|\times |\mathcal{...
- 33k
1
vote
0
answers
40
views
Why do off-policy algorithms suffer from worse computational or time efficiency compared to on-policy algorithms?
When I run Soft-Actor-Critic (off-policy) in my Environment, the calculation of gradient updates takes almost twice the time compared to using PPO (on-policy).
I also saw that ACER has a higher time ...
- 85
2
votes
0
answers
52
views
How to reduce the dimensionality of the actions in RL
I have a single-agent RL model in which the dimension of the dimension of the action space is $70$. This action space is too big and the deep RL agent is not learning properly. The boundaries of the ...
- 49
0
votes
0
answers
18
views
Does policy entropy improve the Signal-to-Noise Ratio (SNR) of the gradient estimate?
I came across a reddit post in which John Schulman claims the following:
Increasing the policy entropy improves the SNR of the gradient estimator at the cost of increasing bias (computing the wrong ...
- 101
1
vote
1
answer
19
views
If the probabilities with which each task is selected for you do not change over time, why would it appear as a single stationary k-armed bandit task?
Sutton-Barto (Section 2.9-Associative Search (Contextual Bandits), page 41):
As an example, suppose there are several different k-armed bandit tasks, and that on
each step you confront one of these ...
- 149
1
vote
1
answer
29
views
Gradient bandit algorithm: is $\bar{R}_t$ average of all rewards or average of rewards corresponding to $A_t$?
Sutton-Barto (Section 2.8-Gradient Bandit Algorithms, page 37):
Question: is $\bar{R}_t$ average of all rewards or average of rewards corresponding to $A_t$?
- 149
0
votes
0
answers
19
views
Is there a multi-task RL algorithm that supports different action spaces for each agent?
I'm currently working on a project in which I need apply multi-task reinforcement learning. Over the same state space, each agent aims to do a separate task, but the action spaces of agents are ...
- 1
2
votes
1
answer
67
views
How to construct a reward function for a "wait and see" problem
I'm working on a problem that I think could probably be represented as a reinforcement learning task, but I'm uncertain about how to design the reward function. The core task is essentially a ...
- 21
1
vote
1
answer
47
views
Reinforcement learning algorithms for large problems that are not based on a neural network
I have a large control problem with multidimensional continuous inputs (13) and outputs (3). I tried several Reinforcement learning algorithms like Deep-Q-Networks (DQN), Proximal Policy Optimization (...
- 213
0
votes
0
answers
19
views
When Hindsight Experience Replay is deployed, does the input need to be augmented as well?
In Hindsight Experience Replay (HER), we augment the state representation $s_t$ with some goal $g_T$, which corresponds to the state reached after $T$ steps, such that $s'_t = s_t || g_T$. Later, some ...
- 71
1
vote
1
answer
52
views
How should I write the reward function to teach the agent the rules of this card game?
I'm quite new to reinforcement learning. I've been training the model for the following problem but the mean reward is stuck.
In a 5 by 5 board, each position can contain a card with a color (0-4) ...
- 111
0
votes
1
answer
31
views
How is policy iteration capable of improving on a deterministic policy?
Given a policy $\pi$ and the improved version upon it using policy iteration $\pi'$ we have, for $\forall s \in S$, $v_{\pi'}(s)\geq v_{\pi}(s)$.
I think the way we choose $\pi'$ makes it ...
- 489
1
vote
1
answer
36
views
How to model a multi-agent reinforcement learning problem where actions of different agents can take different durations?
I am confused on a conceptual scale how I would be able to model a multi-agent reinforcement learning problem when each agent performing an action would take different durations to complete the action....
- 203
0
votes
1
answer
24
views
Does the policy search work if there is no state to state dependency through actions?
There is a game in which the state comes one after the other without depending on the agent's action. The agent gets a reward for its actions at the end of the game. The goal of the agent is to reach ...
1
vote
2
answers
68
views
What is the difference between a policy and rewards?
I don't understand the difference between a policy and rewards. Sure, a policy tells us what to do, but isn't the output of a neural network trained on rewards basically a policy (i.e. choose the ...
1
vote
0
answers
17
views
Can vanilla multi armed bandit problems be solved by RL algorithms like A2C and PPO?
Let's say we have N bandit machines with some distributions (assume some are gaussian, some are uniform, some are chi squared). We want to maximize rewards in X amount of time. I am aware that ...
- 149
0
votes
1
answer
33
views
Does $S_{t+1}$ denote the future information in Q-learning?
In Q-learning, $Q(S_t,a)$ is updated by the Bellman equation. $Q(S_t,a) = r + \max_{a'}(Q(S_{t+1},a'))$ where $S_{t+1}$ is the future state.
Let's say $S$ denotes the stock price, does it mean we are ...
- 1
0
votes
0
answers
10
views
Knowing the futility of discounting in continuing problems, how can we say discounting has no role in control problems with function approximation?
Sutton-Barto (Section 10.4, page 254):
Based on the futility of discounting in continuing problems, how can we conclude that discounting has no role to play in control problems with function ...
- 149
1
vote
0
answers
41
views
What is commonly done for standardization/normalization of the targets in Deep Q-Learning?
I have been searching a lot about standardization/normalization of rewards and targets for the DQN algorithm. For the rewards, I now use the gym wrapper, which only scales but not shifts the rewards ...
- 23
0
votes
0
answers
42
views
In reinforcement learning, how to craft observation space when environment is made of multiple blocks?
In reinforcement learning problems like cartpole, usually the environment is a single system that takes an input and gives an output. For example, in cartpole, given positions, velocities as input we ...
- 149
0
votes
1
answer
31
views
Does $R_{s}=E[R_{t}|S_{t}=s]$ indicate the reward we might expect on getting on average moving from any other state to $s$?
I'm trying to understand correctly what each "variable" in RL is and I'm not sure about $R_{s}$ the reward function. I used to think that it's the reward we may expect on average after ...
- 489
2
votes
1
answer
166
views
Determining to terminate at a reward or not
I am practicing the Bellman equation on Grid world examples and in this scenario, there are numbered grid squares where the agent can choose to terminate and collect the reward equal to the amount ...
- 123
0
votes
0
answers
21
views
How do you determine the optimal policy?
I am following some Grid world examples to understand reinforcement learning. I have a deterministic grid (part of which I have reconstructed below). I am trying to understand how the optimal policy ...
- 1
0
votes
0
answers
10
views
How are previous values updated when performing value iteration?
I have been trying to understand how you determine the value for each square in a grid world and I have seen/watched a few different examples to try and apply it to my own grid and I find myself ...
- 1
0
votes
2
answers
72
views
What is the Bellman equation for V(s) in the case of a deterministic environment?
I am currently trying to practice reinforcement learning for an agent on a grid. The grid is deterministic. Since the grid is deterministic, to calculate the value for each grid square from the reward ...
- 123
0
votes
1
answer
50
views
When calculating the max in DQN, do I have to calculate the Q for every possible action for a particular state?
I'm trying to implement the DQN paper using python/pytorch for my needs (https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf). I'm studying the main algorithm:
I am a bit confused about the $\gamma* \max ...
0
votes
0
answers
24
views
Determine Gridworld values
I am learning Reinforcement learning for games following Gridworld examples. Apologies in advance if this is a basic question, very new to reinforcement learning.
I am slightly confused in scenarios ...
- 123
0
votes
0
answers
23
views
Can DDPG algorithm obtain time-dependent and time-independent actions simultaneously?
I am new to Reinforcement Learning. I have been working on a problem using Deep Deterministic Policy Gradient (DDPG).
I would like to know if it is possible to apply this algorithm to an optimization ...
0
votes
0
answers
35
views
What can we learn from AlphaZero in the development towards AGI?
According to DeepMind,
AlphaZero's creative insights coupled with the encouraging results we see in other projects such as AlphaFold, give us confidence in our mission to create general purpose ...
- 111
1
vote
1
answer
45
views
How can AlphaZero be used in other industries besides gaming?
I'm an AI Engineering student from Belgium and I'm writing my bachelor thesis on the creation of a chess computer with deep reinforcement learning based on AlphaZero. My implementation can be found ...
- 111