Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

Filter by
Sorted by
Tagged with
0 votes
0 answers
7 views

Tensorflow Reinforcement Learning Framework for External Environment

Hi everyone and sorry for the long question, I am trying to solve an environment that has no simulation and cannot be interacted with, but it can consume web service. so my current design for the RL ...
user avatar
  • 3
1 vote
0 answers
19 views

Why is training longer not better in reinforcement learning?

I have trained an RL agent (PPO) for 6 million steps to solve the OpenAI gym LunarLander-v2. Surprisingly, the agent performs best already after 320K steps and is getting worse after that. In the ...
user avatar
  • 203
0 votes
0 answers
17 views

Why should the learned pseudometric be meaningful?

I'm going through the paper 'Offline Reinforcement Learning with Pseudometric Learning' by Dadashi et al. The authors define the following operator to learn a distance function: $$\mathcal{F}(d)(s_{1},...
user avatar
  • 35
0 votes
0 answers
9 views

Are there assumptions made about Self-Play that don't hold up in regular MA competition?

I read about this paper Efficient Competitive Self-Play Policy Optimization which proposes an algorithm for training a population of agents with self-play using a perturbation based matchmaking ...
user avatar
  • 85
3 votes
1 answer
44 views

What should the discount factor for the non-slippery version of the FrozenLake environment be?

I was working with FrozenLake 4x4 from open AI gym. In the slippery case, using a discounting factor of 1, my value iteration implementation was giving a success rate of around 75 percent. It was much ...
user avatar
2 votes
1 answer
26 views

Can we also estimate $V_{\pi}$ with SARSA?

For SARSA, I know we can estimate the action value $Q(s,a)$, and the relationship between $V(s)$ and $Q(s,a)$ is $V_{\pi}(s) = \sum_{a \in \mathcal{A}} \pi(a|s)Q_{\pi} (s,a)$. So my question is, can ...
user avatar
0 votes
1 answer
36 views

How to manage impossible actions? [closed]

I am using Q-learning in julia language. Because of the solver’s configuration, actions have to be defined as the whole action space and impossible actions have to be also considered. It means that I ...
user avatar
  • 53
0 votes
1 answer
33 views

Could Softmax Action Selection be useful to solve an episodic task with more than 100000 possible states and 2000 actions?

I am new in the field of RL. I am trying to use tabular methods, Q-Learning for solving a problem with more than 100000 possible states and 2000 actions. It is an episodic task. It takes a lot of time ...
user avatar
  • 53
0 votes
1 answer
27 views

Is it possible to successively train an RL agent on the same environment with different data

I have a scheduling problem that I am trying to solve with RL (if you are interested in more details you can read about it here Reinforcement learning applicable to a scheduling problem?). I have ...
user avatar
  • 213
2 votes
0 answers
19 views

How should $\epsilon$ decay be done?

I am new to reinforcement learning and I am trying to understand the relationship between its hyperparameters. When I use tabular methods, like Q-learning and SARSA, I know that for SARSA the agent ...
user avatar
  • 53
0 votes
0 answers
20 views

How to find probability and value functions of blackjack states [closed]

Consider a game of blackjack: A player is in the middle of a game, with a hand of cards that sum to 18, no usable ace and the dealer’s card is a 10. If the player decides to hit, what are the possible ...
user avatar
1 vote
1 answer
39 views

Can you make a Neural Network drunk or high?

We know that the human brain can become sozzled by various substances that are released into the brain, but can you make an artificial neural network drunk or high? For example on a RL Agent that ...
user avatar
2 votes
1 answer
45 views

Is it possible learning convergence is lost in Reinforcement Learning as the state space grows? What about catastrophic forgetting?

I am new in the AI field and I am trying to use Reinforcement Learning, Q-Learning and Sarsa algorithms to solve a sequential decision making problem. When I apply those algorithms to a problem with 6....
user avatar
  • 53
0 votes
0 answers
19 views

If we have a working reward function, would adding another action have a significant effect on the agent performance if task remains the same?

If we have a working reward function, providing the desired behavior and optimal policy in a continuous action/state-space problem, would adding another action significantly affect the possible ...
user avatar
2 votes
1 answer
147 views

Is it appropriate to represent 'total failure' as an absorbing state?

My understanding is that, in Markov decision processes, absorbing state are states which can transition only to themselves and that these transitions generate rewards of 0. I know that absorbing ...
user avatar
  • 121
3 votes
1 answer
40 views

What is the total number of actions and rewards count

Reading this two articles about Reinforcement Learning: Deep Reinforcement Learning with Double Q-learning by Hado van Hasselt et al. Human-level control through deep reinforcement learning by ...
user avatar
  • 133
1 vote
2 answers
49 views

Given a set of trajectories produced by a fixed policy, what is the the standard approach to estimate Q?

Let's say that I have a set of trajectories $\mathcal{D} = \{\tau_1, \dots, \tau_n\}$ produced by an agent acting in a (episodic) MDP with a fixed policy $\pi$. I would like to estimate the $Q$ ...
user avatar
  • 163
-1 votes
1 answer
48 views

What is so special about the Bellman Optimality Principle?

In the context of Decision Making and Game Theory, "Bellman's Equations and Bellman's Conditions of Optimality" are said to be some of the most important mathematical principles in this ...
user avatar
0 votes
0 answers
7 views

Is there a way to retain the environment order/sequence of generated observations in the driver and replay buffer

I am using the tf_agents for contextual bandit algorithm. My data is at userlevel and hence very important to make sure that actions and rewards (and trajectories in order to train on them) are ...
user avatar
  • 111
1 vote
1 answer
35 views

How to handled delayed rewards in contextual bandits [closed]

All the examples I see in the tf_Agents for contextual bandits, involves a reward function we generated the reward instantly after an observation has been generated. But, in my real world usecase (say ...
user avatar
  • 111
1 vote
1 answer
38 views

Are there any guidelines on how to map the state space to integers in the case tabular RL algorithms?

Let's say that you want to solve a problem with a tabular reinforcement learning algorithm, for example, Q-learning. You can represent the value function $Q(s, a)$ as a $|\mathcal{S}|\times |\mathcal{...
user avatar
  • 33k
1 vote
0 answers
40 views

Why do off-policy algorithms suffer from worse computational or time efficiency compared to on-policy algorithms?

When I run Soft-Actor-Critic (off-policy) in my Environment, the calculation of gradient updates takes almost twice the time compared to using PPO (on-policy). I also saw that ACER has a higher time ...
user avatar
  • 85
2 votes
0 answers
52 views

How to reduce the dimensionality of the actions in RL

I have a single-agent RL model in which the dimension of the dimension of the action space is $70$. This action space is too big and the deep RL agent is not learning properly. The boundaries of the ...
user avatar
  • 49
0 votes
0 answers
18 views

Does policy entropy improve the Signal-to-Noise Ratio (SNR) of the gradient estimate?

I came across a reddit post in which John Schulman claims the following: Increasing the policy entropy improves the SNR of the gradient estimator at the cost of increasing bias (computing the wrong ...
user avatar
1 vote
1 answer
19 views

If the probabilities with which each task is selected for you do not change over time, why would it appear as a single stationary k-armed bandit task?

Sutton-Barto (Section 2.9-Associative Search (Contextual Bandits), page 41): As an example, suppose there are several different k-armed bandit tasks, and that on each step you confront one of these ...
user avatar
1 vote
1 answer
29 views

Gradient bandit algorithm: is $\bar{R}_t$ average of all rewards or average of rewards corresponding to $A_t$?

Sutton-Barto (Section 2.8-Gradient Bandit Algorithms, page 37): Question: is $\bar{R}_t$ average of all rewards or average of rewards corresponding to $A_t$?
user avatar
0 votes
0 answers
19 views

Is there a multi-task RL algorithm that supports different action spaces for each agent?

I'm currently working on a project in which I need apply multi-task reinforcement learning. Over the same state space, each agent aims to do a separate task, but the action spaces of agents are ...
user avatar
  • 1
2 votes
1 answer
67 views

How to construct a reward function for a "wait and see" problem

I'm working on a problem that I think could probably be represented as a reinforcement learning task, but I'm uncertain about how to design the reward function. The core task is essentially a ...
user avatar
1 vote
1 answer
47 views

Reinforcement learning algorithms for large problems that are not based on a neural network

I have a large control problem with multidimensional continuous inputs (13) and outputs (3). I tried several Reinforcement learning algorithms like Deep-Q-Networks (DQN), Proximal Policy Optimization (...
user avatar
  • 213
0 votes
0 answers
19 views

When Hindsight Experience Replay is deployed, does the input need to be augmented as well?

In Hindsight Experience Replay (HER), we augment the state representation $s_t$ with some goal $g_T$, which corresponds to the state reached after $T$ steps, such that $s'_t = s_t || g_T$. Later, some ...
user avatar
  • 71
1 vote
1 answer
52 views

How should I write the reward function to teach the agent the rules of this card game?

I'm quite new to reinforcement learning. I've been training the model for the following problem but the mean reward is stuck. In a 5 by 5 board, each position can contain a card with a color (0-4) ...
user avatar
  • 111
0 votes
1 answer
31 views

How is policy iteration capable of improving on a deterministic policy?

Given a policy $\pi$ and the improved version upon it using policy iteration $\pi'$ we have, for $\forall s \in S$, $v_{\pi'}(s)\geq v_{\pi}(s)$. I think the way we choose $\pi'$ makes it ...
user avatar
  • 489
1 vote
1 answer
36 views

How to model a multi-agent reinforcement learning problem where actions of different agents can take different durations?

I am confused on a conceptual scale how I would be able to model a multi-agent reinforcement learning problem when each agent performing an action would take different durations to complete the action....
user avatar
  • 203
0 votes
1 answer
24 views

Does the policy search work if there is no state to state dependency through actions?

There is a game in which the state comes one after the other without depending on the agent's action. The agent gets a reward for its actions at the end of the game. The goal of the agent is to reach ...
user avatar
1 vote
2 answers
68 views

What is the difference between a policy and rewards?

I don't understand the difference between a policy and rewards. Sure, a policy tells us what to do, but isn't the output of a neural network trained on rewards basically a policy (i.e. choose the ...
user avatar
1 vote
0 answers
17 views

Can vanilla multi armed bandit problems be solved by RL algorithms like A2C and PPO?

Let's say we have N bandit machines with some distributions (assume some are gaussian, some are uniform, some are chi squared). We want to maximize rewards in X amount of time. I am aware that ...
user avatar
0 votes
1 answer
33 views

Does $S_{t+1}$ denote the future information in Q-learning?

In Q-learning, $Q(S_t,a)$ is updated by the Bellman equation. $Q(S_t,a) = r + \max_{a'}(Q(S_{t+1},a'))$ where $S_{t+1}$ is the future state. Let's say $S$ denotes the stock price, does it mean we are ...
user avatar
  • 1
0 votes
0 answers
10 views

Knowing the futility of discounting in continuing problems, how can we say discounting has no role in control problems with function approximation?

Sutton-Barto (Section 10.4, page 254): Based on the futility of discounting in continuing problems, how can we conclude that discounting has no role to play in control problems with function ...
user avatar
1 vote
0 answers
41 views

What is commonly done for standardization/normalization of the targets in Deep Q-Learning?

I have been searching a lot about standardization/normalization of rewards and targets for the DQN algorithm. For the rewards, I now use the gym wrapper, which only scales but not shifts the rewards ...
user avatar
  • 23
0 votes
0 answers
42 views

In reinforcement learning, how to craft observation space when environment is made of multiple blocks?

In reinforcement learning problems like cartpole, usually the environment is a single system that takes an input and gives an output. For example, in cartpole, given positions, velocities as input we ...
user avatar
0 votes
1 answer
31 views

Does $R_{s}=E[R_{t}|S_{t}=s]$ indicate the reward we might expect on getting on average moving from any other state to $s$?

I'm trying to understand correctly what each "variable" in RL is and I'm not sure about $R_{s}$ the reward function. I used to think that it's the reward we may expect on average after ...
user avatar
  • 489
2 votes
1 answer
166 views

Determining to terminate at a reward or not

I am practicing the Bellman equation on Grid world examples and in this scenario, there are numbered grid squares where the agent can choose to terminate and collect the reward equal to the amount ...
user avatar
  • 123
0 votes
0 answers
21 views

How do you determine the optimal policy?

I am following some Grid world examples to understand reinforcement learning. I have a deterministic grid (part of which I have reconstructed below). I am trying to understand how the optimal policy ...
user avatar
  • 1
0 votes
0 answers
10 views

How are previous values updated when performing value iteration?

I have been trying to understand how you determine the value for each square in a grid world and I have seen/watched a few different examples to try and apply it to my own grid and I find myself ...
user avatar
  • 1
0 votes
2 answers
72 views

What is the Bellman equation for V(s) in the case of a deterministic environment?

I am currently trying to practice reinforcement learning for an agent on a grid. The grid is deterministic. Since the grid is deterministic, to calculate the value for each grid square from the reward ...
user avatar
  • 123
0 votes
1 answer
50 views

When calculating the max in DQN, do I have to calculate the Q for every possible action for a particular state?

I'm trying to implement the DQN paper using python/pytorch for my needs (https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf). I'm studying the main algorithm: I am a bit confused about the $\gamma* \max ...
user avatar
0 votes
0 answers
24 views

Determine Gridworld values

I am learning Reinforcement learning for games following Gridworld examples. Apologies in advance if this is a basic question, very new to reinforcement learning. I am slightly confused in scenarios ...
user avatar
  • 123
0 votes
0 answers
23 views

Can DDPG algorithm obtain time-dependent and time-independent actions simultaneously?

I am new to Reinforcement Learning. I have been working on a problem using Deep Deterministic Policy Gradient (DDPG). I would like to know if it is possible to apply this algorithm to an optimization ...
user avatar
0 votes
0 answers
35 views

What can we learn from AlphaZero in the development towards AGI?

According to DeepMind, AlphaZero's creative insights coupled with the encouraging results we see in other projects such as AlphaFold, give us confidence in our mission to create general purpose ...
user avatar
  • 111
1 vote
1 answer
45 views

How can AlphaZero be used in other industries besides gaming?

I'm an AI Engineering student from Belgium and I'm writing my bachelor thesis on the creation of a chess computer with deep reinforcement learning based on AlphaZero. My implementation can be found ...
user avatar
  • 111

1
2 3 4 5
40