Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

Filter by
Sorted by
Tagged with
0
votes
0answers
7 views

How to limit the action space and normalize at the same time in PPO? (reinforcement learning)

In PPO (or TD3) how can you both determine the minimum and maximum action the agent can take (for example between 0 and 1) and also make sure that all the actions sum up to 1? In python I can use ...
0
votes
1answer
27 views

How should I initialize the weights of the neural network so that the initial policy is uniform?

I would like to train a neural network (NN) so that it learns the policy and value function for my agent. Since I am using reinforcement learning and do not want to prefer certain actions in certain ...
1
vote
0answers
25 views

What to look out for when designing an environment regarding observations?

When designing an environment, what should one look out for when designing the observation space to make the environment as easy to be learnable for an agent as possible? E.g. make sure the markov ...
0
votes
0answers
20 views

Which algorithms work in a non-stationary stochastic environment?

Currently, I am reading into the Multi-Armed-Bandit problem and found the special case of non-stationary (environment and its attributes, like the reward distribution, change over time) stochastic ...
0
votes
1answer
36 views

Why and how can the policy and value iteration methods converge to the OPTIMAL point?

I am reading Reinforcement Learning: An Introduction by Sutton & Barto. According to this textbook, as far as I understood, the authors claim that the policy and value iteration methods converge ...
0
votes
0answers
46 views

How to prove that "w will converge to TD fixed point once A is positive definite"

In Reinforcement Learning: An Introduction 2nd edition section 9.4 (p. 206), it says that when we use TD(0) as target and use semi-gradient method to update : In general, $w_t$ will be reduced ...
0
votes
1answer
46 views

When should discretization of observations be considered?

I found some literature regarding the design of action-spaces and that e.g. a discretization of continuous actions in video-game environments can be crucial for successful learning (Action Space ...
0
votes
0answers
22 views

What is the role of the observation in the standard Multi-Armed-Bandit?

I'm trying to understand how to model the environment for a MAB problem. I'm following the TensorFlow tutorial on it. In the Introduction and Environment chapter is stated: agents do not model state ...
2
votes
1answer
202 views

Where are the parentheses in the Bellman update rule?

I'm not having a lot of intuition about the equation. I have this Bellman update rule: $$v_{\pi}(s) =\sum_a \pi(a|s)\sum_{s',r} p(s',r|s,a)[r+ \gamma v_{k}(s')]$$ But where are the parenthesis? Is the ...
1
vote
0answers
16 views

How can PPO be combined with HER?

I ask because PPO is apparently an on-policy algorithm & the HER paper says that it can be combine with any off-policy algorithm. Yet I see GitHub projects that have combined them somehow? How is ...
1
vote
0answers
42 views

The results changed even though seed is fixed [closed]

I am using a reinforcement learning model for some tasks. and for the model, I am using stable_baselin3 and for the environment, I am using the gym. I made a small change in the environment and the ...
3
votes
1answer
76 views

Are multi agent or self-play environments always automatically POMDPs?

As part of my thesis, I'm working on a zero sum game with RL to train an agent. The game is a real-time game, a derivation of pong, one could imagine playing pong with both sides being foosball rods. ...
0
votes
0answers
23 views

Are these two definitions of regret in RM algorithm equivalent?

(2.1c) on p.1130 of the original paper on regret matching states that $$ D_t^i(j,k) = \frac{1}{t} \sum_{\tau=1}^t \big[ u^i(k,s_\tau^{-j}) - u^i(s_\tau) \big] \;. $$ At the end of page 5 of this ...
1
vote
1answer
26 views

Why do terms in the computation of state space size scale exponentially?

The image below is from a Berkeley AI course pdf I found. My question is, why do the terms accounting for the ghosts and pellets come in raised to the number of units? For example, there are two ...
1
vote
1answer
27 views

Does reinforcement learning lend itself well to changes in the environment due to external factors?

I've been reading about reinforcement learning, and it seems to me that reinforcement learning assumes the environment is static, and therefore that the reward for taking a particular action will be ...
0
votes
0answers
11 views

How to reduce the variance of stochastic policy gradient for continuous actions in a partially observable environment?

I am trying to implement a stochastic policy gradient for continuous actions in a partially observable CartPole environment. Specifically, only the current cart position and pole angle are visible, ...
1
vote
1answer
29 views

How to output an integer/discrete number n with a single output neuron?

Say I have a game with 4 base actions [left, right, up, down] and then a value n, which determines how many times the chosen action is repeated. For example, action = left, n = 3 -> go left 3 times....
0
votes
0answers
38 views

How to pass the rewards in zero-sum multiplayer context when using REINFORCE?

Suppose there are two players in my zero-sum game and they play in a row like chess. And I want to learn the policy function using the REINFORCE algorithm. I have doubts about passing reward values in ...
1
vote
1answer
49 views

Is it the high probability action that is always selected by the agent in REINFORCE algorithm?

Consider the following algorithm from the textbook titled Reinforcement Learning: An Introduction (second edition) by Richard S. Sutton and Andrew G. Bart While playing the game for the generation of ...
2
votes
0answers
41 views

In Policy Gradient methods, why are actions always chosen from a Gaussian in the literature?

In Sutton's 2020 Reinforcement Learning text (in chapter 13.7 Policy Parameterization for Continuous Actions) it's stated actions [may be] chosen from a normal (Gaussian) distribution. However, I ...
2
votes
2answers
169 views

Is the optimal policy the one with the highest accumulative reward (Q-Learning vs SARSA)?

I was looking at the following diagram, The reward obtained with SARSA is higher. However, the path that Q learning chooses is eventually the optimal one, isn't it? Why is the SARSA reward higher if ...
0
votes
0answers
30 views

How Does The DDQN Step/interact In The Environment?

I have made a (D)DQN Model. In this model, regardless of whether I initialize it in DDQN or DQN mode, it uses an experience replay memory. The way I gather transitions for this experience replay ...
0
votes
1answer
50 views

Why is old/off-policy data harmful to on-policy/online RL? [closed]

I ask because if RL is indeed an MDP, then there should be absolutely no problem with training an agent on any available episode roll-out data, right? Because an MDP implies for any state S, the ...
1
vote
0answers
31 views

How to use Actor-Critic RL with a categorical, state-dependent action space?

I have a problem where the agent is given an embedding vector to represent the state. Then it is also given a set of possible actions in the environment, let's say that the actions are each ...
1
vote
0answers
29 views

What method is better to use for a two-player reinforcement learning environment?

I want to create an RL agent for a mancala-type two-player game as my first actual project in the field. I've already completed the game itself and coded a minimax algorithm. The question is: how ...
1
vote
0answers
37 views

Proper way to count environment steps / frames in distributed RL architecture for algorithms like CLEAR or LASER => modified impala with replay

In classical - on-policy - vtrace/Impala algorithm env_steps are incremented every training iteration like this : ...
3
votes
1answer
71 views

How can we convert an MDP to a POMDP?

If a partially observable Markov decision process (POMDPs) is a generalisation of a (fully observable) MDP, then how exactly can we mathematically formulate an MDP as a POMDP?
0
votes
0answers
18 views

Augmented an Image with other data when training CNN

In the typical RL/MDP framework, I have offline data of $(s,a,r,s')$ of expert Atari gameplay. I'm looking to train a CNN to predict $r$ based on $(s, a)$. The states are represented by a $4 \times 84 ...
1
vote
0answers
27 views

What is the difference between the $Q_a$ calculated to update delta and those to select next action in the exploitation phase?

As the title suggests, I have a doubt about the computation of the $Q_a$ used to update the delta and the $Q_a$ used to select the next action in the exploitation phase, as shown below (source of ...
0
votes
1answer
55 views

How to derive the dual function step by step in relative entropy policy search (REPS)?

TL:DR, (Why) is one of the terms in the expectation not derived properly? Relative entropy policy search or REPS is used to optimize a policy in an MDP. The update step is limited in the policy space (...
2
votes
0answers
44 views

Watkins' Q(λ) with function approximation: why is gradient not considered when updating eligibility traces for the exploitation phase?

I'm implementing the Watkins' Q(λ) algorithm with function approximation (in 2nd edition of Sutton & Barto). I am very confused about updating the eligibility traces because, at the beginning of ...
0
votes
0answers
17 views

Are there any resources about using RL with RNN to produce Open AI Five-type of AI?

I want to make a minimal working version of Open AI Five. It seems it uses PPO with LSTM, but I don't know how to implement the actual code, and couldn't find any online tutorials for it. Are there ...
1
vote
0answers
17 views

Is there a paper/article on contextual $\epsilon$-greedy algorithm?

I am reading the paper A Contextual-Bandit Approach to Personalized News Article Recommendation, where it refers to $\epsilon$-greedy (disjoint) algorithm. I suspect, that it is just a version of a K-...
0
votes
0answers
27 views

How to normalize rewards in DQN?

I want to use a Deep Q-Network for a specific problem. My immediate rewards ($r_t = 0$) are all zeros. But my terminal reward is a large positive value $(r_T=100$). How could I normalize rewards to ...
1
vote
0answers
18 views

Should I represent my reinforcement learning as an episodic or continuous task?

I would like the community to help me understand if the following example would be better represented as episodic or continuous task, this will help me structure the problem and chose the right RL ...
1
vote
0answers
21 views

Why Acme is using own uniform initializer?

Why is Acme using own initializer for both tanh and ELU, when commonly used for tanh is Xavier and for ELU is He initializer? What mathematics is behind them? Here is the code. ...
1
vote
1answer
28 views

How to mix grid matrix and explicit values when designing RL state?

I'm trying to do multi-agent reinforcement learning on the grid world navigation task where multiple agents try to collectively reach multiple goals while avoiding collisions with stationary obstacles ...
1
vote
1answer
49 views

Would it be possible to enforce the same $s_{t + 1}$ between the model's estimate and the target function's Q-value?

Say I have a game of blackjack, and I am trying to teach a single forward-pass neural network to approximate the Q value of the current state and action. There are 3 inputs: The current card in hand, ...
0
votes
0answers
19 views

Why does the Bandit Slippery Walk environment have complimentary probabilities?

I am learning about Reinforcement learning in the book Grokking Deep Reinforcement Learning. Below are snippets. Below is the description of Bandit Slippery Walk (BSW) Below is the description of two ...
0
votes
0answers
16 views

Model for predicting whether an event will or will not happen

I am not very learned in the realm of ai and coding, but want to try to learn! There's a specific type of model I'm looking for but don't know how to find. I want to see if ai can predict the chances ...
1
vote
0answers
32 views

How to teach Machine Learning Agent to destroy replicating objects in a puzzle game?

I have an unusual but very interesting problem. I have a game that is very similar to Toon Blast (a puzzle mobile game). It's based on a Match-2 mechanic in which you can destroy 2 or more connected ...
0
votes
0answers
16 views

Algorithms for solving Contextual Bandits Problem with multiples continuous actions

I am currently working on a problem that has 7 continuous actions and instantly gives a reward. I was thinking that there are Contextual-Bandits-Algorithms applicable to this kind of problem, but so ...
0
votes
1answer
27 views

What is the meaning of the shaded area in the reinforcement learning literature graphs?

In most of the reinforcement learning literature, I see that there is a shaded area in the graphs. I couldn't understand what it exactly represents? For example, from the A3C paper: Or another ...
0
votes
1answer
56 views

How is $Q(s', a')$ calculated in SARSA and Q-Learning?

I have a question about how to update the Q-function in Q-learning and SARSA. Here (What are the differences between SARSA and Q-learning?) the following updating formulas are given: Q-Learning $$Q(s,...
1
vote
1answer
37 views

Are the two policies in SARSA for choosing an action the same?

Here is the pseudocode for SARSA (which I took from here) Are the two policies in SARSA for choosing an action equal? I guess yes, because it is called an on-policy learning algorithm. But could I, ...
0
votes
1answer
15 views

Are we choosing the same action in every step in SARSA?

Here is the pseudocode for SARSA (which I took from here) Do we only select one action at the very beginning and then we always choose the same action for each step? Does it really make sense to ...
0
votes
1answer
20 views

Why are we choosing more than 1 action in SARSA?

Here is the pseudocode for SARSA (which I took from here) Why are we choosing more than 1 action in SARSA? One for going into the next state and the other one for updating the Q function?
1
vote
0answers
28 views

Reinforcement Learning applied to Optimisation Problem

Problem Statement: We are given an optimisation problem; with production centres, source airport, destination airports, transfer points and finally delivered to the customers. This is better explained ...
1
vote
1answer
73 views

What is meant by "two action selections" in SARSA?

I have some difficulties understanding the difference between Q-learning and SARSA. Here (What are the differences between SARSA and Q-learning?) the following updating formulas are given: Q-Learning $...
2
votes
1answer
68 views

What components of reinforcement learning influence the result the most?

I'm working on my thesis concerning a reinforcement learning problem and am trying to prioritise my time on different components of it: Formalising the agent environment (like the design of state-, ...

1
2 3 4 5
38