Skip to main content

Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

Filter by
Sorted by
Tagged with
-1 votes
1 answer
31 views

Is the Invalid Action Masking in RL related only to the state or can be dependent on the reward as well?

I'm reading about Invalid Action Masking in RL in order to use it in my PPO algorithm for a specific task. The problem is that I read such explanations: here, here and here the there the invalid ...
Dave's user avatar
  • 192
1 vote
1 answer
21 views

Doubt regarding Actor-Critic method

As stated in Sutton and Barto: In REINFORCE with baseline, the learned state-value function estimates the value of the first state of each state transition. This estimate sets a baseline for the ...
DeadAsDuck's user avatar
1 vote
1 answer
24 views

What are the most common methods for handling non-stationary environments in reinforcement learning?

What are the most common algorithms, methods for handling non-stationary environments in reinforcement learning?
Mika's user avatar
  • 361
0 votes
0 answers
24 views

Two-agent sequential RL

I have the following RL model that I want to train (see the diagram below). My idea is to have two agents: agent A and agent B. Agent A observes the input I1 and ...
zdm's user avatar
  • 309
1 vote
2 answers
64 views

Doubt regarding policy gradient theorem and REINFORCE algorithm

After reading Sutton and Barto, I was able to understand the derivation of this theorem. The only thing I don't get is the following part from REINFORCE algorithm: How are these terms equivalent and ...
DeadAsDuck's user avatar
0 votes
0 answers
20 views

What is the best strategy to train a model with multi (sub)goals in the same environment?

To be able to explain my question I thought it is probably better to consider the following example: Let's take an environment, where a bridge crane need to lift a barrel from the position "start&...
Dave's user avatar
  • 192
1 vote
1 answer
49 views

Why is this RL derivation right?

This comes from the paper, Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review. I don't know the why the following derivation is true. The paper only briefly explains ...
yeebo xie's user avatar
0 votes
1 answer
34 views

How to build an AI, which is supposed to show emotions to an human User? [closed]

I plan to develop a computer pet that is as natural as possible. I think that neural networks can be used well for this, because they make it possible to constantly adapt to the user and also don't ...
b bud's user avatar
  • 1
1 vote
1 answer
87 views

How to use reinforcement Learning to solve DP with continuous action?

So I am trying to solve a control problem where the agent solves V(s) = max_a r(a,s) + EV(s'|s,a) Suppose the model is known, say the functional form of r is known and transition probabitliy is known. ...
Hank Chen's user avatar
0 votes
0 answers
26 views

Can a trained RL network outperforms the best training sample?

I'm working on solving a problem where I need to determine the optimal set of actions to find the path that yields the maximum reward. I'm currently using a Deep Q-Network (DQN) for this task. However,...
Amanli's user avatar
  • 1
-1 votes
0 answers
21 views

RL PPO Algorithem Doesn't Learn with Fixed Sigma?

I have a problem with my PPO Implementation in Python with Jax. My Agent for example optimizes the ant-v4 environment but only if the sigmas out of my Neural Net of the Policy are be between [1e-6,1]. ...
PC Benutzer's user avatar
0 votes
1 answer
38 views

Why parameter-based RL methods are not widely used?

Parameter-based RL methods perturb parameters of a general function approximator directly, rather than adding noise to the resulting actions. Why they haven't received the same attention and gain ...
Mika's user avatar
  • 361
1 vote
0 answers
17 views

The observation space of a robot arm should include the target position or only the joint values?

I'm interested in developing a RL application for a robotic arm in a virtual environment. But I stuck at the question, whether: The observation space should contain the target position + the actual ...
Dave's user avatar
  • 192
1 vote
1 answer
37 views

Actor Critic need to find the goal to have good update and suceed?

Im trying to solve MoutainCar and CarRacing (i love cars) in gym environnement with DDPG and my algo struggle. I have think on why this don't work and I would like to know if my resoning is false. ...
Cauchy_Chlasse's user avatar
0 votes
1 answer
53 views

Reinforcement learning for Bridge - NN model fails to learn legal actions

Summary I have built a neural network model for reinforcement learning that is supposed to learn to play the card game of Bridge. So it must predict Q-values of player's actions but also distinguish ...
Radek's user avatar
  • 11
1 vote
0 answers
52 views

Handling large or multiple agent actions that cause non-stationarity

How to handle a problem where the agents' actions are changing the environment? For instance, a trading agent can hypothetically make a huge transaction that is enough to push the stock prices to go ...
Mika's user avatar
  • 361
1 vote
2 answers
64 views

Learning rate and rewards in Deep Reinforcement Learning

In environments with sparse and (sometimes) binary rewards such as Taxi Cab, Mountain Car and Frozen Lake, where agents rarely encounter positive outcomes, I've observed that the gradient descent's ...
HenDoNR's user avatar
  • 106
0 votes
3 answers
77 views

Do we use RL to train ANNs, or use ANNs as par of a RL solution?

I am learning about deep learning. Currently just at a superficial level, I think I am misunderstanding how reinforcement learning and artificial neural networks are used together. For what I first ...
Esteban's user avatar
  • 23
0 votes
1 answer
31 views

Can all RL algorithms learn with discrete state spaces?

This question come to mind when i was planing to do a benchmark of RL algorithms to my Environment. In fact, Q-Learning, SARSA actually only handles with discrete state spaces because they are tabular ...
Vitor Martins's user avatar
0 votes
1 answer
36 views

Please help me understand constant-α Monte Carlo method

This is my understanding thus far about Monte Carlo method for approximating value function: Instead of using a recursive Bellman equations and knowledge of environment dynamics, Monte Carlo methods ...
DeadAsDuck's user avatar
2 votes
1 answer
72 views

Why do big policy updates cause performance drop in deep RL?

In the TRPO and PPO papers, it is mentioned that large policy updates often lead to performance drops in policy gradient methods. By "large policy updates," they mean a significant KL ...
Druudik's user avatar
  • 171
0 votes
1 answer
38 views

To which value does converge the objective: $target = (r + \gamma Q(s',*)) / 2$

Conditions: Assume you have a DRL SAC implementation. You train it as usual from Replay Buffer with uniform sampling with replacement. Assume you change the target of the Q-network from: $$T = r + \...
Jose Antonio Martin H's user avatar
0 votes
1 answer
53 views

Is deep learning suitable/preferable for string similarity detection and application automation? If so, which type?

newbie here. I have developed an app that basically does: Perform OCR, check if words are contained in the resulting text and then perform an action. If no words are detected from the given list, ...
zaxunobi's user avatar
  • 101
0 votes
1 answer
76 views

Why are there two different q-learning formulas?

I found the following q-learning formula: in this youtube video: https://www.youtube.com/watch?v=4C133ilFm3Q&t=521s I'm now a bit confused, since I thought, that the following one is the correct ...
Hans123's user avatar
  • 25
1 vote
1 answer
47 views

What is the right DRL algorithm to use when the goal in an environment is not fixed?

Let's take the LunarLander environment from the package Gym as an example. In this case, one can run thousand of episodes until the agent learns a good policy. However, there is a condition: the goal ...
Dave's user avatar
  • 192
1 vote
1 answer
39 views

DDPG model outputting a fixed action at every timestep

I am trying to create a Car Following model, for which i am using DDPG. My action is acceleration bounded in a range of [-3,3] m/s2. While training the model, for every state it gives a single ...
Aditya Mishra's user avatar
0 votes
0 answers
12 views

A concept for a simple NN as a transfer function for an hydraulic cylinder

For a RL project, my collagues and me need to create a virtual environment with an excavator, which needs to replicate a real existing excavator. The idea is to have a simulator, which simulate ...
Dave's user avatar
  • 192
0 votes
1 answer
33 views

Would the DDPG algorithm still function effectively if some transitions stored in its replay buffer are generated by a completely unrelated policy?

Let's hypothesize a scenario where some of the records (si, ai, ri, si+1) in the replay buffer are generated by another completely unrelated random policy. If the DDPG algorithm still samples random ...
JJbow's user avatar
  • 1
0 votes
1 answer
74 views

What are disadvantages/limitations of Monte Carlo Tree Search in RL?

What are disadvantages/limitations of Monte Carlo Tree Search in RL, and hence for what kind of applications might its use not be appropriate?
DSPinfinity's user avatar
0 votes
1 answer
95 views

Is reinforcement learning suitable for application automation?

I have basically automatised the use of an app through the use of OCR and computer vision. So basically when a word or an image is detected it will perform a certain action. When that action is ...
zaxunobi's user avatar
  • 101
0 votes
1 answer
89 views

Why is policy gradient theorem so important?

What is the problem that the policy gradient solves? From what I understand the problem is taking the gradient of the state distirbution $d^{\pi_{\theta}}$, but what is exactly the problem here (maybe ...
craaaft's user avatar
  • 111
1 vote
0 answers
27 views

Proper way to load a RL (reinforcement learning) model (pytorch) for "testing"...?

I'm working on a RL problem where, in a nutshell, an agent has to go from point A to point B, in that order, with as few steps as possible, using DQN with PyTorch, to train the agent. During training, ...
Jose Alberto Salazar's user avatar
1 vote
1 answer
58 views

Confusing convention in Sutto-Barto on Monte Carlo Tree Search: is a leaf node a state leaf node or state-action leaf node?

Figure 8.10: Monte Carlo Tree Search. When the environment changes to a new state, MCTS executes as many iterations as possible before an action needs to be selected, incrementally building a tree ...
DSPinfinity's user avatar
0 votes
1 answer
69 views

Proof of gradient of $v_{\pi}(s)$ via Kronecker Product

Hi I am reading Mathematical Foundation of Reinforcement Learning by Shiyu Zhao and I try to understand a proof regarding policy gradients. The part is on page 209/210 in Policy Gradient Methods. ...
craaaft's user avatar
  • 111
2 votes
1 answer
45 views

Should the experience replay memory only contain unique experiences?

I'm training an RL agent/model (DRL/DQN). Say that, for each learning step, the memory replay used by the agent to learn, has N elements (experiences) stored, where only X are unique elements (...
Jose Alberto Salazar's user avatar
0 votes
1 answer
42 views

Are there problems where the optimal policy is stochastic?

I know that in a MDP there always exists a unique optimal deterministic policy. Does a statement like this also exist for optimal stochastic policies? Is there also always a unique optimal stochastic ...
craaaft's user avatar
  • 111
-2 votes
1 answer
35 views

Unclear line in prioritized sweeping algorithm [closed]

Could someone explain the red line (especially, the meaning of the difference) in prioritized sweeping algorithm below? Sutton-Barto, page 170:
DSPinfinity's user avatar
0 votes
0 answers
14 views

Soft Actor Critic policy update depending on q function in discrete action space

Reference: https://spinningup.openai.com/en/latest/algorithms/sac.html In the psuedo-code of the algorithm, line 14, the actor update is written as to maximize the q-function. Theoretically, this ...
moe asal's user avatar
  • 133
1 vote
2 answers
88 views

tic-tac-toe - tabular q-learning - what is the formula to calculate the number of entries in the q-table

i implemented the tabular q-learning algorithm for 3x3 tictactoe multiple times and everytime the number of entries in the q-table is 16,167. I wanna know how to calculate the number of 16,167. what ...
Hans123's user avatar
  • 25
0 votes
1 answer
23 views

Proof that Temporal-Difference TD(1) is Equivalent to Widrow-Hoff

I'm reading Sutton's "Learning to Predict by the Methods of Temporal Differences" and I'm getting hung up on a derivation (p. 14). We are considering (observation-sequence, outcome) pairs. $[...
Matheo Xenakis's user avatar
-1 votes
1 answer
78 views

Violation of Markov property

Consider the following cases: a-) In solving an episodic problem we observe that all trajectories from the start state to the goal state pass through a particular state exactly twice. b-) In solving ...
DSPinfinity's user avatar
0 votes
1 answer
49 views

Confusing statement in Sutton-Barto on trajectory sampling

Suuton-Barto, page 176: experiment to assess the effect empirically. To isolate the e↵ect of the update distribution, we used entirely one-step expected tabular updates, as defined by (8.1). In the ...
DSPinfinity's user avatar
3 votes
1 answer
200 views

Can we implement a memory in a REINFORCE algorithm for RL?

In Q-learning with function approximators such as Neural Networks, we typically implement a memory so that at the end of each episode we also train on past experiences. This is typically fine because ...
FluidMechanics Potential Flows's user avatar
0 votes
0 answers
31 views

Confusing statement in Sutton-Barto on expected versus sample updates

Sutton-Barto, page 174. b successor states are equally likely and in which the error in the initial estimate is 1. The values at the next states are assumed correct, so the expected update reduces ...
DSPinfinity's user avatar
0 votes
0 answers
16 views

Reinforcement Learning Gymnasium ValueError

I am testing out reinforcement learning for the first time with gymnasium. I am following a youtube tutorial. I am getting the following error when I run the training loop: ValueError: setting an ...
Sheila's user avatar
  • 1
0 votes
0 answers
25 views

Q-Learning conditions for convergence and ergodicity

Q-learning is guaranteed to convergence if the learning rate satisfies the Robbins-Monro conditions and if every state-action pair is visited infinitely often. Regarding the latter, does it mean that ...
Simon's user avatar
  • 153
0 votes
0 answers
39 views

Optimizing a nonlinear objective function in Deep Reinforcement Learning

I'm working on a reinforcement learning problem where the environment returns a reward pair $(r_{t+1}^{(a)}, r_{t+1}^{(b)})$. The goal is to maximize the following nonlinear objective function. $$ E[\...
Alex's user avatar
  • 1
0 votes
2 answers
71 views

Why is dynamic programming an example of planning?

Sutton-Barto, page 160, towards bottom: Why is dynamic programming an example of planning? There is no simulation in dynamic programming.
DSPinfinity's user avatar
0 votes
0 answers
12 views

Is this a typo in n-step tree backup section in Sutton-Barto?

Sutton-Barto, page 153. Should not it be $t<T-n$ in Eq.16? The reason is we have $t<T-1$ and $t<T-2$ for the 1 and 2 step returns, respectively.
DSPinfinity's user avatar
1 vote
1 answer
24 views

Why is importance sampling ratio in n-step TD multiplying error rather than return n-step return?

Why is importance sampling ratio in n-step TD is multiplying error rather than return? In Monte Carlo methods for state values, importance sampling ratio was simply a multiplier for the return.
DSPinfinity's user avatar

1
2 3 4 5
51