Skip to main content

Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

Filter by
Sorted by
Tagged with
0 votes
0 answers
5 views

Linear Quadratic Regulator's gradient

This comes from cs285 lesson10, model-based reinforcement learning. https://rail.eecs.berkeley.edu/deeprlcourse/ $$ Q_{x_{T}, \mu_{T}} = const + \frac{1}{2} \begin{bmatrix} x_{T}\\ \mu_{T} \end{...
yeebo xie's user avatar
0 votes
0 answers
18 views

Parametric noise over Input noise

I came across this research from 2017 that talked about using "Parametric noise" instead of input based noise. I have tried to have it in my PPO based Boid flocking custom environment but ...
Hamza's user avatar
  • 1
0 votes
0 answers
17 views

Extending learnt policy for greater than "n" agents [closed]

I have a custom Boid flocking environment that uses PPO for the RL algorithm. I have trained and tested it using 3 Boids. However I want to have n Boids do the training and then have > n Boids for ...
Hamza's user avatar
  • 1
0 votes
1 answer
15 views

Optimality of two policies versus variance of returns from a state

If for an MDP there exist two optimal policies, it may be possible that the variance of their returns are different for a given state. This is correct, right?
DSPinfinity's user avatar
0 votes
0 answers
12 views

Properties of an example environment

Let us consider the following problem. A student is developing a robot that can roll a die. The robot grips and releases the die with an arm that can also twist, and uses two cameras that can see the ...
DSPinfinity's user avatar
1 vote
0 answers
13 views

Why some papers focus on constructing large dataset from real robots, instead of simulations?

Recently, I have seen papers about large datasets for robotics such as DROID(https://droid-dataset.github.io/) or Open X-Embodiment(https://robotics-transformer-x.github.io/). As I see, the datasets ...
user3315463's user avatar
0 votes
0 answers
20 views

Why policy gradient can be rewritten to the expection of Q values?

This comes from SAC algorithm. Actor with REINFORCE $$ \nabla_{\theta}J = \mathbb{E}_{s\sim \mathcal{D}, a\sim \pi( a|s)}[\nabla_{\theta}\log(\pi_{\theta} (a|s) Q_{\phi}(s,a))] $$ Actor with ...
yeebo xie's user avatar
0 votes
0 answers
16 views

Difficulty training PPO agent for robotic arm navigation task

 I'm currently working on training a PPO agent for a robotic arm navigation task, where the goal is to navigate the robotic arm to different positions in the environment. I've been training the agent ...
Weitao Kang's user avatar
1 vote
1 answer
36 views

How does the Belman optimality equation with altered transition probabilities in the second equation follow?

Sutton-Barto, page 102 (second edition). How does the Belman optimality with altered transition probabilities in the second equation follow? The point which confuses me is the first part inside the ...
DSPinfinity's user avatar
2 votes
0 answers
22 views

In a real-world scenario where no simulation program can be used, is a good idea to train a policy in offline with a pre-collected dataset?

I'm currently designing an RL architecture for a real-world application. My application involves a human-robot collaboration task. Sadly, i can't use a simulator to simulate the human behavior, ...
Vitor Martins's user avatar
1 vote
1 answer
18 views

Why soft Bellman backup operator is converged in tabular settings?

This comes from Soft Actor-Critic:Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. Consider the soft Bellman backup operator$\tau^{\pi}$ in equation and a mapping $Q^{0}:...
yeebo xie's user avatar
1 vote
0 answers
12 views

How to measure accuracy of learned value function of a fixed policy?

Let's say we've a given policy whose value function is to be evaluated. One way to get the value function can be using expected SARSA, as in this stack exchange answer. However, my MDP's state space ...
ModCon's user avatar
  • 111
0 votes
0 answers
19 views

In Markov Decision Process, how to understand the calculation of the average length of episode?

In the Sec. 13.2 of RL: An Introduction (Sutton & Barto), the concept of average episode length is discussed for both episodic MDP and continuing MDP. In an episodic MDP, the average length of an ...
Yancy Pan's user avatar
1 vote
0 answers
19 views

Two player grid-based game, minimax not good, what to try?

In this game, a matrix with size N x M is given (10<=N,M<=15). There a three main type of value in the matrix: A integer show how many gold is it Char 'D' show obstacle Char 'S' show shield (...
Parrottos 's user avatar
2 votes
1 answer
226 views

Where does the TD formula for tic-tac-toe in Sutton & Barto come from?

In section $1.5$ of the book "Reinforcement Learning: An Introduction" by Sutton and Barto they use tic-tac-toe as an example of an RL use case. They provide the following temporal ...
mNugget's user avatar
  • 167
0 votes
2 answers
38 views

Can DQN lead with discrete state spaces?

For example in Cart Pole v1 gym environment the state space is continuous, but we discretize it to apply the Q-Learning algorithm because Q-Learning is a tabular method and only works with discrete ...
Vitor Martins's user avatar
2 votes
2 answers
787 views

Is there an algorithm that produces a uniform distribution over the set of trajectories with maximum reward sum?

I am not an expert in reinforcement learning. I am applying it to my field of study. I am training a model such that given a state, it predicts the probability of taking an action for every action ...
moe asal's user avatar
  • 131
0 votes
2 answers
26 views

In reinforcement learning policy evaluation, which reward is received?

Assume the agent is in a state $s$ and takes action $a$, intending to move to $s_1$, however the dynamics may drop the agent in $s_1$ or $s_2$: may move to $s_1$ and receive reward $r_1$ with a ...
Nat's user avatar
  • 13
1 vote
1 answer
69 views

Is it possible that an RLNN generates actions by itself based on the info and observation state provided by the environment?

Is it possible that an RLNN generates actions by itself based on the info and observation state provided by the environment? For example a function G(s) where it takes in the state as input and ...
19216811's user avatar
1 vote
1 answer
116 views

What is the optimal policy for this MDP?

What is the optimal policy for this MDP?
DSPinfinity's user avatar
1 vote
1 answer
20 views

Understanding the policy improvement theorem for Monte Carlo Control without Exploring Starts

I am currently studying the equations 5.2 in Reinforcement Learning An Introduction By Sutton and Barto on page 101. I want to comprehent the proof by a simple example: Having only one State with two ...
bake_thi's user avatar
0 votes
0 answers
25 views

Need some feedback on an idea for using reinforcement learning in the context of medical imaging reconstruction

Disclaimer -- this idea may be totally half-baked, I'm not sure. I have used deep learning models in image reconstruction before (and this is a super hot topic in the field right now), but only in the ...
t_h's user avatar
  • 1
0 votes
1 answer
35 views

Policy in on-policy algorithms and experience replay

For SARSA algorithm, assuming that we initialize all $Q(s,a)$ to $0$, then in the first iteration, all actions are the best actions as $Q$ values are the same ($0$). So the behavior policy in this ...
k2pctdn's user avatar
  • 55
2 votes
1 answer
51 views

Proof of existence of optimal policy

I have been trying to prove existence of an optimal policy for RL. I have proved that the Bellman optimality operator, $B: \mathbb{R}^{|\mathcal{S}|} \to \mathbb{R}^{|\mathcal{S}|}$ given by $$B(v_\pi)...
mNugget's user avatar
  • 167
0 votes
1 answer
79 views

Can Q(s,a) be replaced by V(s) when certrain requirements are met?

I read this post, was thinking about it and now I have a hypothesis but I am not sure whether or not its correct. I claim that in Q-learning $Q(s,a)$ can be replaced by V(s) when $p(s'|a,s)$ is ...
NMO's user avatar
  • 161
0 votes
0 answers
22 views

Policy gradient - future looking returns

In the policy gradient approach, one differentiates the expected reward $$ \mathbb{E}J=\sum P(\tau;\theta) R(\tau) $$ to obtain $$ \Sigma R(\tau) \nabla \log P(\tau;\theta) $$ (with some abuse of ...
Eli's user avatar
  • 1
2 votes
1 answer
214 views

Proof of bellman optimality equations

I am studying RL and have a hard time proving the existence of an optimal policy. I found some resources online, and I am trying to prove the following theorem: If there exists a policy $\pi$, state $...
mNugget's user avatar
  • 167
2 votes
2 answers
353 views

Is the Bellman backup unbiased?

This is comes from cs285 2023Fall hw3. In my opinion, if $\hat{Q}$ is unbiased estimate of $Q$, then $$ \begin{align} \mathbb{E}_{D \sim P}[B_{D}\hat{Q} - B_{D}Q] &= \mathbb{E}_{D \sim P}[r(s,a) +...
yeebo xie's user avatar
1 vote
2 answers
41 views

Why is the better policy defined with respect to all the states values being greater?

In Sutton & Barto (Section 3.6 - Optimal Policies and Optimal Value Functions), they say that : Value functions define a partial ordering over policies. A policy $\pi$ is defined to be better ...
pew31's user avatar
  • 13
1 vote
1 answer
49 views

Why soft actor critic uses exponential of Q when updating policy? and what is a partition function?

Soft Actor-Critic paper proposes $\pi_{new}=\arg\min_{\pi'\in \prod}D_{KL}\left(\pi'(\cdot|s_t)\big|\big| \frac{\exp(Q^{\pi_{old}}(s_t,\cdot))}{Z^{\pi_{old}}(s_t)}\right)$ Paper says, we update the ...
user3315463's user avatar
2 votes
1 answer
31 views

Why is there the potential problem of "learning only from the tails of episodes" in off-policy MC control?

Sutton-Barto page 111, first paragraph (Off-policy Monte Carlo Control): A potential problem is that this method learns only from the tails of episodes, when all of the remaining actions in the ...
DSPinfinity's user avatar
0 votes
2 answers
64 views

Textbooks (or other sources) on deep reinforcement learning which explain theory along with good examples

I am looking for a textbook/other sources on deep reinforcement learning which explain theory along with good examples. I will be happy for suggestions.
DSPinfinity's user avatar
1 vote
0 answers
56 views

Computing variance of gradient of infinite horizon MDP

This comes from CS285 2023Fall homework2. I'm self-learning RL, but analysis question seems a little hard for me. In question one, we could get $\mathbb{E}_{\tau \sim \pi_{\theta}}R(\tau)=\theta(1-\...
yeebo xie's user avatar
0 votes
1 answer
39 views

In Sutton-Barto a confusing point regarding $\epsilon$-soft policies in the proof for optimality of MC control without exploring starts

Sutton-Barto, page 102: In the second paragraph, we have: Consider a new environment that is just like the original environment, except with the requirement that policies be $\epsilon$-soft “moved ...
DSPinfinity's user avatar
0 votes
1 answer
28 views

RL simulation: Does the Gym-like RL training solution fits the real-world environments?

I am new to the RL community and I am working on projects about using RL to control robots like drones to fly in the scenes. I have been using Nvidia's Issac Gym for some time, but I have a question ...
bz2000's user avatar
  • 1
3 votes
2 answers
70 views

Confusing point in Sutton-Barto: replacing $a$ in $q(s,a)$ with a stochastic policy $\pi^\prime$

Sutton-Barto, page 101, Eq (5.2): Assume that $\pi^\prime$ is the $\epsilon$-greedy policy. Then, \begin{align} q_{\pi}\big(s,\pi'(s)\big)&= \sum_{a}\pi'(a|s)q_{\pi}(s,a) \\ ...
DSPinfinity's user avatar
1 vote
1 answer
166 views

Policiy Gradient Infinite-Horizon Analysis

Source: CS285 2023Fall homework2 analysis. In the part B, the expectation $\mathbb{E}[R(\tau)] = \theta + \theta^2 ... = \theta/(1-\theta)$. Derivate this with respect to $\theta$ is $1/(1-\theta)^2$....
yeebo xie's user avatar
0 votes
0 answers
18 views

A question about the goal selection in the second loop in the HER algorithm

I have to admit, that I still have problem digesting the HER algorithm proposed in the famous paper and despite the fact, that the idea behind it should be intuitive, I realize that it is not so easy ...
Dave's user avatar
  • 155
0 votes
1 answer
31 views

Can we use MAB in problems that reset after some time?

I have this scheduling problem. There are $n$ jobs, one machine and $T$ time slots. To be satisfied, each job $i=1,\ldots,n$ must receive at least the quantity $v_i$ from the machine. The machine can ...
zdm's user avatar
  • 301
2 votes
1 answer
30 views

Can a policy with gaussian distribution allow two distinct optimal actions to have distinctively high probabilities?

As an example to show the benefits of stochastic policy, I often have seen the below grid world example. Five blocks in a row. the first, third, and fifth are white(distinguishable states), and the ...
user3315463's user avatar
1 vote
1 answer
25 views

What is the difference between two approaches in solving "Infinite number of episodes" in Monte Carlo Control with exploring starts?

Here, https://lrscy.github.io/2020/07/09/Coursera-Reinforcement-Learning-Course2-Week2-Notes (See the "Monte Carlo Control" and then "Solutions of Two Assumptions" sections) two ...
DSPinfinity's user avatar
3 votes
1 answer
170 views

Does the DoubleDQN algorithm use a target network or two separate policies?

I've been looking for ways to improve my DQN. That is when I found the Double DQN algorithm. After looking at explanatory videos and posts, I've seen conflicting information: The Double DQN algorithm ...
Vladislav Korecký's user avatar
0 votes
0 answers
23 views

Critic Loss with MADDPG for Fixed-Point Reach Task in MuJoCo

I've been working on implementing a path control algorithm for a robotic arm in a MuJoCo environment, aiming to reach a fixed point. To eliminate the potential instability issues of MuJoCo, I've ...
Weitao Kang's user avatar
2 votes
1 answer
144 views

Q learning (DQN) strategy for a multiplayer zero-sum game

I have been looking for ways to train a Q-learning agent for a multiplayer zero-sum game (a variation of Tic-Tac-Toe in my case). I came up with a learning strategy I haven't found anywhere else, and ...
Vladislav Korecký's user avatar
0 votes
1 answer
25 views

Can/should a reward function depend on something other than state in a DQN

Question: Is it OK to have a reward function on a DQN or any RL algorithm that depends on variables other than the enviroment state? I'm asking because, so far I'm learning from tutorials, but I've ...
Oliver Mohr Bonometti's user avatar
0 votes
1 answer
32 views

Use your own simulation to train a reinforcement learning multi-agent

I am wanting to train an RL multi-agent model to run in a propietary simulation, which is written in C++. Is there a way to change the simulation itself to create an agent, or must I use a ...
michael-c-michael's user avatar
1 vote
1 answer
21 views

Why are there up to $m^2$ action values when we consider the complexity of DP based on $q(x,u)$?

Please see slide 32 in the following lecture slides on DP: https://groups.uni-paderborn.de/lea/share/lehre/reinforcementlearning/lecture_slides/built/Lecture03.pdf Let $m$ the size size of action ...
DSPinfinity's user avatar
0 votes
0 answers
26 views

why PPO performs better than TRPO?

what is the logistic behind why PPO performs better than TRPO(at least in the PPO's paper results)? When I read both papers, it seems like PPO is just simpler version of TRPO, that it does not run ...
user3315463's user avatar
1 vote
2 answers
34 views

Why do exhaustive search require 14 travel segment evaluations but dynamic programming require 10 for this shortest path problem?

Why do exhaustive search require 14 travel segment evaluations but dynamic programming require 10 for this shortest path problem? I need a clear explanation.
DSPinfinity's user avatar
0 votes
1 answer
20 views

MDP and a given policy and the correctness of the state-value function

Is the following statement correct? "For an MDP and a given policy, the Bellman equation can be used to check the correctness of the state-value function."
DSPinfinity's user avatar

1
2 3 4 5
50