Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

Filter by
Sorted by
Tagged with
0 votes
0 answers
2 views

Why not use only expert demonstrations in Imitation Learning approaches?

Some IL approaches train the agents by using some specific ratio of expert demonstrations to trajectories generated using the policy being optimized. In the specific paper I'm reading they say "...
user avatar
1 vote
0 answers
6 views

Reward design or Inverse reinforcement learning?

I'm working on a reinforcement learning project where I only have demonstrations (i.e. set of states and actions). During my research on how handle the reward signal, I noticed that research papers ...
user avatar
0 votes
0 answers
13 views

Why does providing an extra prediction output help stabilize training?

I am reading the PRIMAL: Pathfinding via Reinforcement and Imitation Multi-Agent Learning paper where they tackle the multi-agent path finding problem using reinforcement learning. The problem is ...
user avatar
1 vote
1 answer
7 views

What does it mean for an episode to start in a state-action pair?

In Sutton and Barto on chapter 5 (p.96), they talk about estimating state-action values with Monte Carlo: For policy evaluation to work for action values, we must assure continual exploration. One ...
user avatar
  • 169
3 votes
1 answer
47 views

Why does REINFORCE perform badly at first in Sutton and Barto Figure 13.1?

In Sutton and Barto (PDF, page 265), 2nd edition, Figure 13.1 applies REINFORCE to the "short corridor with switched actions" environment from Example 13.1. The figure looks like this: My ...
user avatar
0 votes
0 answers
11 views

Control variables and cofounding effects in stochastic programming/,model predictive control/reinforcement learning

How can we be sure that confounding variables/control variables don’t pickup the effect our decisions w.r.t decision variables had on the actual control variable? Since the term control variable ...
user avatar
0 votes
0 answers
17 views

In the Policy Gradient Theorem proof, why is $d^\pi(s) = \sum_{k=0}^{\infty}\gamma^{k}Pr(s_0 \rightarrow s, k, \pi)$ true?

I was reading the original Policy Gradient Paper. I didn't quiet get the last step of the proof for the policy gradient theorem. The proof given in the paper is below: I don't understand how the last ...
user avatar
  • 9
-1 votes
0 answers
29 views

Why do we use mean squared error loss with deep q networks?

For computing the TD error (pseudo) loss in DQNs, we have the following formula - $$L(\theta) = 0.5*(R_{t+1} + \gamma[max_aq_{\theta}(S_{t+1}, a)] - q_{\theta}(S_t, A_t))^2$$ However, in practice, we ...
user avatar
0 votes
0 answers
18 views

"Early stopping" in policy iteration?

In exercise 4.4 in Sutton and Barto Reinforcement Learning, they ask: The policy iteration algorithm on page 80 has a subtle bug in that it may never terminate if the policy continually switches ...
user avatar
  • 169
1 vote
0 answers
24 views

Question for the derivation of the probability of a trajectory

I'm studying reinforcement learning now and I'm quite a newbie to this field. I have some questions about how to derive the equation as below. $p_{\theta}(s_{1},a_{1},\dots,s_T,a_T)=p(s_1)\prod_{t=1}^...
user avatar
  • 11
0 votes
1 answer
9 views

What's the relationship between Bayesian RL and POMDPs?

Bayesian RL seems concerned with having uncertainty over the transition function of the environment, while POMDPs try to capture uncertainty over the state one is currently in. However, both end up ...
user avatar
  • 136
0 votes
1 answer
21 views

Why does a PPO agent perform only the action that costs the least?

I am trying to implement an intelligent agent that can perform penetration testing within the nasim (link) environment, a network simulator. I would like to try to use parametric mode for actions, and ...
user avatar
0 votes
0 answers
9 views

How to utilize real-world data to improve performance of RL Agent?

I'm still a newbie in RL, so please forgive me if the answer is obvious or the question totally dumb. I'm currently trying to train a robot RL agent in OpenAI gym with a custom environment. This alone ...
user avatar
1 vote
0 answers
9 views

Comparing Reinforcement Learning models

I am currently doing my thesis on optimising combinatorial problems. We decided to uitilize RL. The problem is that: I am not sure which model to go with. Is there a comprehensive table or guide to ...
user avatar
0 votes
0 answers
10 views

How state is combined with action in crtitic networks?

Actor-critic networks are present in deep reinforcement learning algorithms. Actor-network takes a state as input and gives action as output. Critic-network takes state and action as input and gives a ...
user avatar
  • 3,191
0 votes
0 answers
5 views

How to partition the belief space of a POMDP using a "granularity" parameters?

as I understand, to a solve a pomdp we transform it into a belief-MDP. The value function for this belief-MDP is proven to be piecewise linear and convex (PWLC) [Smallwood and Sondik, 1973].To apply ...
user avatar
0 votes
0 answers
11 views

UCB algorithm exercise

I am trying to understand the UCB algorithm and I'm trying to understand it using an exercise. Here's the Upper Confidence Bound algorithm explanation: Now I have the following exercise: Suppose we ...
user avatar
1 vote
1 answer
35 views

Can RL still learn if part of my actions are only used once, at the beginning of the episode?

I am working in an environment with 3-dimensional action space. The first two actions are only used at the first timestep and never again. The third action is used at every timestep. Say, the action ...
user avatar
1 vote
0 answers
14 views

How can I get an integer as output for continuous action space PPO reinforcement learning?

I have a huge discrete action space, the learning stability is not good. I'd like to move to continuous action space but the only output for my task can be a positive integer (let's say in the range 0 ...
user avatar
  • 111
1 vote
1 answer
162 views

Why are AI Safety discussions almost always from the perspective of reinforcement learning?

I have been reading some articles on AI safety and they almost always speak of AI Safety from the reinforcement learning (RL) perspective, i.e. where we have some artificially intelligent agent acting ...
user avatar
0 votes
1 answer
77 views

What is the need for _agency_ in AI? [closed]

Why seek to develop artificially intelligent agents? Are there certain advantages and/or needs provided by such supposed intelligent agents that are preferred to simply using intelligent tools that ...
user avatar
1 vote
1 answer
24 views

Markov's Decision Process - calculate value in each iteration

I have the following decision tree: I calculated the value of the plan using the following paramenters (given): {𝑆0 β†’ π‘Ž1 , 𝑆1 β†’ π‘Ž3 , 𝑆2 β†’ π‘Ž4 }, Discount factor (𝛾)= 0.2 I used this formula to ...
user avatar
0 votes
0 answers
17 views

How to learn the value function in a two-player game?

In single-player games, the optimal policy can be derived from the state value function v(s): $$ \pi(s) = \underset{a}{\text{argmin}} \sum_{s'} p(s'|a,s)(c(a) + v(s')) $$ where c(a) is the cost of ...
user avatar
  • 113
0 votes
0 answers
16 views

Why would it be easy to evaluate a probability, when it is hard to sample from for importance sampling?

Suppose we want to perform importance sampling where we have trajectories from some behavioral policy $b$, but we want to perform off-policy evaluation. From these prior questions, I understand that ...
user avatar
0 votes
0 answers
11 views

When does it make sense to switch from discrete action space to continuous?

I'm currently working on a custom RL environment for a PPO model that I'd like to have 40-100 discrete actions with integer-level precision (no decimals). Looking through some papers on the topic, it'...
user avatar
  • 1
-1 votes
1 answer
34 views

How to build the actor policy of Soft-Actor-Critic after sampling from a Multivariate normal distribution?

I'm trying to solve LunarLanderContinuous-v2 (https://www.gymlibrary.ml/environments/box2d/lunar_lander/) using Soft Actor-Critic algorithm (following the pseudocode above) To update the actor policy ...
user avatar
0 votes
0 answers
19 views

Double DQN performs worse than DQN

I have an agent that has to explore a customized environment. The environment is a grid (100 squares horizontally, 100 squares vertically, each square is 10 meters wide). In the environment, there are ...
user avatar
  • 146
0 votes
1 answer
48 views

Which paper describes the effect of learning_starts in Reinforcement Learning?

I have seen many popular RL libraries have a learning_start parameter. This allows the agent to collect enough experiences before training on the replay_buffer. However, I am unable to find the paper ...
user avatar
0 votes
0 answers
18 views

Is it possible to use Reinforcement Learning to learn good weights for another algorithm?

I developed an algorithm that drives a car on a road inside a simulated environment. The algorithm needs weights (parameters) to be set in advance in order to find the best route. These weights are ...
user avatar
0 votes
0 answers
36 views

What does maximal |Q| mean in DQN?

I am reading this paper and came across the term maximal |Q|. I'd like to know whether it refers to the Q values of the current state $Q(s_t,a_t)$ or that of the target $\mathbb{max}_aQ(S_{t+1}, A_t)$....
user avatar
1 vote
2 answers
27 views

Are RL algorithms suppose to keep learning?

I don't understand if the purposes of RL agents is simply optimizing a model with a reward instead of using labeled data (i.e. in a supervision fashion), or they have also the purpose of keep training ...
user avatar
  • 41
0 votes
0 answers
15 views

Why DDPG losses don't decrease while the reward grows?

I've noticed that training a DDPG agent in the Reacher-v2 environment of OpenAI Gym, the losses of both actor and critic first decrease but after a while start increasing but the episode mean reward ...
user avatar
  • 41
1 vote
1 answer
27 views

How to prove that an action-value function optimal for one problem formulation is also optimal for another?

I want to ask about the intuition/where-to-look/what-to-try if I want to prove that an action value function optimal for a problem is also optimal for another reformulation of that smae problem. For ...
user avatar
1 vote
2 answers
38 views

RL solutions for OpenAI Gym environments?

Is there any place where people share their agent's settings for solving OpenAI Gym Environments? For example, I'd like to know what are good parameters for a DDPG agent to learn the task in Reacher-...
user avatar
  • 41
0 votes
1 answer
50 views

Has there been a study done in tuning hyper-parameters for off-policy reinforcement learning?

I am interested in learning about hyper-parameter tuning for off-policy reinforcement learning (specifically DQN). Could someone point me to papers published or empirical observations in this area?
user avatar
-1 votes
1 answer
26 views

Play against your own RL-trained AI from gym retro

so far I have seen people implementing reinforcement learning to build an AI to play and complete games on gym retro, such as street fighter, racing games and so on. However, I was wondering if it is ...
user avatar
0 votes
0 answers
30 views

Reinforcement learning SOTA with continuous action space

as of July 2022 what is the SOTA in reinforcement learning with continuous action space? DDPG PPO … other?
user avatar
  • 101
0 votes
0 answers
18 views

Is there benefit to autoregressive models for deep RL tasks with long episodes and short required context?

General Case In deep RL (specifically in the space of policy gradient methods) it seems very common that encoder-decoder models (either transformer or RNN-variant) are used in the policy/value ...
user avatar
  • 101
2 votes
1 answer
288 views

How to deal with small reward values

In my environment rewards are generally small, e.g. [-0.01, 0.01]. My concern is that small reward values might get dominated or distorted by the noise during the training. Does it make sense to scale ...
user avatar
  • 197
1 vote
1 answer
47 views

Time taken to solve cartpole environment using DQN

I am trying to solve the cartpole environment (GitHub) using DQN agent. I have been building my own DQN agent by following a tutorial by Jon Krohn. I am able to solve the environment with a maximum ...
user avatar
0 votes
0 answers
12 views

Why episode mean rewards drawdown?

I'm new to RL. I'm using RecurrentPPO with parameter MlpLstmPolicy and the other defaults. Why the ...
user avatar
  • 123
0 votes
0 answers
49 views

Tensorflow-gpu and multiprocessing

I have finished implementing an Asynchronous Advantage Actor-Critic (A3C) agent for TensorFlow (gpu). By using a single RMSprop optimizer with shared statistics. To do so, a central controller holds ...
user avatar
1 vote
1 answer
32 views

Mathematics books for reinforcement learning

This question is not about the math prerequisites of reinforcement learning, but about the textbooks of mathematics that are enough to understand the literature on reinforcement learning. What are the ...
user avatar
  • 3,191
0 votes
0 answers
19 views

Model learning only to not to trade in a static sequence

I am learning RL and using out-of-the-box Stable-Baselines 3 PPO algorithm, I made a custom environment with the following ...
user avatar
0 votes
1 answer
25 views

Is this simple game solvable with reinforcment learning?

Let's imagine this simple environment : Each episode has a length of 1 step. Each action leads to a reward for this action. The action space is of 3 : 'UP', 'DOWN', 'UNKNOWN' Most of the time, the ...
user avatar
0 votes
0 answers
12 views

How to deal with multiple jumps in checkers for a reinforcement learning implememtation?

I have written my own envirnoment without OpenAI Gym. I had some ideas, I just want to hear you, if they are bad for the learning later on. Here is what a catch looks like. This is how I have ...
user avatar
3 votes
0 answers
34 views

What is the correct interpretation of the discount factor in MDPs?

In infinite-horizon MDPs one can consider the expected discounted return from a distribution of start states as the objective[^1]. i.e. $\mathbb{E}[V^{\pi}(S_0)] = \mathbb{E}[G_0] = \mathbb{E}[\sum_{t=...
user avatar
  • 31
1 vote
1 answer
28 views

What does "All store and access operations (for S(t) , A(t), and R(t)) can take their index mod n + 1" mean?

It's from the book Introduction to Reinforcement Learning. Second edition, chapter7: n-step Bootstrapping, page 147, n-step Sarsa. I made the algo work, but I still don't understand the phrase. ...
user avatar
  • 97
0 votes
1 answer
36 views

Is my derivation of the Bellman equation for $q_{\pi}$ in terms of $p(s'|s,a)$ and $r(s,a)$ correct?

I have done exercise 3.29 from Sutton and Barto and I'd like to check if it's correct. Here's the exercise: Rewrite the Bellman equation for the function $q_{\pi}$ in terms of the three argument ...
user avatar
  • 169
0 votes
0 answers
26 views

Is my derivation of the Bellman equation for $v_{*}$ in terms of $p(s'|s,a)$ and $r(s,a)$ correct?

I have done exercise 3.29 from Sutton and Barto and I'd like to check if it's correct. Here's the exercise: Rewrite the Bellman equation for the function $v_*$ in terms of the three argument function ...
user avatar
  • 169

1
2 3 4 5
…
42