Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

715 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
2 votes
0 answers
263 views

What is the return-to-go in reinforcement learning?

In reinforcement learning, the return is defined as some function of the rewards. For example, you can have the discounted return, where you multiply the rewards received at later time steps by ...
user avatar
  • 34.3k
2 votes
1 answer
61 views

What are the strategies for computationally heavy environments or long-time waiting environments?

I have an environment that is computationally heavy (takes several seconds to get a reward and next state). This limits reinforcement capability, due to poor sampling of the problem. There is any ...
user avatar
2 votes
0 answers
35 views

Is there a UCB type algorithm for linear stochastic bandit with lasso regression?

Why is there no upper confidence bound algorithm for linear stochastic bandits that uses lasso regression in the case that the regression parameters are sparse in the features? In particular, I don't ...
user avatar
  • 21
2 votes
0 answers
114 views

Handling a Large Discrete Action Space in Deep Q Learning

I am attempting to solve a timetabling problem using deep Q learning. It could be thought of as a resource allocation problem to obtain some certificate of 'optimality'. However, how to define and ...
user avatar
2 votes
0 answers
51 views

What are some other real-life examples of simple policies but complex value functions?

Hado van Hasselt, a researcher at DeepMind, mentioned in one of his videos (from 7:20 to 8:20) on Youtube (about policy gradient methods) that there are cases when the policy is very simple compared ...
user avatar
2 votes
0 answers
106 views

Combine DQN with the Average Reward setting

I have to deal with a non-episodic task, where there is addittionally a continuous state space and more specifically in each time step there is always a new state that has never been seen before. I ...
user avatar
  • 889
2 votes
0 answers
27 views

Do we assume the policy to be deterministic when proving the optimality?

In reinforcement learning, when we talk about the principle of optimality, do we assume the policy to be deterministic?
user avatar
2 votes
0 answers
86 views

Should I use the discounted average reward as objective in a finite-horizon problem?

I am new to reinforcement learning, but, for a finite horizon application problem, I am considering using the average reward instead of the sum of rewards as the objective. Specifically, there are a ...
user avatar
  • 121
2 votes
0 answers
28 views

Alternatives to Hierarchical RL for centralized control tasks?

Consider a problem where the agent must learn to control a hierarchy of agents acting against another such agent in a competitive environment. The agents on each team need to learn cooperate in order ...
user avatar
  • 121
2 votes
0 answers
61 views

How to combine two differently equally important signals into the reward function, that have different scales?

I have two signals that I want to use to model my reward. The first one is the CPU TIME: running mean from this diagram: The second one is the MAX RESIDUAL from this diagram: Since they are both ...
user avatar
  • 362
2 votes
0 answers
80 views

What happens if our target network overestimates the value?

When we use DDQN, we often use the target network in case our online network overestimates a value, but this doesn't make sense to me, because What happens if our target network is the one that ...
user avatar
  • 269
2 votes
0 answers
112 views

Most of state-action pairs remain unvisited in the q-table

In building my first Q-learning algorithm for OpenAI gym's CartPole problem, many of my states remain unvisited. I believe it is the reason that my agent does not learn. Can I be told of the reasons I ...
user avatar
  • 331
2 votes
1 answer
95 views

What is the advantage of using experience replay (as opposed to feeding it sequential data)?

Let's suppose that our RL agent needs to play a game with different levels. If we train our RL agent sequentially or with sequential data, our agent will learn how to play level 1, but then it will ...
user avatar
  • 269
2 votes
0 answers
63 views

If the performance of an RL agent in a partially observable environment is "good", is this likely only accidental?

In my research, I remember to have read that, in case of an environment which can be modeled by partially observable MDP, there are no convergence guarantees (unfortunately, I do not find the paper ...
user avatar
  • 291
2 votes
0 answers
588 views

Why is DDPG not learning and it does not converge?

I have used a different setting, but DDPG is not learning and it does not converge. I have used these codes 1,2, and 3 and I used different optimizers, activation functions, and learning rate but ...
user avatar
2 votes
0 answers
127 views

What kind of policy evaluation and policy improvement AlphaGo, AlphaGo Zero and AlphaZero are using

I'm trying to find out what kind of policy improvement and policy evaluation AlphaGo, AlphaGo Zero, and AlphaZero are using. By looking into their respective paper and SI, I can conclude that it is a ...
user avatar
2 votes
0 answers
59 views

Prioritised Remembering in Experience Replay (Q-Learning)

I'm using Experience Replay based on the original Prioritized Experience Replay (PER) paper. In the paper authors show ~ an order of magnitude increase in data efficiency from prioritized sampling. ...
user avatar
2 votes
0 answers
52 views

How can I formulate a prediction problem (given labeled data) as an RL problem and solve it with Q-learning?

One of my friends sent me a problem he was working on lately, and I couldn't help but I wonder how could it be solved using Q-learning. The statement is as follows: Given the following datasets, the ...
user avatar
  • 21
2 votes
0 answers
86 views

Solving multi-armed bandit problems with continuous action space

My problem has a single state and an infinite amount of actions on a certain interval (0,1). After quite some time of googling I found a few paper about an algorithm called zooming algorithm which can ...
user avatar
  • 21
2 votes
0 answers
56 views

What is the proof that the variance of the gradient estimate in Actor-Critic is smaller than in REINFORCE?

The intuition provided when introducing actor-critic algorithms is that the variance of its gradient estimates is smaller than in REINFORCE as, e.g., discussed here. This intuition makes sense for the ...
user avatar
  • 121
2 votes
1 answer
197 views

Continuous state and continuous action Markov decision process time complexity estimate: backward induction VS policy gradient method (RL)

Model Description: Model based(assume known of the entire model) Markov decision process. Time($t$): Finite horizon discrete time with discounting factor State($x_t$): Continuous multi-dimensional ...
user avatar
2 votes
0 answers
247 views

What is the proof that "reward-to-go" reduces variance of policy gradient?

I am following the OpenAI's spinning up tutorial Part 3: Intro to Policy Optimization. It is mentioned there that the reward-to-go reduces the variance of the policy gradient. While I understand the ...
user avatar
2 votes
0 answers
173 views

If the minimum Q value is decreasing and the maximum Q value increasing, is this a sign that dueling double DQN is diverging?

I'm training a dueling double DQN agent with prioritized replay buffer and notice that the min Q values are decreasing, while the max Q values are increasing. Is this a sign that it is diverging? ...
user avatar
2 votes
1 answer
207 views

How to implement RAM versions of Atari games

I have coded the breakout RAM version, but, unfortunately, its highest reward was 5. I trained it for about 2 hours and never reached a higher score. The code is huge, so I can't paste here, but, in ...
user avatar
2 votes
0 answers
79 views

How can the target rely on untrained parameters?

I'm trying to understand DQN. I understand where the loss function comes from. I'm just unsure about why the target function works in practice. Given the loss function $$ L_i(\theta_i) = [(y_i - Q(s,a;...
user avatar
2 votes
0 answers
90 views

Are the final states not being updated in this $n$-step Q-Learning algorithm?

I am reading this paper and in algorithm 3 they describe an $n$-step Q-Learning algorithm. Below is the pseudo-code. From this pseudo-code, it looks as though the final tuples that they would ...
user avatar
2 votes
0 answers
130 views

How to prevent deep Q-learning algorithms to overfit?

I have recently solved the Cartpole problem using double deep Q-learning. When I saw how the agent was doing, it used to go right every time, never left, and it did similar actions all the time. Did ...
user avatar
2 votes
0 answers
64 views

Can you find another reason for sample inefficiency of model-free on-policy Deep Reinforcement Learning?

The following mindmap gives an overview of multiple reasons for sample inefficiency. The list is definitely not complete. Can you see another reason not mentioned so far? Some related links: ...
user avatar
2 votes
0 answers
59 views

Can weighted importance sampling be applied to off-policy evaluation for continuous state space MDPs?

Can weighted importance sampling (WIS) and importance sampling (IS) be applied to off-policy evaluation for continuous state spaces MDPs? Given that I have trajectories of $(s_t,a_t)$ pairs and the ...
user avatar
  • 1,221
2 votes
0 answers
70 views

DQN not showing the agent is learning in a snake grid environment game

I've been trying to train a snake for the snake game in DQN. Which the snake can essentially just move up, down, left and right. I'm having a hard time getting the snake to stay alive longer. So my ...
user avatar
2 votes
0 answers
35 views

Is there an online RL algorithm that receives as input a camera frame and produces an action as output?

I want to build a reinforcement learning model, which takes a camera picture as input, that learns online (in terms of machine learning). Based on the position of an object on the camera, I want the ...
user avatar
  • 21
2 votes
0 answers
61 views

When to do discretization to decrease the state/action space in RL?

When to do discretization to decrease the state/action space in RL? Can you give me some references that such a technique is used?
user avatar
  • 889
2 votes
1 answer
186 views

How long should the state-dependent baseline for policy gradient methods be trained at each iteration?

How long should the state-dependent baseline be trained at each iteration? Or what baseline loss should we target at each iteration for use with policy gradient methods? I'm using this equation to ...
user avatar
2 votes
0 answers
54 views

Can a typical supervised learning problem be solved with reinforcement learning methods?

Let's say I want to teach a neural to classify images, and, for some reason, I insist on using reinforcement learning rather than supervised learning. I have a dataset of images and their matching ...
user avatar
2 votes
0 answers
118 views

PPO algorithm converges on only one action

I have taken some reference implementations of PPO algorithm and am trying to create an agent which can play space invaders . Unfortunately from the 2nd trial onwards (after training the actor and ...
user avatar
2 votes
0 answers
142 views

How can DDPG handle the discrete action space?

I am wondering how can DDPG or DPG handle the discrete action space. There are some papers saying that use Gumbel softmax with DDPG can make the discrete action problem be solved. However, will the ...
user avatar
  • 41
2 votes
0 answers
180 views

What is the original source of the TD Advantage Actor-Critic algorithm?

What is the original source of the TD Advantage Actor-Critic algorithm? I found this tutorial really helpful for learning the algorithm. However, what is the original source of this algorithm?
user avatar
  • 121
2 votes
1 answer
85 views

Adversarial Q Learning should use the same Q Table?

I'm creating a RF Q-Learning agent for a two player fully-observable board game and wondered, if I was to train the Q Table using adversarial training, should I let both 'players' use, and update, the ...
user avatar
  • 583
2 votes
0 answers
125 views

What is the gradient of the Q function with respect to the policy's parameters?

I have been recently studying Actor-Critic algorithms, and I ran into the following question. Let $Q_{\omega}$ be the critic network, and $\pi_{\theta}$ be the actor. It is known that in order to ...
user avatar
  • 151
2 votes
0 answers
70 views

How are the classical MDP and the object-oriented MDP views different?

I've been reading the attached paper - which aims to model entities in the world as objects, including the learning agent itself! To say the least, the goal is to navigate through what seems like a ...
user avatar
2 votes
0 answers
41 views

How should I design a reward function for a NLP problem where two models interoperate?

I would like to design a reward function. I am training two models from the first model that classify set of texts (paragraphs and keywords) and I also got some hidden states. The second model is ...
user avatar
  • 21
2 votes
0 answers
124 views

Action masking for on policy algorithm like PPO

I have an environment, in which my agent learns according to PPO. The environment has a maximum of 80 actions, however not all of them are always allowed. My idea was to mask them, by setting the ...
user avatar
2 votes
0 answers
40 views

How can I formalise a non-zero-sum game of $N$ agent as Markov game?

I coded a non-zero-sum game of $N$ agents in a discrete dynamic environment to RL with Q-learning and DQN agents. It's like a marathon. Only two actions are available per agent: $\{ G \text{ (move ...
user avatar
  • 21
2 votes
0 answers
54 views

Is Mean Squared Error Loss function a good loss function for continuous variables $0 < x < 1$

Suppose I am utilising a neural network to predict the next state, $s'$ based on the current $(s, a)$ pairs. all my neural network inputs are between 0 and 1 and the loss function for this network ...
user avatar
  • 1,221
2 votes
0 answers
35 views

How was the DQN trained to play many games?

Some people claim that DQN was used to play many Atari games. But what actually happened? Was DQN trained only once (with some data from all games) or was it trained separately for each game? What was ...
user avatar
  • 583
2 votes
0 answers
32 views

How should I define the reward function for the Connect Four game?

I'm creating an RL application for the game Connect Four. I've researched the different strategies for the game and which positions are more favourable to lead to a win. Should I be assigning ...
user avatar
  • 583
2 votes
0 answers
177 views

Whats the correct loss function to use during deep Q-learning (discrete action space)

After playing around with normal Q-learning I have decided to switch to deep Q-learning and I have encountered this problem. As I understand, for a task with discrete action space, where there are 4 ...
user avatar
  • 151
2 votes
0 answers
69 views

Why is my DQN model not getting better?

I've created a deep Q network. My model does not get better, and can't see what I'm doing wrong. I'm new to RL. Replay Memory ...
user avatar
2 votes
0 answers
162 views

Policy Gradient Reward Oscillation in MATLAB

I'm trying to train a Policy Gradient Agent with Baseline for my RL research. I'm using the in-built RL toolbox from MATLAB (https://www.mathworks.com/help/reinforcement-learning/ug/pg-agents.html) ...
user avatar
  • 21
2 votes
0 answers
61 views

How to draw backup diagram when reward is in expectation but next state is iterated?

I am working through Sutton and Barto's RL book. So far in the text, when backup diagrams are drawn, the reward and next state are iterated together (i.e. the equations always have $\sum_{s',r}$), ...
user avatar
  • 171

1 2 3
4
5
15