# Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

2,039 questions
Filter by
Sorted by
Tagged with
5 views

### How to quickly train your agent to get a sense of the problem?

I have a custom implementation of DQN. My robot/agent is running in Gazebo with ROS. Though I am trying a very simple task of pushing a cube, the DQN agent is taking too much time. DQN of 300 episode ...
19 views

### PPO vs SAC for discrete action spaces

I am currently using Proximal Policy Optimization (PPO) to solve my RL task. However, after reading about Soft Actor-Critic (SAC) now I am unsure whether I should stick to PPO or switch to SAC. ...
• 13
24 views

### $E_{\pi}[R_{t+1}|S_t=s,A_t=a] = E[R_{t+1}|S_t=s,A_t=a]$?

I would like to solve the first question of Exercise 3.19 from Sutton and Barto: Exercise 3.19 The value of an action, $q_{\pi}(s, a)$, depends on the expected next reward and the expected sum of the ...
• 143
1 vote
24 views

### Is it possible for AlphaGo Zero to use recurrent networks to achieve similar performance?

AlphaGo Zero stacks 7 board history along with the current board together to form the input to the network. However, is it possible to use an RNN to replace the input of history and achieve similar ...
• 183
308 views

### Is there a standardized method to train a reinforcement learning NN by demonstration?

I'm less familiar with reinforcement learning compared to other neural network learning approaches, so I'm unaware of anything exactly like what I want for an approach. I'm wondering if there are any ...
16 views

### What known routes to AGI are there, other than whole brain emulation and deep RL? [closed]

Clearly, a very physically realistic simulation of the brain would suffice as AGI. There's also the prospect of achieving AGI through reinforcement learning, which DeepMind has convincingly argued is ...
29 views

### CS 285 Prof Sergey Levine Lecture, Bounding Derivation for Reinforcement Learning (TRPO)

How can we derive the final result? I can understand the first line, but don't know how the absolute term in the summation is replaced with $2\epsilon t$. https://www.youtube.com/watch?v=LtAt5M_a0dI&...
• 101
18 views

### What reinforcement learning architecture is recommended for multiple outputs in continuous resource management?

I would like to develop an agent to provide resources to multiple machines simultaneously. The overall resources are limited. The agent should distribute the resources in such a way that the machines ...
50 views

### Why and when do we need to normalize weights in Reinforcement Learning?

I recently came across this SO question, wherein the poster was asked to normalize their weights while using a function approximator with SARSA. I don't remember having to normalize any weights while ...
9 views

### Is the interpretation of the "batch size" in policy gradient algorithms the number of trajectories sampled in VPG and TRPO?

Good afternoon. I would like to shore up my interpretation of the concept of "batch size". It is my understanding that in Vanilla Policy Gradients and TRPO, the "batch size" is the ...
8 views

### Random Network Distillation for short episodic tasks

I have a task with short episodic steps (maybe up to 20) that I'm training for using PPO, but it seems to get stuck easily on local optima. Searching for solutions to this, I've stumbled upon Random ...
8 views

### Correct reward for trading applications. RL agent learns according to immediate reward instead of cumulative reward

I have coded a RL environment for trading. The action space is discrete with 3 components [0,1,2]; where 0 corresponds to selling an amount of shares; 1 corresponds to holding; and 2 corresponds to ...
• 49
84 views

### How do I create an AI controller for Pacman?

How do I create an AI controller, which can play pacman - by taking in pixel values (or some other data by represents the state) which perhaps runs on a separate thread, which can control the game? It ...
• 11
1 vote
34 views

### Number of possible joint policies in a Dec-POMDP and the time required to evaluate each one

I was reading a book about Dec-POMDPs and came across this curious result where the author specifies the number of possible joint policies to evaluate and the time needed to evaluate a single joint ...
24 views

### How to form 10 and 20 actions in corridor environment, in the paper "Dueling Network Architectures for Deep Reinforcement Learning"? [closed]

I am just reading a paper titled "Dueling Network Architectures for Deep Reinforcement Learning". In this paper, 4. Experiment (4.1 Policy evaluation), I just wonder how to form 5, 10, and ...
1 vote
28 views

### Is the described Q-table considered large?

I never saw any rule of thumb as to what size is said as large for a q-table but I have a Q-table with like 2500 entries. Is it considered large for a tabular approach? Anyone from experience can ...
1 vote
35 views

### How many training steps does it usually take to train an RL model?

This is my model average rewards as follow image. How to tell if it is undertrained or not convergent? How many training steps does it usually take to train an RL model? And I'm using PPO to train.
• 123
50 views

### How to compare memory requirements for tabular Q-learning vs deep neural network?

I want to compare the space complexity/memory requirement of tabular Q-learning v.s. deep neural Q-network (DQN). I think DQN would be faster and Q-table has a disadvantage at large table sizes but ...
26 views

### How to get into robotic simulation for RL purposes? [closed]

For my master's thesis, I've joined a robotics team that tries to build a flying robot based on the mechanism of flies (i.e - two wings that perform reciprocating motion to generate lift). My part in ...
30 views

### Is Bayesian Reinforcement Learning used as off-policy RL?

Are there any examples where Bayesian Reinforcement Learning is used as off-policy RL? What are the pros and cons of using it for this purpose?
21 views

### Why is it difficult to train large RL networks?

First of all I know that: 'it makes training less stable' & 'RL is already inherently unstable'. I'm asking why those things are true? Intuitively it seems very strange & to be perhaps a ...
• 276
1 vote
53 views

### Should PPO always converge toward the global optimum?

I'm trying to "solve" the OpenAI gym environment "Humanoid-v3" using PPO. I got it to work to some degree (The NN is learning a policy and perfecting it. Average reward of about 5....
• 13
1 vote
16 views

### Reinforcement Learning with constant reward in constant episodes time length

I have a situation where I'm trying to maximize the number of steps in a fixed training time frame. It's possible that specific steps will lead to a delay until the agent can act again, and thus less ...
• 108
1 vote
48 views

### How to tune hypeparametes in A2C-ppo?

Im currently working with A2C. The model was able to learn open ai pong, i ran this as a sanity check that i havent made any bugs. Now im trying to make the model play breakout, but still after 10m ...
27 views

### How do we estimate the value of a stochastic policy?

I'm learning about reinforcement learning, particularly policy gradient methods and actor-critic methods. I've noticed that many algortihms use stochastic policies during training (i.e. they select ...
53 views

### Why is a large replay buffer inefficient?

Open AI spin up says ... the replay buffer should be large enough to contain a wide range of experiences, but it may not always be good to keep everything. If you only use the very-most recent data, ...
• 131
30 views

### What if we modify some Q-values while taking the action?

Just a passing thought about Q-learning. In the tabular Q-learning, what if I play around and modify any Q-values as I am using them to take actions? Would it be a violation of any (1) theoretical ...
46 views

### What method to use when optimizing an array of data

Say I have an array of data, where each element describes a shape made of points, in vector form (each vector has several hundred dimensions). Each element also has a rating that gets higher, the ...
43 views

### RL-based trading bot: how to deal with overfitting

I've been playing around building a reinforcement learned-based trading bot using the stable-baselines3 library. I've come up with an environment that seems to be able to learn how to make profitable ...
• 101
40 views

### Is it possible to add states to the Q-table after the game has started?

I would like to implement Q-learning in a game. Here is the board: It's a 2 player game. At each turn, each player can put a pawn on a line of their choice. They can't choose the column. The right ...
1 vote
8 views

### DDPG agent with convolutional layers for feature extraction [closed]

I'm trying to come up with a definition of the critic for a DDPG agent in PyTorch using a CNN as a feature extractor. It is pretty straight forward for the actor model. However, for the critic model I ...
1 vote
45 views

### Does it make sense to provide a DQN with negative rewards for a network with relu and sigmoid activations?

The creation of negative rewards leads to the chance of Q-values being negative. However, networks with relu or sigmoid activations, just cannot predict negative values. This will lead to a case where ...
33 views

### How to deal with changing rewards in Q-learning? DQN?

I read the working of Q-learning through a grid-based taxi routing wherein a taxi has to pick and drop off a passenger from source to destination. Likewise, I have a routing problem and hence, I tried ...
1 vote
27 views

### PPO: multiple discrete actions per step, one depends on the other

I have a custom PPO implementation, and it works fine, but I need to add to it the ability to select 2 actions per turn, one different in nature from the other, one dependent on the other. Imagine ...
27 views

### Why are agents trained in episodes, even in non-episodic tasks?

Let's consider some non-episodic problem. Maybe a game which can go on forever. My question is: Why are agents still trained in episodes? My understanding is that the agent's neural network is updated ...
928 views

### Mathematically, what is happening differently in the neural net during exploration vs. exploitation?

I want to understand roughly what is happening in the neural network of an RL agent when it is exploring vs. exploiting. For example, are the network weights not being updated when the agent is ...
2k views

### Why is exploitation necessary during training?

I have read many blog articles making all kinds of broad analogies to explain the exploration/exploitation trade-off. However, I still can't fully grasp it. On an extremely abstract level, I ...
1 vote
72 views

### Which RL algorithm should I use to learn an optimal weight vector?

What is the best practice in order to learn the optimal weight vector $W^*$? By optimal I mean the weights that will produce the agent with the highest win-rate. I have an agent that plays a ...
• 365
62 views

### How to solve a reinforcement learning problem with a stochastic reward function?

In a discrete time system, an environment has an unknown reward probability $p(r|s,a)$. However, the transition probability $p(s'\mid s,a)$ is deterministic. In my case, the reward for the same action ...
25 views

### Ongoing competitions on reinforcement learning? [duplicate]

Are there any ongoing competitions except the following? Battlesnake, SC2 AI Arena, Bomberland, ConnectX
66 views

### How to solve a reinforcement learning problem with changing rewards?

I'm working on a problem with non-stationary environments. The state space is discrete and limited. The action is limited too. But the reward for the same action $a$ can change. Even the reward for ...
25 views

### PPO: how to scale rewards

I have a custom PPO implementation and a problem that has costs rather than rewards, so I basically need to take the negative value for PPO to work. As the values are somewhat large, I've tried ...
36 views

### For continuing tasks, is the choice of episode length completely arbitrary?

Let's say I'm training a reinforcement learning agent to act in some environment that perpetually continues to give the agent opportunities to earn rewards, and there is no cap on the score and there ...
138 views

### Why are only neural networks (and not SVMs, for example) used for reinforcement learning?

I know that neural networks are the "universal function approximator", but they also have a huge number of trainable parameters and are extremely prone to overfitting. So my question is: Why ...
492 views

### In Value Iteration, why can we initialize the value function arbitrarily?

I have not been able to find a good explanation of this, other than statements that the algorithm is guaranteed to converge with arbitrary choices for initial values in each state. Is this something ...
• 73
415 views

### Is there a way to easily simulate video games, without actually rendering the pixels on screen?

Youtube was recently suggesting to me videos of people training NEAT neural networks for video games. I've noticed that often the training process was quite slow (for example in this Trackmania ...
• 141
52 views

### Why aren't neural networks contractions?

I'm not sure I understand why neural networks aren't considered contractions, as Geoffrey J. Gordon says in his paper: Stable Function Approximation in Dynamic Programming: "Our theorems in the ...
1 vote
29 views

### When should I use an MARL approach instead of training one agent while keep the others fixed?

I have built a custom multi-agent environment with PettingZoo, where a turn-based game with two agents, A and B, is setup. I want to examine situations where malicious behavior may arise, given the ...
• 111
1 vote