Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

Filter by
Sorted by
Tagged with
0 votes
0 answers
5 views

How to quickly train your agent to get a sense of the problem?

I have a custom implementation of DQN. My robot/agent is running in Gazebo with ROS. Though I am trying a very simple task of pushing a cube, the DQN agent is taking too much time. DQN of 300 episode ...
user avatar
0 votes
1 answer
19 views

PPO vs SAC for discrete action spaces

I am currently using Proximal Policy Optimization (PPO) to solve my RL task. However, after reading about Soft Actor-Critic (SAC) now I am unsure whether I should stick to PPO or switch to SAC. ...
user avatar
  • 13
3 votes
1 answer
24 views

$E_{\pi}[R_{t+1}|S_t=s,A_t=a] = E[R_{t+1}|S_t=s,A_t=a]$?

I would like to solve the first question of Exercise 3.19 from Sutton and Barto: Exercise 3.19 The value of an action, $q_{\pi}(s, a)$, depends on the expected next reward and the expected sum of the ...
user avatar
  • 143
1 vote
0 answers
24 views

Is it possible for AlphaGo Zero to use recurrent networks to achieve similar performance?

AlphaGo Zero stacks 7 board history along with the current board together to form the input to the network. However, is it possible to use an RNN to replace the input of history and achieve similar ...
user avatar
  • 183
3 votes
1 answer
308 views

Is there a standardized method to train a reinforcement learning NN by demonstration?

I'm less familiar with reinforcement learning compared to other neural network learning approaches, so I'm unaware of anything exactly like what I want for an approach. I'm wondering if there are any ...
user avatar
-1 votes
0 answers
16 views

What known routes to AGI are there, other than whole brain emulation and deep RL? [closed]

Clearly, a very physically realistic simulation of the brain would suffice as AGI. There's also the prospect of achieving AGI through reinforcement learning, which DeepMind has convincingly argued is ...
user avatar
  • 1
0 votes
0 answers
29 views

CS 285 Prof Sergey Levine Lecture, Bounding Derivation for Reinforcement Learning (TRPO)

How can we derive the final result? I can understand the first line, but don't know how the absolute term in the summation is replaced with $2\epsilon t$. https://www.youtube.com/watch?v=LtAt5M_a0dI&...
user avatar
  • 101
0 votes
0 answers
18 views

What reinforcement learning architecture is recommended for multiple outputs in continuous resource management?

I would like to develop an agent to provide resources to multiple machines simultaneously. The overall resources are limited. The agent should distribute the resources in such a way that the machines ...
user avatar
0 votes
1 answer
50 views

Why and when do we need to normalize weights in Reinforcement Learning?

I recently came across this SO question, wherein the poster was asked to normalize their weights while using a function approximator with SARSA. I don't remember having to normalize any weights while ...
user avatar
0 votes
0 answers
9 views

Is the interpretation of the "batch size" in policy gradient algorithms the number of trajectories sampled in VPG and TRPO?

Good afternoon. I would like to shore up my interpretation of the concept of "batch size". It is my understanding that in Vanilla Policy Gradients and TRPO, the "batch size" is the ...
user avatar
0 votes
0 answers
8 views

Random Network Distillation for short episodic tasks

I have a task with short episodic steps (maybe up to 20) that I'm training for using PPO, but it seems to get stuck easily on local optima. Searching for solutions to this, I've stumbled upon Random ...
user avatar
0 votes
0 answers
8 views

Correct reward for trading applications. RL agent learns according to immediate reward instead of cumulative reward

I have coded a RL environment for trading. The action space is discrete with 3 components [0,1,2]; where 0 corresponds to selling an amount of shares; 1 corresponds to holding; and 2 corresponds to ...
user avatar
  • 49
0 votes
1 answer
84 views

How do I create an AI controller for Pacman?

How do I create an AI controller, which can play pacman - by taking in pixel values (or some other data by represents the state) which perhaps runs on a separate thread, which can control the game? It ...
user avatar
  • 11
1 vote
0 answers
34 views

Number of possible joint policies in a Dec-POMDP and the time required to evaluate each one

I was reading a book about Dec-POMDPs and came across this curious result where the author specifies the number of possible joint policies to evaluate and the time needed to evaluate a single joint ...
user avatar
-1 votes
1 answer
24 views

How to form 10 and 20 actions in corridor environment, in the paper "Dueling Network Architectures for Deep Reinforcement Learning"? [closed]

I am just reading a paper titled "Dueling Network Architectures for Deep Reinforcement Learning". In this paper, 4. Experiment (4.1 Policy evaluation), I just wonder how to form 5, 10, and ...
user avatar
1 vote
1 answer
28 views

Is the described Q-table considered large?

I never saw any rule of thumb as to what size is said as large for a q-table but I have a Q-table with like 2500 entries. Is it considered large for a tabular approach? Anyone from experience can ...
user avatar
1 vote
2 answers
35 views

How many training steps does it usually take to train an RL model?

This is my model average rewards as follow image. How to tell if it is undertrained or not convergent? How many training steps does it usually take to train an RL model? And I'm using PPO to train.
user avatar
  • 123
0 votes
1 answer
50 views

How to compare memory requirements for tabular Q-learning vs deep neural network?

I want to compare the space complexity/memory requirement of tabular Q-learning v.s. deep neural Q-network (DQN). I think DQN would be faster and Q-table has a disadvantage at large table sizes but ...
user avatar
0 votes
0 answers
26 views

How to get into robotic simulation for RL purposes? [closed]

For my master's thesis, I've joined a robotics team that tries to build a flying robot based on the mechanism of flies (i.e - two wings that perform reciprocating motion to generate lift). My part in ...
user avatar
0 votes
0 answers
30 views

Is Bayesian Reinforcement Learning used as off-policy RL?

Are there any examples where Bayesian Reinforcement Learning is used as off-policy RL? What are the pros and cons of using it for this purpose?
user avatar
0 votes
0 answers
21 views

Why is it difficult to train large RL networks?

First of all I know that: 'it makes training less stable' & 'RL is already inherently unstable'. I'm asking why those things are true? Intuitively it seems very strange & to be perhaps a ...
user avatar
  • 276
1 vote
2 answers
53 views

Should PPO always converge toward the global optimum?

I'm trying to "solve" the OpenAI gym environment "Humanoid-v3" using PPO. I got it to work to some degree (The NN is learning a policy and perfecting it. Average reward of about 5....
user avatar
  • 13
1 vote
0 answers
16 views

Reinforcement Learning with constant reward in constant episodes time length

I have a situation where I'm trying to maximize the number of steps in a fixed training time frame. It's possible that specific steps will lead to a delay until the agent can act again, and thus less ...
user avatar
1 vote
1 answer
48 views

How to tune hypeparametes in A2C-ppo?

Im currently working with A2C. The model was able to learn open ai pong, i ran this as a sanity check that i havent made any bugs. Now im trying to make the model play breakout, but still after 10m ...
user avatar
2 votes
1 answer
27 views

How do we estimate the value of a stochastic policy?

I'm learning about reinforcement learning, particularly policy gradient methods and actor-critic methods. I've noticed that many algortihms use stochastic policies during training (i.e. they select ...
user avatar
3 votes
1 answer
53 views

Why is a large replay buffer inefficient?

Open AI spin up says ... the replay buffer should be large enough to contain a wide range of experiences, but it may not always be good to keep everything. If you only use the very-most recent data, ...
user avatar
  • 131
0 votes
0 answers
30 views

What if we modify some Q-values while taking the action?

Just a passing thought about Q-learning. In the tabular Q-learning, what if I play around and modify any Q-values as I am using them to take actions? Would it be a violation of any (1) theoretical ...
user avatar
0 votes
0 answers
46 views

What method to use when optimizing an array of data

Say I have an array of data, where each element describes a shape made of points, in vector form (each vector has several hundred dimensions). Each element also has a rating that gets higher, the ...
user avatar
0 votes
0 answers
43 views

RL-based trading bot: how to deal with overfitting

I've been playing around building a reinforcement learned-based trading bot using the stable-baselines3 library. I've come up with an environment that seems to be able to learn how to make profitable ...
user avatar
  • 101
0 votes
1 answer
40 views

Is it possible to add states to the Q-table after the game has started?

I would like to implement Q-learning in a game. Here is the board: It's a 2 player game. At each turn, each player can put a pawn on a line of their choice. They can't choose the column. The right ...
user avatar
  • 3
1 vote
0 answers
8 views

DDPG agent with convolutional layers for feature extraction [closed]

I'm trying to come up with a definition of the critic for a DDPG agent in PyTorch using a CNN as a feature extractor. It is pretty straight forward for the actor model. However, for the critic model I ...
user avatar
1 vote
1 answer
45 views

Does it make sense to provide a DQN with negative rewards for a network with relu and sigmoid activations?

The creation of negative rewards leads to the chance of Q-values being negative. However, networks with relu or sigmoid activations, just cannot predict negative values. This will lead to a case where ...
user avatar
0 votes
1 answer
33 views

How to deal with changing rewards in Q-learning? DQN?

I read the working of Q-learning through a grid-based taxi routing wherein a taxi has to pick and drop off a passenger from source to destination. Likewise, I have a routing problem and hence, I tried ...
user avatar
1 vote
1 answer
27 views

PPO: multiple discrete actions per step, one depends on the other

I have a custom PPO implementation, and it works fine, but I need to add to it the ability to select 2 actions per turn, one different in nature from the other, one dependent on the other. Imagine ...
user avatar
0 votes
0 answers
27 views

Why are agents trained in episodes, even in non-episodic tasks?

Let's consider some non-episodic problem. Maybe a game which can go on forever. My question is: Why are agents still trained in episodes? My understanding is that the agent's neural network is updated ...
user avatar
4 votes
1 answer
928 views

Mathematically, what is happening differently in the neural net during exploration vs. exploitation?

I want to understand roughly what is happening in the neural network of an RL agent when it is exploring vs. exploiting. For example, are the network weights not being updated when the agent is ...
user avatar
4 votes
6 answers
2k views

Why is exploitation necessary during training?

I have read many blog articles making all kinds of broad analogies to explain the exploration/exploitation trade-off. However, I still can't fully grasp it. On an extremely abstract level, I ...
user avatar
1 vote
1 answer
72 views

Which RL algorithm should I use to learn an optimal weight vector?

What is the best practice in order to learn the optimal weight vector $W^*$? By optimal I mean the weights that will produce the agent with the highest win-rate. I have an agent that plays a ...
user avatar
  • 365
0 votes
0 answers
62 views

How to solve a reinforcement learning problem with a stochastic reward function?

In a discrete time system, an environment has an unknown reward probability $p(r|s,a)$. However, the transition probability $p(s'\mid s,a)$ is deterministic. In my case, the reward for the same action ...
user avatar
0 votes
0 answers
25 views

Ongoing competitions on reinforcement learning? [duplicate]

Are there any ongoing competitions except the following? Battlesnake, SC2 AI Arena, Bomberland, ConnectX
user avatar
2 votes
0 answers
66 views

How to solve a reinforcement learning problem with changing rewards?

I'm working on a problem with non-stationary environments. The state space is discrete and limited. The action is limited too. But the reward for the same action $a$ can change. Even the reward for ...
user avatar
0 votes
0 answers
25 views

PPO: how to scale rewards

I have a custom PPO implementation and a problem that has costs rather than rewards, so I basically need to take the negative value for PPO to work. As the values are somewhat large, I've tried ...
user avatar
2 votes
1 answer
36 views

For continuing tasks, is the choice of episode length completely arbitrary?

Let's say I'm training a reinforcement learning agent to act in some environment that perpetually continues to give the agent opportunities to earn rewards, and there is no cap on the score and there ...
user avatar
5 votes
1 answer
138 views

Why are only neural networks (and not SVMs, for example) used for reinforcement learning?

I know that neural networks are the "universal function approximator", but they also have a huge number of trainable parameters and are extremely prone to overfitting. So my question is: Why ...
user avatar
7 votes
2 answers
492 views

In Value Iteration, why can we initialize the value function arbitrarily?

I have not been able to find a good explanation of this, other than statements that the algorithm is guaranteed to converge with arbitrary choices for initial values in each state. Is this something ...
user avatar
  • 73
4 votes
2 answers
415 views

Is there a way to easily simulate video games, without actually rendering the pixels on screen?

Youtube was recently suggesting to me videos of people training NEAT neural networks for video games. I've noticed that often the training process was quite slow (for example in this Trackmania ...
user avatar
0 votes
1 answer
52 views

Why aren't neural networks contractions?

I'm not sure I understand why neural networks aren't considered contractions, as Geoffrey J. Gordon says in his paper: Stable Function Approximation in Dynamic Programming: "Our theorems in the ...
user avatar
1 vote
0 answers
29 views

When should I use an MARL approach instead of training one agent while keep the others fixed?

I have built a custom multi-agent environment with PettingZoo, where a turn-based game with two agents, A and B, is setup. I want to examine situations where malicious behavior may arise, given the ...
user avatar
  • 111
1 vote
0 answers
19 views

Is it possible to combine two policy-based RL agents?

I am developing an RL agent for a game environment. I have found out that there are two strategies to do well in the game. So I have trained two RL agents using neural networks with distinct reward ...
user avatar
1 vote
1 answer
35 views

In RL, is it possible to design a multiplicative/exponential reward function? A reward func that depends on current accumulated reward?

In the context of my problem, the "true" reward is not additive. Realistically, the more reward the agent has already accumulated, the easier it becomes to accumulate even more. That's to ...
user avatar

1
2 3 4 5
41