Questions tagged [reinforcement-learning]

For questions related to learning controlled by external positive reinforcement or negative feedback signal or both, where learning and use of what has been thus far learned occur concurrently.

221 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
1
vote
1answer
25 views

Deepmind Spriteworld run_demo.py not found

I'm trying to run the Deepmind Spriteworld demo described on the project's GitHub page, but I'm not finding run_demo.py in the distribution and the closest sounding ...
1
vote
0answers
28 views

Reinforcement learning number of episodes per epoch not matching with paper

I am trying to reproduce results presented in this paper. On page 4, the authors state: ... we train for 50 epochs (one epoch consists of 19*2*50 = 1900 full episodes), which amounts to a total ...
1
vote
0answers
24 views

RF: How to deal with environments changing state due to external factors

I have a use case where the state of the environment could change due to random events in between time steps that the agent takes actions. For example, at t1, the agent takes action a1 and is given ...
1
vote
0answers
13 views

TD losses are descreasing, but also rewards are decreasing, increasing sigma

I'm using Q-learning with some extensions such as noisy linear layers, n-steps and double DQN. The training, however, isn't that successful, my rewards are descreasing over time after a steep ...
1
vote
0answers
53 views

Is Value Iteration better than Policy Iteration for first few iterations?

In Policy Iteration (PI), the action generated by the policy, whether it's optimal or not w.r.t the current value function $v(s)$. Whereas, in Value Iteration, the action is greedily generated w.r.t ...
1
vote
0answers
36 views

PPO: action std or entropy for exploration?

When trying to implement my own PPO (Proximal Policy Optimizer), I came accross two different implementations : Exploration with action std : Collect trajectories on ...
1
vote
0answers
19 views

Loss reduction, but constant performance with CNN

I made a CNN with a reasonable loss curve, but the performance of the model does not improve. I have tried making the model larger, I am using three convolutional layers with batch norms. Thanks for ...
1
vote
0answers
46 views

Is it possible to use Reward Function of type R(s, a, s') if more than one action is applied?

I am applying a reinforcement learning agent (PPO2, stable baselines implementation) to a custom built environment using OpenAI Gym. One reward function (formualted as loss function, that is, all ...
1
vote
0answers
30 views

How is computed the gradient with respect to each output node from a loss value?

newbie here. I am studying the REINFORCE method in "Deep Reinforcement Learning Hands-On". I can't understand how, after computing the loss of the episode, that loss is backpropagated in a NN with ...
1
vote
0answers
27 views

What is a high performing network architecture to use in a PPO2 MlpLnLstmPolicy RL model?

I am playing around with creating custom architectures in stable-baselines. Specifically I am training an agent using a PPO2 model. My question is, are there some rules of thumb or best practices in ...
1
vote
0answers
32 views

How to solve optimal control problem with reinforcement learning

The problem I am trying to attack is a predator-prey pursuit problem. There are multiple predators that pursue multiple preys and preys tried to evade predators. I am trying to solve a simplified ...
1
vote
0answers
103 views

How to use the LSTM layer in PPO architecture?

What is the best way of using the LSTM layer in PPO architecture? Should I use them in the first layer of both actor and critic, or use them just before the final layer of these networks? Should I ...
1
vote
0answers
37 views

Reward problem in A2C with multiple simultaneous discrete actions

I've built an A2C model whose actor's network has two different kinds of discrete actions, so the critic would take state and action (note that critic takes 2 actions because in each timestep we will ...
1
vote
0answers
38 views

Unique game problem (ML, DP, PP etc)

Looking for a solution to my below game problem. I believe it to require some sort of reinforcement learning, dynamic programming, or probabilistic programming solution, but am unsure... This is my ...
1
vote
0answers
33 views

Can multiple reinforcement algorithms be applied to the same system?

Can a system, for instance robotic vehicle, be controlled by more than one reinforcement learning algorithm. I intend to use one to address collision avoidance whereas the other to tackle autonomous ...
1
vote
0answers
15 views

Can the agent of reinforcement learning system serve as the environment for other agents and expose actions as services?

Can the agent of reinforcement learning system serve as the environemnt for other agents and expose actions as services? Are there research that consider such question? I tried to formulate the ...
1
vote
0answers
62 views

How do I determine the generalisation ability of a neural network?

I am trying to ascertain if my neural network is able to generalize or if it’s simply using memory/overfitting to solve a task. I would like my model to generalise. Currently, I train the neural ...
1
vote
0answers
59 views

How can I develop this ML/AI system that I want to use in my new mobile app?

I have an idea for a new mobile app. Here is what I want to accomplish using AI; I want to get an image (png format), (maybe just byte data too), from my application (I'm developing with Unity3D/C#), ...
1
vote
0answers
51 views

Doubt regarding improvement of State Value by n-step returns

Excerpt from Sutton and Barto: The expected value of all $n$-step returns is guaranteed to improve in a certain way over the current value function as an approximation to the true value ...
1
vote
1answer
89 views

Deep Reinforcement Learning: Rewards suddenly dip down

I am working on a deep reinforcement learning problem. The policy network has the same architecture as the one Deepmind published in 'Playing Atari with Deep Reinforcement Learning'. I am also using ...
1
vote
0answers
28 views

How do I know if the assumption of a static environment is made?

An important property of a reinforcement learning problem is whether the environment of the agent is static, which means that nothing changes if the agent remains inactive. Different learning methods ...
1
vote
0answers
43 views

Can next state and action be same in Deep Deterministic Policy Gradient?

I am trying to apply deep deterministic policy gradient (DDPG) on a robotic application. My states consist of the joint angle positions of the robot and my actions are also its joint angle positions. ...
1
vote
0answers
80 views

Gym dict space as keras DQN agent input

I'm trying to make an AI to play my own card game. I have an OpenAI gym for the game with Dict as an observation space. It is nested dict, so I can't easily replace it with a tuple. I want to pass ...
1
vote
0answers
47 views

How should I deal with vector states in reinforcement learning?

I am wondering if anyone has any experience dealing with vector states, that is, a state that has multiple values. I cannot find any coding examples online and I am trying to implement Q-learning. Any ...
1
vote
0answers
145 views

Implementation of PPO - Value Loss not converging, return plateauing

Copy from my reddit post: (Sorry if this does not fit here, please tell me and i delete it) Help regarding I'm working on an implementation of PPO, which i plan to use in my (Bachelors) Thesis. To ...
1
vote
0answers
52 views

Hashed Tile Coding vs Regular Tile Coding

In the book "Reinforcement Learning: An Introduction" (2018) Sutton and Barto explain at page 221 a form of tile coding using hashing, to reduce memory consumption. I have two questions about that: ...
1
vote
0answers
35 views

Action spaces for an RTS game

I think reinforcement learning would be a good fit for this problem, but I am not sure of how to deal with a seemingly infinite number of actions. In the beginning of each game (generic RTS game), the ...
1
vote
0answers
21 views

Learning utility function for AIS data

I am trying to learn utility functions for ships through their AIS data. I have a lot of data available and plan on focusing on fishing boats. So far I've researched a lot of IRL algorithms but I'm ...
1
vote
0answers
23 views

Why should we use TD prediction as opposed to TD control algorithms?

Consider a problem where we have a finite number of states and actions. Given a state and an action, we can easily know the reward and the next state (deterministic). The state space is really large ...
1
vote
0answers
28 views

High variance in performance of q-learning agents trained with same parameters

I am training an agent to play a simple game using double deep q learning. However, the variance in agent performance is very high, even for agents trained with same model parameters. For example, I ...
1
vote
0answers
36 views

DQN not able to learn in a game where other agents perform random walks

I am making a school project where I should develop any kind of game where I can have one reactive agent and one agent based on machine learning competing with each other. My game consists of a ...
1
vote
0answers
16 views

DQN ANN input vs Linear function approximator feature vector

So when using semi-gradient td(0) you need to convert your state representation into a feature vector that represents the state and as far as I know, should not be correlated. Is the input on the ANN ...
1
vote
0answers
83 views

Do we need to use the experience replay buffer with the A3C algorithm?

I have skimmed through a bunch of deep learning books, but I have not yet understood whether we must use the experience replay buffer with the A3C algorithm. The approached I used is the following: ...
1
vote
0answers
19 views

What are a list of board game environments for RL practice?

Recently OpenAI removes their board game environments. (It may be possible to install an older version to get access to them, but I haven’t downgraded). Is there a list of repositories or resources ...
1
vote
0answers
23 views

Measure grid-world environments difference for reinforcement learning

I'd like to measure the difference between 2 grid-worlds to determine the generalization capacity of my agent using tabular Q-learning. Example (OpenAI Frozen Lake) : SFFF FHFH FFFH HFFG and : ...
1
vote
1answer
38 views

How to properly optimize shared network between actor and critic?

I'm building an actor-critic reinforcment learning algorithm to solve environments. I want to use a single encoder to find representation of my environment. When I share the encoder with the actor ...
1
vote
0answers
85 views

Difficulty in balancing Pendulum using Deep Reinforcement Learning Algorithm

I am using OpenAI Gym framework for reinforcement learning where I am trying solve classic control problem of balancing an Inverted Pendulum, which is similar to the "Pendulum-v0" with some changes in ...
1
vote
0answers
94 views

Policy gradient loss for neural network training

Say i want to train a neural network with 10 classes as outputs and use categorical_cross_entropy as a loss function in keras. This will try to fit the training ...
1
vote
0answers
22 views

Deciding the rewards for different actions in Pong for a DQN agent

I am attempting to implement an agent that learns to play in the Pong environment, the environment was created in PyGame and I return the pixel data and score at each frame. I use a CNN to take a ...
1
vote
0answers
96 views

Difficulty understanding Monte Carlo policy evaluation (state-value) for gridworld

I've been trying to read Sutton & Barto book chapter 5.1, but I'm still a bit confused about the procedure of using Monte Carlo policy evaluation (p.92), and now I just cant proceed anymore coding ...
1
vote
0answers
66 views

Actor-critic algorithm using gaussian Radial Basis Function, Local Linear Regression and shallow Neural Network

I'm attempting to implement the actor-critic algorithm on Matlab using Radial Basis Function, Local Linear Regression, and shallow Neural Network for inverted pendulum system. the state space and the ...
1
vote
0answers
52 views

Do we need to reset the DQN network after every episode?

I was going through this implementation of Reinforcement learning where model is being trained to manage the number of bikes at a station. Here, line 78 represents the loop over all episodes (if I ...
1
vote
0answers
55 views

Reinforcement Learning with limited number of episodes

I try to implement RL to a case something like this: This game consist of several rounds. Every round the players need to generate a maze that consists of rooms. There are around 1000 different ...
1
vote
2answers
67 views

Running 2 NEAT nets on the same observations

So i have been playing around with neat-python. I made a program, applying neat, to play pinball on the Atari 2600. The code for that can be found in the file ...
1
vote
0answers
32 views

Comparison and understanding of different version of DDQN?

There are several version of DDQN floating around. Sutton gives one that is a simple symmetric random update of the two Q functions. I think other papers (Silver paper for example) use a kind of ...
1
vote
0answers
28 views

Should we multiply the target of actor by the importance sampling ratio when prioritized replay is applied to DDPG?

According to PER, we have to multiply the $Q$ error $\delta_i$ by the importance sampling ratio to correct the bias introduced by the imbalance sampling of PER, where importance sampling ratio is ...
1
vote
0answers
82 views

How the actor use the output from the critic to make action in actor-critic network?

I am reading about the actor-critic architecture. I am confused about how the actor determines the action using the value (or future reward) from the critic network. Below you have the most popular ...
1
vote
1answer
145 views

Where are reinforcement algorithms used in financial services?

One of the most common misconceptions about reinforcement learning (RL) applications is that, once you deploy them, they continue to learn. And, usually, I'm left having to explain this. As part of my ...
1
vote
0answers
18 views

How to train chat bot on infinite non-stationary data?

I have continual simulated data of million sentences of two simulated persons talking to each other in a room and I want to model one of the persons speech. Now, during this period things in the room ...
1
vote
0answers
147 views

Input for the Env.step() in the 'Pendulum-v0' environment

I want to customize the 'Pendulum-v0' environment such that the action (the torque) from previous time step as well as from the current timestep serve as the inputs in the Env.step() function. My ...