Questions tagged [reinforcement-learning]

For questions related to learning controlled by external positive reinforcement or negative feedback signal or both, where learning and use of what has been thus far learned occur concurrently.

Filter by
Sorted by
Tagged with
2
votes
0answers
65 views

Infinite horizon in Reinforcement Learning

I read this article: "Towards Autonomous Data Ferry Route Design through Reinforcement Learning" by Daniel Henkel and Timothy X Brown. It specifies an infinite horizon problem where they use as a ...
4
votes
0answers
77 views

Catastrophic Forgetting on Pong Environment using DQN

I am running a basic DQN on the Pong environment. Not a CNN, just a 3 layer linear neural net with ReLUs. It seems to work for the most part, but at some point my model suffers from catastrophic ...
0
votes
0answers
37 views

How to learn to sample?

Imagine you have access to a dataset of pairs $(s, \hat{\pi}(s))$ where s is a state in a high dimension continuous space $S$, $\pi(s)$ is a probabilistic distribution on a large discrete space $D$ (...
0
votes
1answer
45 views

Understanding the configuration of replay memory and epsilon in deep reinforcement learning

I am tentatively reusing a codebase of pacman to train my own deep reinforcement learning model. While most of the components seems reasonable and understandable to me, there are two things that seem ...
2
votes
1answer
261 views

How to create a custom environment for reinforcement learning

I am a newbie in reinforcement learning working on a college project. The project is related to optimizing the hardware power. I am running proprietary software in Linux distribution (16.04). The goal ...
2
votes
2answers
62 views

How do we get the true value in the prediction objective in reinforcement learning?

In the book "Reinforcement Learning: An Introduction" (2018) Sutton and Barto define the prediction objective ($\overline{VE}$) as follows (page 199): $$\overline{VE}\doteq\sum_{s\epsilon S} \mu(s)[v_{...
1
vote
0answers
22 views

Training a reinforcement learning model with multiple images

I am tentatively trying to train a deep reinforcement learning model the maze escaping task, and each time it takes one image as the input (e.g., a different "maze"). Suppose I have about $10K$ ...
3
votes
1answer
76 views

beautify an image with reinforcement learning

I am trying to formulate and solve the following problem of image mutation. Suppose I am trying to insert an object image into a "background" image of several objects, and I will need to look for a "...
0
votes
1answer
48 views

OpenAI Gym interface when reward calculation is delayed? (continuous control with considerable reaction time)

I'm about to create an OpenAI Gym environment for a flight simulator. I'm wondering, how to cope with the fact, that the result and reward for some action needs a considerable time to advance through ...
2
votes
0answers
51 views

Reinforcement Learning in Real Life/Practical Terms

In every day life, it seems that we all have various habits and actions that we perform. For example, we wake up and check our email/facebook etc. on our phones. We don't look at are current state ...
1
vote
0answers
112 views

Implementation of PPO - Value Loss not converging, return plateauing

Copy from my reddit post: (Sorry if this does not fit here, please tell me and i delete it) Help regarding I'm working on an implementation of PPO, which i plan to use in my (Bachelors) Thesis. To ...
0
votes
0answers
49 views

A2C Critic Loss Interpretation

I'm working on an Advantage A2C implementation, and I just finished creating the value network $\hat{V_{\phi}}$. I train this network with the standard MSE loss of discounted rewards-to-go:$$\|\hat{V_{...
12
votes
1answer
323 views

When should I use Reinforcement Learning vs PID Control?

When designing solutions to problems such as the Lunar Lander on OpenAIGym, Reinforcement Learning is a tempting means of giving the agent adequate action control so as to successfully land. But ...
1
vote
1answer
78 views

The problem with the Gambler's Problem in RL

Recently I simulated the Gambler's Problem in RL: Now, the problem is, the curve does not at all appear the way as given in the book. The "best policy" curve appears a lot more undulating than it is ...
1
vote
0answers
42 views

Hashed Tile Coding vs Regular Tile Coding

In the book "Reinforcement Learning: An Introduction" (2018) Sutton and Barto explain at page 221 a form of tile coding using hashing, to reduce memory consumption. I have two questions about that: ...
1
vote
0answers
35 views

Action spaces for an RTS game

I think reinforcement learning would be a good fit for this problem, but I am not sure of how to deal with a seemingly infinite number of actions. In the beginning of each game (generic RTS game), the ...
1
vote
0answers
21 views

Learning utility function for AIS data

I am trying to learn utility functions for ships through their AIS data. I have a lot of data available and plan on focusing on fishing boats. So far I've researched a lot of IRL algorithms but I'm ...
5
votes
2answers
214 views

Why am I getting the incorrect value of lambda?

I am trying to solve for lambda using Temporal Difference Learning I am trying to figure out what lambda I need, to make TD(λ)=TD(1) but I get the incorrect value of lambda. Here is how I did: <...
3
votes
0answers
25 views

Can $\Phi$ measure of Integrated Information Theory serve as reward for reinforcement learning system?

Can $\Phi$ measure (computed rigorously or approximately) of Integrated Information Theory serve as reward for self-evolving/learning reinforcement learning system and hence we let this system to ...
1
vote
1answer
161 views

Adversarial Game playing using RL

I asked a question relating to TicTacToe playing in RL. From the answer it seems to me a lot is dependent on the opponent (rightly so, if we write down the Expectation Equations). My question is (in ...
6
votes
1answer
73 views

Purpose of Actor in Actor-Critic algorithm?

For discrete action spaces, what is the purpose of the actor in Actor-Critic algorithms? My current understanding is that the critic estimates the future reward given an action, so why not just take ...
3
votes
0answers
23 views

Humanoid agent reward shaping

I'm trying to teach a humanoid agent how to stand up after falling. The episode starts with the agent lying on the floor with its back touching the ground, and its goal is to stand up in the shortest ...
0
votes
0answers
40 views

DQN Algorithm not Converging for HVAC Control Task

I am trying to implement the DQN algorithm for the task of HVAC control. I have the algorithm implemented using Pytorch. I know that the HVAC simulator is working as it works for other control methods....
4
votes
1answer
96 views

Q-learning, am I interpreting correctly $Q(s,a) = r + \gamma \max_{a'} Q(s',a')$?

Ok, due to previous question I was pointed to use reinfrocement learning. So far what I understood from random websites is the following: there is a Q(s,a) function involved I can assume my neural ...
0
votes
0answers
15 views

Problems while training a DQN Agent on DSTC dataset

I am trying to create a dialogue policy model on DSTC data. This model takes in a state of the conversation and outputs an act the machine must take. I am creating this model using reinforcement ...
1
vote
0answers
22 views

Why should we use TD prediction as opposed to TD control algorithms?

Consider a problem where we have a finite number of states and actions. Given a state and an action, we can easily know the reward and the next state (deterministic). The state space is really large ...
0
votes
0answers
29 views

Are there real-world problems where case-based reasoning is not suitable?

I know case-based reasoning has four stages: retrieve, retain, re-use and revise. Used for solving new problems by adapting solutions that were used to solve old problems, like car issues. The ...
2
votes
1answer
63 views

What is the relation between Monte Carlo and model-free algorithms?

Monte Carlo (MC) methods are methods that use some form of randomness or sampling. For example, we can use an MC method to approximate the area of a circle inside a square: we generate random 2D ...
1
vote
0answers
28 views

High variance in performance of q-learning agents trained with same parameters

I am training an agent to play a simple game using double deep q learning. However, the variance in agent performance is very high, even for agents trained with same model parameters. For example, I ...
2
votes
0answers
20 views

Are there reinforcement learning algorithms that ensure convergence for continuous state space problems?

The Q-learning does not guarantee convergence for continuous state space problems (Why doesn't Q-learning converge when using function approximation?). In that case, is there an algorithm which ...
-1
votes
1answer
118 views

What is the difference between a stochastic and a deterministic policy?

In reinforcement learning, there are the concepts of stochastic (or probabilistic) and deterministic policies. What is the difference between them?
3
votes
1answer
127 views

What is Policy Iteration in RL?

Consider the Gridworld problem in RL. Formally, policy in RL is defined as $\pi(a|s)$. If we are solving Gridworld by Policy Iteration then the following pseudocode is used: Now the question is in ...
1
vote
1answer
48 views

Static or dynamic learning rate (Q-learning)

I have the following code (below), where an agent uses Q-learning (RL) to play a simple game. What appears to be questionable for me in that code is the fixed learning rate. When it's set low, it's ...
4
votes
2answers
125 views

Deciding on a reward per each action in a given state (Q-learning)

I looked for existing posts on Stack Exchange, which kind of answer the questions about the reward system and reward function, but not specifically what I want to ask here, which is how do you ...
2
votes
1answer
31 views

Training actor-critic algorithms in games with opponents

I am wondering how am I supposed to train a model using actor/critic algorithms in environments with opponents. I tried the followings (using A3C and DDPG): Play against random player. I had rather ...
5
votes
2answers
161 views

Is Q-Learning suitable for continous (state or action) spaces?

Many examples work with a table based method for Q-Llearning. This may be suitable for discrete state(observation) or actions like a robot in a grid world but is there a way to use Q-Learning for ...
2
votes
1answer
50 views

Possible inconsistency in the Policy Improvement equation

I came across this formula in Sutton And Barto: RL an Intro (2nd Edition) equation number 4.7 (page number 78). If $\pi$ and $\pi'$ are deterministic policies and $q_\pi(s, \pi'(s)) \geq v_\pi(s)$ ...
0
votes
1answer
71 views

What is the reward system of reinforcement learning?

Can you describe this system in more detail? I understand that the environment sends a signal indicating whether or not the action taken by the agent was 'good' or not, but it seems too simple. ...
0
votes
0answers
85 views

Details of implementing an LSTM in Reinforcement Learning

I'm currently looking into the context of adding an LSTM to my PPO pytorch implementation. My plan is to add one LSTM layer right after the last convolutional layer. I'm wondering now whether it is ...
0
votes
0answers
32 views

How do I define the reward function in the case of self-driving raspberry pi car?

I am working on a self driving car powered by a raspberry pi. My first step is to use PPO to teach it to not run into walls. But I am having trouble getting it to work. I want to allow the car to ...
0
votes
1answer
121 views

Reinforcement learning with PPO: rewards decreasing

I'm trying to train a PPO agent and my average rewards graph looks like this. Could this indicate that it's stuck at a local maximum? Do I need to promote exploring by increasing the entropy or does ...
1
vote
3answers
129 views

How can I develop a prediction algorithm for a game of chance?

How can I develop a prediction algorithm in the case of games of chance? Suppose there is a 50:50 chance of winning. Is there way of creating a prediction algorithm?
1
vote
0answers
32 views

DQN not able to learn in a game where other agents perform random walks

I am making a school project where I should develop any kind of game where I can have one reactive agent and one agent based on machine learning competing with each other. My game consists of a ...
1
vote
0answers
14 views

DQN ANN input vs Linear function approximator feature vector

So when using semi-gradient td(0) you need to convert your state representation into a feature vector that represents the state and as far as I know, should not be correlated. Is the input on the ANN ...
1
vote
1answer
31 views

DQN Agent helped by a controller: on which action should I perform backprop?

Background I am working on a robotic arm controlled by a DQN + a python script I wrote. The DQN receives the 5 joint states, the coordinates of a target, the coordinates of the obstacle and outputs ...
0
votes
1answer
298 views

State aggregation methods

In Sutton's RL:An introduction 2nd edition it says the following(page 203): State aggregation is a simple form of generalizing function approximation in which states are grouped together, with ...
2
votes
1answer
102 views

Can reinforcement learning be used for tasks where only one final reward is received?

Is reinforcement learning problem adaptable to the setting when there is only one - final - reward. I am aware about problems with sparse and delayed rewards, but what about the only one reward and ...
0
votes
1answer
94 views

How fast does Monte Carlo tree search converge?

How fast does Monte Carlo Tree Search converge? Is there proof that it does converge? How does it compare to Temporal Difference learning in terms of convergence speed (assuming the evaluation step ...
1
vote
1answer
71 views

A3C fails to solve MountainCar-v0 enviroment (implementation by OpenAi gym)

While I've been able to solve MountainCar-v0 using Deep Q learning, no matter what I try I can't solve this enviroment using policy-gradient approaches. As far as I learnt searching the web, this is a ...
2
votes
0answers
10 views

How is GARB implemented in PGRD-DL to calculate gradients w.r.t. internal rewards?

In section 3 of this paper the author outlines how GARB was adapted to reduce the variance in updating parameters to an internal reward function estimator. I have read it a number of times and ...