Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

604 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
1
vote
0answers
71 views

What happens if our target network overestimates the value?

When we use DDQN, we often use the target network in case our online network overestimates a value, but this doesn't make sense to me, because What happens if our target network is the one that ...
1
vote
0answers
57 views

What is the best way to make a deep reinforcement learning environment with a continuous 2D action space?

I understand that the actor-critic method is probably where I want to start because of how it works with continuous action spaces. However, the problem I am trying to solve would require the action be ...
1
vote
0answers
76 views

What is the advantage of using experience replay (as opposed to feeding it sequential data)?

Let's suppose that our RL agent needs to play a game with different levels. If we train our RL agent sequentially or with sequential data, our agent will learn how to play level 1, but then it will ...
1
vote
0answers
44 views

How can I build a deep reinforcement learning model that can be trained with multiple time series datasets

I built a DRL model to trade stocks in the financial market but the number of observations is relatively small and I would like to increase it by training the same model with stocks from several ...
1
vote
0answers
57 views

When to apply reward for time series data?

Reading the paper 'Reinforcement Learning for FX trading 'at https://stanford.edu/class/msande448/2019/Final_reports/gr2.pdf it states: While our end goal is to be able to make decisions on a ...
1
vote
0answers
40 views

What is a multi channel supervised classifier?

I came across a paper that describes its model architecture in the following way. Our TRIL network is a two-channel network jointly trained to predict the expert’s action given state and the system’s ...
1
vote
0answers
111 views

Classification or regression for deep Q learning

DQN implemented at https://github.com/PacktPublishing/PyTorch-1.x-Reinforcement-Learning-Cookbook/blob/master/Chapter07/chapter7/dqn.py uses the mean square error loss function for the neural network ...
1
vote
0answers
63 views

In continuous action spaces, how is the standard deviation, associated with Gaussian distribution from which actions are sampled, represented?

I have a question about implementing policy gradient methods for problems with continuous action spaces. Assume that actions are sampled from a diagonal Gaussian distribution with mean vector $\mu$ ...
1
vote
0answers
39 views

How can I convert a simple CLI RPG to a compatible environment for training an RL agent via stable-baselines?

What would be the good choice of algorithm to use for character action selection in an RPG, implemented in Python? I had previously asked this question in the hopes of getting headway on the AI ...
1
vote
0answers
48 views

Correct dimensionality of parameter vector for solving an MRP with linear function approximation?

I'm in the process of trying to learn more about RL by shadowing a course offered collaboratively by UCL and DeepMind that has been made available to the public. I'm most of the way through the course,...
1
vote
0answers
49 views

Solving multi-armed bandit problems with continuous action space

My problem has a single state and an infinite amount of actions on a certain interval (0,1). After quite some time of googling I found a few paper about an algorithm called zooming algorithm which can ...
1
vote
0answers
25 views

Should the importance sampling ratio be updated at the end of the for loop in the off-policy Monte Carlo control algorithm?

I'm studying RL with Sutton and Barto's book. I'd like to ask about the order of execution of a statement in the algorithm below. Here, $W$ (importance sampling ratio) is updated at the end of the <...
1
vote
1answer
68 views

Do we need multiple parallel environments to train in batches an on-policy algorithm?

When using an on-policy method in reinforcement learning, like advantage actor-critic, you shouldn't use old data from an experience buffer, since a new policy requires new data. Does this mean that ...
1
vote
0answers
72 views

Tic-tac-toe: How would standard SARSA and Q-learning yield different results in the agent's behaviour?

I know this is deceptively simple. Tic tac toe is a well studied game for RL. Assume your agent is playing aggainst a strong opponent. I know you deal in after states. I know that in Q learning the ...
1
vote
0answers
29 views

How are the coefficients of the Region of Interest being selected?

I was reading the following paper: Rl-Ncs: Reinforcement Learning Based Data-Driven Approach For Nonuniform Compressed Sensing, and my question is: how do they decide whether a signal is characterized ...
1
vote
0answers
91 views

Atari Games: Pretrained CNN to accelerate training?

DQN for Atari takes considerable training time. For example, the 2015 paper in Nature notes that algorithms are trained for 50 million frames or equivalently around 38 days of game experience in total....
1
vote
0answers
38 views

Does importance sampling for off-policy estimation also apply to the case of negative rewards?

Importance sampling is a common method for calculating off-policy estimates in RL. I have been reading through some of the original documentation (D.G. Horvitz and D.J. Thompson, Powell, M.J. and ...
1
vote
0answers
28 views

What is the proof that the variance of the gradient estimate in Actor-Critic is smaller than in REINFORCE?

The intuition provided when introducing actor-critic algorithms is that the variance of its gradient estimates is smaller than in REINFORCE as, e.g., discussed here. This intuition makes sense for the ...
1
vote
0answers
33 views

Why do bootstrapping methods produce nonstationary targets more than non-bootstrapping methods?

The following quote is taken from the beginning of the chapter on "Approximate Solution Methods" (p. 198) in "Reinforcement Learning" by Sutton & Barto (2018): reinforcement ...
1
vote
1answer
38 views

How to deal with the addition of a new state to the environment during training?

Let's say we have a dynamic environment: a new state gets added after 2000 episodes have been done. So, we leave room for exploration, so that it can discover the new state. When it gets to that new ...
1
vote
0answers
37 views

How to choose an RL algorithm for a Gridworld that models a much more complex problem

I am considering using Reinforcement Learning to do optimal control of a complex process that is controlled by two parameters $(n_O, n_I), \quad n_I = 1,2,3,\dots, M_I, n_O = 1,2,3,\dots, M_O$ In this ...
1
vote
0answers
27 views

Are policy-based methods better than value-based methods only for large action spaces?

In different books on reinforcement learning, policy-based methods are motivated by their ability to handle large (continuous) action spaces. Is this the only motivation for the policy-based methods? ...
1
vote
1answer
77 views

Continuous state and continuous action Markov decision process time complexity estimate: backward induction VS policy gradient method (RL)

Model Description: Model based(assume known of the entire model) Markov decision process. Time($t$): Finite horizon discrete time with discounting factor State($x_t$): Continuous multi-dimensional ...
1
vote
0answers
53 views

Is Reinforcement Learning what I need for this image to image translation problem?

I have a paired dataset of binary images A and B: A1 paired with B1, A2-B2, etc., with simple shapes (rectangles, squares). The external software receives both images A and B and it returns a number ...
1
vote
0answers
37 views

Is there a way to show convergence of DQN other than by eye observation?

I made a DQN model and plot its reward curve. You can see intuitively that the curve already converged since its reward value now just oscillates. How can I show confidence that my DQN already reached ...
1
vote
0answers
46 views

Is this figure a correct representation of off-policy actor-critic methods?

Does this figure correctly represent the overall general idea about actor-critic methods for on-policy (left) and off-policy (right) case? I am a bit confused about the off-policy case (right figure). ...
1
vote
0answers
70 views

Does the off-policy evaluation work for non-stationary policies?

As the title says, in reinforcement learning, does the off-policy evaluation work for non-stationary policies? For example, IS (importance sampling)-based estimators, such as weighted IS or doubly ...
1
vote
0answers
51 views

In layman's terms, what is stochastic computation graph?

I'm going through the distributions package on PyTorch's documentation and came across the term stochastic computation graph. In layman's terms, what is it?
1
vote
0answers
44 views

What would be the good choice of algorithm to use for character action selection in an RPG, implemented in Python?

I have developed an RPG in likeness to the features showcased in the Final Fantasy series; multiple character classes which utilise unique action sets, sequential turn-based combat, front/back row ...
1
vote
0answers
44 views

Designing a reward function for my reinforcement learning problem

I'm working on a project lately and I'm trying to solve a problem with reinforcement learning and I have serious issues with shaping the reward function. The problem is designing a device with maximum ...
1
vote
0answers
54 views

Can reinforcement learning algorithms be applied on problems involving a very large number of possible actions?

There is a question already about applying RL to "large scale problems", where large scale refers to the problem of a relatively small number of actions (that could be from a continous space) ...
1
vote
0answers
50 views

Why does the n-step return being zero result in high variance in off policy n-step TD?

In the paragraph given between eq 7.12 and 7.13 in Sutton & Barto's book: $G_{t:h} = R_{t+1} + G_{t+1:h} , t < h < T$ where $G_{h:h} = V_{h-1}(S_h)$. (Recall that this return is used at ...
1
vote
1answer
134 views

How to implement RAM versions of Atari games

I have coded the breakout RAM version, but, unfortunately, its highest reward was 5. I trained it for about 2 hours and never reached a higher score. The code is huge, so I can't paste here, but, in ...
1
vote
0answers
56 views

Calculating the advantage 'gain' of actions in model-free reinforcement learning

I have a simple question about model-free reinforcement. In a model I'm writing about, I want to know the value 'gain' we'd get for executing an action, relative to the current state. That is, what ...
1
vote
0answers
29 views

How to learn how to select a subgraph via reinforcement learning?

I have the following problem. I am given a graph with a lot (>30000) nodes. Nodes are associated with a low (<10)-dimensional feature vector, and edges are associated with a low (<10)-...
1
vote
0answers
71 views

Actor-Critic implementation not learning

I've implemented a vanilla actor-critic and have run into a wall. My model does not seem to be learning the optimal policy. The red graph below shows its performance in cartpole, where the algorithm ...
1
vote
0answers
90 views

Understanding the role of the target network in this DQN algorithm

I've found online this interesting algorithm: From what I understand reading this algorithm, I can't figure out why I should "perform the opposite action" and consequently storing that second ...
1
vote
0answers
51 views

Using a model-based method to build an accurate day trading environment model

There are several different angles we can classify Reinforcement Learning methods from. We can distinguish three main aspects : Value-based and policy-based On-policy and off-policy Model-free and ...
1
vote
0answers
63 views

Policy Gradient on Tic-Tac-Toe not working

I wanted to implement the Policy Gradient on Tic-Tac-Toe. I tried to use the code that worked for any environment like CartPole-v0 to my Tic-Tac-To game. But it is not learning. There are no errors. ...
1
vote
0answers
35 views

How can I increase the exploration in the Proximal Policy Optimation algorithm?

How can I increase the exploration in the Proximal Policy Optimation reinforcement learning algorithm? Is there a variable assigned for this purpose? I'm using the stable-baseline implementation: ...
1
vote
0answers
60 views

How do I set up rewards to account for unmanned aerial vehicle crashes?

I am working on a project to implement a collision avoidance algorithm on a real unmanned aerial vehicle (UAV). I'm interested in understanding the process to set up a negative reward to account for ...
1
vote
1answer
67 views

How to best make use of learning rate scheduling in reinforcement learning?

How to best make use of learning rate scheduling in reinforcement learning? To me, a low learning rate towards the end to fine-tune what you've learned with subtle updates makes sense. But I don't ...
1
vote
0answers
44 views

DQN not showing the agent is learning in a snake grid environment game

I've been trying to train a snake for the snake game in DQN. Which the snake can essentially just move up, down, left and right. I'm having a hard time getting the snake to stay alive longer. So my ...
1
vote
0answers
52 views

Benchmarking SAC on Pybullet

So far I have seen TD3 and DDPG benchmarks on Pybullet environments, but I am looking for SAC benchmarks on Pybullet too, anyone can help?
1
vote
1answer
79 views

Do smaller loss values during DQN training produce better policies?

During the training of DQN, I noticed that the model with prioritized experience replay (PER) had a smaller loss in general compared to a DQN without PER. The mean squared loss was an order of ...
1
vote
0answers
62 views

Is the reward following after time step $t+1$ collected based on current policy?

I am currently learning policy gradient methods from the Deep RL boot camp by Pieter Abbeel in which he explains the actor-critic algorithm derivation. At around minute 39, he explains that the sum ...
1
vote
0answers
24 views

Trying to Train Cards Game with RL

I am trying to train a card game called Callbreak I tried inputs like all the opponents discarded cards all hands everything a human can see and calculate with "common sense" I fed it to the Agent but ...
1
vote
0answers
85 views

How should I deal with variable batch size in A3C?

I am fairly new to reinforcement learning (RL) and deep RL. I have been trying to create my first agent (using A3C) that selects an optimal path with the reward being some associated completion time (...
1
vote
1answer
76 views

Q table not converging for an arbitrary experiment

This is an experiment in order to understand the working of Q table and Q learning. I have the states as states = [0,1,2,3] I have an arbitrary value for each ...
1
vote
0answers
38 views

State-of-the-art algorithms not working on a custom RL environment

I'm trying to train a RL agent on a custom, highly stochastic environment (MDP). In order to do so I'm using existing implementations of state-of-the-art RL algorithms as provided by Stable Baselines. ...

1
5 6
7
8 9
13