What's the deal with Deno? We talk with a major contributor to find out. Listen now.

Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

Filter by
Sorted by
Tagged with
0
votes
1answer
38 views

Should I build an environment from scratch myself or it is not always needed?

I am inspired by the paper Neural Architecture Search with Reinforcement Learning to use reinforcement learning for optimizing a child network (learner). My meta-learner (controller or parent network) ...
1
vote
1answer
31 views

How is trajectory sampling different than normal (importance) sampling in reinforcement learning?

I am using Sutton and Barto's book for Reinforcement Learning. In Chapter 8, I am having difficulty in understanding the Trajectory Sampling. I have read the particular section on trajectory sampling (...
2
votes
1answer
62 views

Implementing SARSA for a 2-stage Markov Decision Process

I am a bit confused as to how exactly I should be implementing SARSA (or Q-learning too) on what is a simple 2-stage Markov Decision Task. The structure of the task is as follows: Basically, there ...
3
votes
1answer
84 views

Has reinforcement learning been used to prove mathematical theorems?

Coq exists, and there are other similar projects out there. Further, Reinforcement Learning has made splashes in the domain of playing games (a la Deepmind & OpenAI and other less well-known ...
2
votes
1answer
41 views

Should illegal moves be excluded from loss calculation in DQN algorithm?

I'm implementing DQN algorithm to train my agent to play a turn-based game. The action space for the game is small, but not all moves are available at all the states. Therefore, when deciding on which ...
1
vote
0answers
26 views

Why do bootstrapping methods produce nonstationary targets more than non-bootstrapping methods?

The following quote is taken from the beginning of the chapter on "Approximate Solution Methods" (p. 198) in "Reinforcement Learning" by Sutton & Barto (2018): reinforcement ...
2
votes
0answers
27 views

Why is it the case that off - policy evaluation using importance sampling suffers from high variance?

The average return for trajectories, $V^{\pi_e}$(s) is often computed via the importance sampling estimate $$V^{\pi_e}(s) = \frac{1}{n}\sum_{i=1}^n\prod_{t=0}^{H}\frac{\pi_e(a_t | s_t)}{\pi_b(a_t|s_t)}...
2
votes
1answer
73 views

q learning appears to converge but does not always win against random tic tac toe player

q learning is defined as: Here is my implementation of q learning of the tic tac toe problem: ...
2
votes
1answer
44 views

Why isn't it wise for us to completely erase our old Q value and replace it with the calculated Q value?

Why isn't it wise for us to completely erase our old Q value and replace it with the calculated Q value? Why can't we forget the learning rate and temporal difference? Here's the update formula.
3
votes
2answers
91 views

Why is it not advisable to have a 100 percent exploration rate? [duplicate]

During the learning phase, why don't we have a 100% exploration rate, to allow our agent to fully explore our environment and update the Q values, then during testing we bring in exploitation? Does ...
0
votes
1answer
44 views

Why do we update the weights of the target network in deep Q learning?

I know we keep the target network constant during training to improve stability, but why exactly are we updating the weights of our target network? In particular, if we've already reached convergence, ...
0
votes
2answers
48 views

Does “transition model” alone in an MDP imply it's non-deterministic?

I am looking at a lecture on POMDP, and the context is that, when the quadcopter can't see the landmarks, it has to use reckoning. And then he mentions the transition model is not deterministic, hence ...
1
vote
1answer
35 views

How to deal with the addition of a new state to the environment during training?

Let's say we have a dynamic environment: a new state gets added after 2000 episodes have been done. So, we leave room for exploration, so that it can discover the new state. When it gets to that new ...
0
votes
1answer
43 views

Is there a good website where I can learn about Deep Deterministic Policy Gradient?

Is there a good website where I can learn about Deep Deterministic Policy Gradient?
0
votes
0answers
29 views

How is centralised training and decentralised execution in multi agent reinforcement learning implemented?

In the paper Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learning, it is written We allow centralised training but require decentralised execution, from which follows that the policies ...
1
vote
2answers
57 views

Why do we explore after we have an accurate estimate of the value function?

Suppose we have a small space state and that, after about 2000 episodes, we've accurately explored the environment and known the accurate $Q$ values. In that case, why do we still leave a small ...
3
votes
1answer
49 views

How to handle the final state in experience replay?

I'm using the DQN algorithm to train my agent to play a turn-based game. The memory replay buffer stores tuples of experiences $(s, a, r, s')$, where $s$ and $s'$ are consecutive states. At the last ...
4
votes
1answer
85 views

What happens when you select actions using softmax instead of epsilon greedy in DQN?

I understand the two major branches of RL are Q-Learning and Policy Gradient methods. From my understanding (correct me if I'm wrong), policy gradient methods have an inherent exploration built-in as ...
2
votes
1answer
63 views

What is the bias-variance trade-off in reinforcement learning?

I am watching DeepMind's video lecture series on reinforcement learning, and when I was watching the video of model-free RL, the instructor said the Monte Carlo methods have less bias than temporal-...
2
votes
1answer
55 views

How to handle changing goals in a DQN?

I created a virtual 2D environment where an agent aims to find a correct pose corresponding to a target image. I implemented a DQN to solve this task. When the goal is fixed, e.g. the aim is to find ...
1
vote
0answers
34 views

How to choose an RL algorithm for a Gridworld that models a much more complex problem

I am considering using Reinforcement Learning to do optimal control of a complex process that is controlled by two parameters $(n_O, n_I), \quad n_I = 1,2,3,\dots, M_I, n_O = 1,2,3,\dots, M_O$ In this ...
0
votes
0answers
20 views

Optimal RL agent's representation of 3D-grid data: 2D Slices and CNN encoding. Suggestions?

environment My agent needs to navigate in a 64x64x64 discrete 3d grid environment, and remove certain voxels. Voxels can be in a number of states: should-remove, should-not-remove, empty. It can move ...
2
votes
2answers
42 views

Understanding the “unroling” step in the proof of the policy gradient theorem

In the proof of the policy gradient theorem in the RL book of Sutton and Barto (that I shamelessly paste here): there is the "unrolling" step that is supposed to be immediately clear With ...
1
vote
0answers
25 views

Are policy-based methods better than value-based methods only for large action spaces?

In different books on reinforcement learning, policy-based methods are motivated by their ability to handle large (continuous) action spaces. Is this the only motivation for the policy-based methods? ...
0
votes
0answers
31 views

Reinforcement Learning Diagnostic: Total reward doesn't converge

I'm implementing DDQN in my toy scenario. During training, I'm surprised to see that the total reward doesn't converge and have a tendency to degrade. What could be the problem? Here's the picture: ...
1
vote
1answer
52 views

Continuous state and continuous action Markov decision process time complexity estimate: backward induction VS policy gradient method (RL)

Model Description: Model based(assume known of the entire model) Markov decision process. Time($t$): Finite horizon discrete time with discounting factor State($x_t$): Continuous multi-dimensional ...
1
vote
1answer
35 views

What would happen if we sampled only one tuple from the experience replay?

The concept of experience replay is saving our experiences in our replay buffer. We select at random to break the correlation between consecutive samples, right? What would happen if we calculate our ...
1
vote
0answers
51 views

Is Reinforcement Learning what I need for this image to image translation problem?

I have a paired dataset of binary images A and B: A1 paired with B1, A2-B2, etc., with simple shapes (rectangles, squares). The external software receives both images A and B and it returns a number ...
1
vote
1answer
95 views

What's the best practice for Boltzmann Exploration temperature in RL?

I'm currently modeling DQN in Reinforcement Learning. My question is: what are the best practices related to Boltzmann Exploration? My current thoughts are: (1) Let the temperature decay through ...
0
votes
1answer
40 views

How to validate that my DQN hyperparameters are the optimal?

My DQN model outputs the best traffic light state in an intersection. I used different values of batch size and learning rate to find the best model. How would I know if I got the optimal ...
1
vote
0answers
31 views

Is there a way to show convergence of DQN other than by eye observation?

I made a DQN model and plot its reward curve. You can see intuitively that the curve already converged since its reward value now just oscillates. How can I show confidence that my DQN already reached ...
0
votes
0answers
47 views

Why care about the value of the action which I'm not gonna take in policy iteration?

In this article, there is an explanation (with an example) of how policy iteration works. It seems that, if we replace all the probabilities of moves in the example by new probabilities where the best ...
0
votes
1answer
71 views

How do I test an LSTM-based reinforcement learning model using any Atari games in OpenAI gym?

I am writing a couple of different reinforcement learning models based on Rainbow DQN or some PG models. All of them internally use an LSTM network because my project is using time series data. I ...
1
vote
0answers
35 views

Is this figure a correct representation of off-policy actor-critic methods?

Does this figure correctly represent the overall general idea about actor-critic methods for on-policy (left) and off-policy (right) case? I am a bit confused about the off-policy case (right figure). ...
0
votes
1answer
50 views

In Deep Deterministic Policy Gradient, are all weights of the policy network updated with the same or different value?

I'm trying to understand the DDPG algorithm shown at this page. I don't know what should the result of the gradient at step 14 be. Is it a scalar that I have to use to update all the weights (so all ...
2
votes
1answer
103 views

Updating action-value functions in Semi-Markov Decision Process and Reinforcement Learning

Suppose that the transition time between two states is a random variable (for example, unknown exponential distribution); and between two arrivals, there is no reward. If $\tau$ (real number not an ...
0
votes
0answers
24 views

How to deal with GAE ineffectiveness because of critic value adaptation?

I've noticed if you have a small negative reward (e.g.,-0.01) per step for idling and a series of idle steps, an agent seems to learn to trick GAE by learning a ...
0
votes
0answers
45 views

Does the off-policy evaluation work for non-stationary policies?

As the title says, in reinforcement learning, does the off-policy evaluation work for non-stationary policies? For example, IS (importance sampling)-based estimators, such as weighted IS or doubly ...
1
vote
0answers
40 views

In layman's terms, what is stochastic computation graph?

I'm going through the distributions package on PyTorch's documentation and came across the term stochastic computation graph. In layman's terms, what is it?
2
votes
1answer
66 views

Two DQNs in two different time scales

I have the following situation. An agent plays a game and wants to maximize the accumulated reward as usual, but it can choose its adversary. There are $n$ adversaries. In episode $e$, the agent must ...
0
votes
0answers
29 views

Patient PPO: how to handle imbalanced discrete action space?

PPO agent. The action space includes 3 actions: 0: do nothing 1: act (start) 2: stop The agent has to perform thousands of steps doing nothing, then perform step 1 only once (act), then do nothing ...
1
vote
1answer
68 views

VC Dimension of Reinforcement Learning (RL)

Is the VC dimension meaningful for the reinforcement learning (RL) as a machine learning (ML) method? How?
3
votes
2answers
110 views

What introductory books to reinforcement learning do you know, and how do they approach this topic?

Currently, I'm only going through these two books Reinforcement Learning: An Introduction, by Sutton and Barto: RL explained on an engineering level (mathematical, but readable for a non-...
0
votes
1answer
40 views

Looping over Sarsa algorithm for better Q values

Let's say an RL trading system places trades based on pricing data. Each episode represents 1 hour of trading, and there are 24 hours of data available. The Q table represents for a given state, what ...
6
votes
1answer
228 views

How to measure sample efficiency of a reinforcement learning algorithm?

I want to know if there is any metric to use for measuring sample-efficiency of a reinforcement learning algorithm? From reading research papers, I see claims that proposed models are more sample ...
1
vote
1answer
73 views

In Deep Q-learning, are the target update frequency and the batch training frequency related?

In a Deep Q-learning algorithm, we perform a batch training every train_freq and we update the parameters of the target network every ...
1
vote
0answers
40 views

What would be the good choice of algorithm to use for character action selection in an RPG, implemented in Python?

I have developed an RPG in likeness to the features showcased in the Final Fantasy series; multiple character classes which utilise unique action sets, sequential turn-based combat, front/back row ...
1
vote
0answers
35 views

Designing a reward function for my reinforcement learning problem

I'm working on a project lately and I'm trying to solve a problem with reinforcement learning and I have serious issues with shaping the reward function. The problem is designing a device with maximum ...
1
vote
1answer
44 views

Connection between the Bellman equation for the action value function $q_\pi(s,a)$ and expressing $q_\pi(s,a) = q_\pi(s, a,v_\pi(s'))$

When deriving the Bellman equation for $q_\pi(s,a)$, we have $q_\pi(s,a) = E_\pi[G_t | S_t = s, A_t = a] = E_\pi[R_{t+1} + \gamma G_{t+1} | S_t = s, A_t = a]$ (1) This is what is confusing me, at this ...
2
votes
1answer
39 views

How to express $v_\pi(s)$ in terms of $q_\pi(s,a)$?

This is the exercise 3.18 in Sutton and Barto's book. The task is to express $v_\pi(s)$ using $q_\pi(s,a)$. Looking at the diagram above, the value of $q_\pi(s,a)$ at $s$ for each $a \in A$ we take ...

1
3 4
5
6 7
28