Questions tagged [reinforcement-learning]
For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.
1,617
questions
0
votes
0answers
3 views
Where does the hierarchical reinforcement learning framework name “MAXQ” come from?
I've been researching different frameworks for hierarchical RL (mainly options, HAMs, and MAXQ) and noticed that both options and HAMs have names that relate to how they function. I can't seem to find ...
2
votes
1answer
29 views
Intuition behind $1-\gamma$ and $\frac{1}{1-\gamma}$ for calculating discounted future state distribution and discounted reward
In the appendix of the Constrained Policy Optimization (CPO) paper (Arxiv), the authors denote the discounted future state distribution $d^\pi$ as:
$$d^\pi(s) = (1-\gamma) \sum_{t=0}^\infty{\gamma^t P(...
1
vote
0answers
34 views
What's the best way to take a list of lists as DQN input?
I have my own environment for the DQN algorithm. In my environment, the state space is represented by a list of lists, where each sublist can be of different lengths. In my case, the length of the ...
-2
votes
0answers
32 views
Can reinforcement learning and evolutionary algorithms be the same? [closed]
https://ai.googleblog.com/2018/03/using-evolutionary-automl-to-discover.html
As shown in the table, evolutionary algorithms are superior.
Question 1.
If reinforcement learning algorithms are rewarded ...
2
votes
0answers
17 views
Are there any known disadvantages of implementing vanilla Q-learning on a discretized-state space environment?
For an RL problem on a continuous state space, the states could be discretized into buckets and these buckets used in implementing the Q-table. I see that is what is done here. However, according to ...
1
vote
1answer
33 views
What is a good convergence criterion for Q-learning in a stochastic environment?
I have a stochastic environment and I'm implementing a Q-table for the learning that happens on the environment. The code is shown below. In short, there are ten states (0, 1, 2,...,9), and three ...
2
votes
1answer
59 views
Does stochasticity of an environment necessarily mean non-stationarity in MDPs?
Is a stochastic environment necessarily also non-stationary? To elaborate, consider a two-state environment ($s_1$ and $s_2$), with two actions $a_1$ and $a_2$. In $s_1$, taking action $a_1$ has a ...
0
votes
0answers
23 views
Control Strategy for Multi-Agent Systems
I am working on a smart grid system which can be modeled with multiple agents interacting with each other.
The agents are physically coupled with non-linearities, the action of one agent has a direct ...
1
vote
0answers
42 views
Which RL algorithm would be suitable for this multi-dimensional and continuous action space?
Is there an RL approach/algorithm that would be suited for the following kind of problem?
There is a continuous action space with an action value $A_{a,t}$ for each action dimension $a$.
The ...
1
vote
2answers
60 views
What is the purpose of storing the action $a$ within an experience tuple?
From what I understand, experience replay works by storing tuples of $(s, a, r, s')$ to be sampled for training. I understand why we store $s$, $r$ and $s'$. However, I do not understand the need for ...
2
votes
1answer
59 views
How would I compute the optimal state-action value for a certain state and action?
I am currently trying to learn reinforcement learning and I started with the basic gridworld application. I tried Q-learning with the following parameters:
Learning rate = 0.1
Discount factor = 0.95
...
3
votes
1answer
44 views
Is there a document with a list of conjectures or research problems regarding reinforcement learning (like the Millennium Prize Problems)?
Is there a document with a list of conjectures or research problems regarding reinforcement learning like the Millennium Prize Problems?
4
votes
1answer
161 views
Why does Q-learning converge under 100% exploration rate?
I am working on this assignment where I made the agent learn state-action values (Q-values) with Q-learning and 100% exploration rate. The environment is the classic gridworld as shown in the ...
1
vote
0answers
42 views
Why is Openai's PPO2 implementation differentiable?
I'm trying to understand the concept behind the implementation of the OpenAI PPO2 algorithm. The loss function that is minimized is as follows: ...
1
vote
0answers
41 views
How should I simulate this Markov Decision Process?
I am working on solving a problem on nodes in a graph communicating with each other. They try to estimate a central state using Kalman consensus filter, with the connections described by the graph's ...
1
vote
0answers
25 views
PPO2: Intuition behind Gumbel Softmax and Exploration?
I'm trying to understand the logic behind the magic of using the gumbel distribution for action sampling inside the PPO2 algorithm.
This code snippet implements the action sampling, taken from here:
<...
0
votes
0answers
16 views
How DynaQ behaves in stochastic world in comparison with other reinforcement learning algorithms?
I came across of implementations of a bunch of algorithms on stochastic windy gridworld. This is the graph comparing their performance:
So clearly, it seems that DynaQ performs better than all other ...
2
votes
1answer
36 views
Clarifying representation of Neural Nerwork input for Chess Alpha Zero
In the Alpha Zero paper (https://arxiv.org/pdf/1712.01815.pdf) page 13, the input for the NN is described. In the beggining of the page, the authors state that:
"The input to the Neural Network ...
0
votes
1answer
44 views
Can I train a DQN on the same dataset for multiple epochs?
I am trying to learn about reinforcement learning and chose the stock market to experiment with. I have minute by minute historical data on a particular stock for the past 20 years. I am using a ...
1
vote
0answers
20 views
Is (log-)standard deviation learned in TRPO and PPO or fixed instead?
After having read Williams (1992), where it was suggested that actually both the mean and standard deviation can be learned while training a REINFORCE algorithm on generating continuous output values, ...
1
vote
0answers
30 views
Confusion about computing policy gradient with automatic differentiation ( material from Berkeley CS285)
I am taking Berkeleyās CS285 via self-study. On this particular lecture regarding Policy Gradient, I am very confused about the inconsistency between the concept explanation and the demonstration of ...
4
votes
1answer
1k views
Is there a machine learning model that can be trained with labels that only say how “right” or “wrong” it was?
I'm trying to find the name for a model that is used to output a decision (maybe something like right, left, or ...
5
votes
2answers
138 views
What is the relation between the context in contextual bandits and the state in reinforcement learning?
Conceptually, in general, how is the context being handled in contextual bandits (CB), compared to states in reinforcement learning (RL)?
Specifically, in RL, we can use a function approximator (e.g. ...
0
votes
0answers
26 views
Understanding policies in helicopter control in the paper by Andrew Ng et al
I was going through this paper on helicopter flight control using reinforcement learning by Andrew Ng et al.
It defines two policy classes to learn two policies, one for hovering the helicopter and ...
2
votes
0answers
33 views
What trait of a planning problem makes reinforcement learning a well suited solution?
Planning problems have been the first problems studied at the dawn of AI (Shakey the robot). Graph search (e.g. A*) and planning (e.g. GraphPlan) algorithms can be very efficient at generating a plan. ...
1
vote
1answer
33 views
Embedding Isolation game states into key values for RL
I'm trying to think of how I can embed a game's state into a unique key value. The game I'm specifically working with is Isolation: https://en.wikipedia.org/wiki/Isolation_(board_game). The game state ...
-1
votes
0answers
29 views
Frequency of State Update in a DQN?
I have built my DQN. The environment is a blank white image. The goal is for the agent to draw a single line on the image consisting of two points for each step. The image resolution is $500 \times ...
0
votes
0answers
13 views
How to create a customized environment in Open AI based on a trained neural network? [migrated]
I am aware of the way to create a custom environment in Open AI.But is it possible to do it based on a trained neural network?
I have a trained neural network which specifies the relationship between ...
2
votes
1answer
41 views
What should the input and output of the Q-network be in the case of an ordinal action space?
I recently started looking into implementations of the DQN algorithm (e.g. TensorFlow) in some more detail. All the implementations that I found use a network that gives an output for each possible ...
5
votes
1answer
94 views
How to improve the reward signal when the rewards are sparse?
In cases where the reward is delayed, this can negatively impact a models ability to do proper credit assignment. In the case of a sparse reward, are there ways in which this can be negated?
In a ...
1
vote
1answer
38 views
Why is the update in-place faster than the out-of-place one in dynamic programming?
In Barto and Sutton's book, it's written that we have two types of updates in dynamic programming
Update out-of-place
Update in-place
The update in-place is the faster one. Why is that the case?
...
1
vote
2answers
93 views
What is the target output for updating a Deep Q Network
I'm trying to implement Deep Q-Learning for a pet problem having a continuous state space and discretized action space.
The algorithm for table-based Q-Learning updates a single entry of the Q table - ...
2
votes
1answer
50 views
Can the rewards be matrices when using DQN?
I have a basic question. I'm working towards developing a reward function for my DQN. I'd like to train an RL agent to edit pixels on an image. I understand that convolutions are ideal for working ...
0
votes
0answers
36 views
What are the popular approaches to Q-value approximation?
I need the q-value for my RL training, there are some approaches:
Brute-force the action sequence (this won't work for long sequence)
Use classic algorithm to optimise and estimate (this ain't much ...
5
votes
3answers
457 views
In Q-learning, wouldn't it be better to simply iterate through all possible states?
In Q-learning, all resources I've found seem to say that the algorithm to update the Q-table should start at some initial state, and pick actions (which are sometimes random) to explore the state ...
1
vote
0answers
28 views
Understanding loss function gradient in asynchronous advantage actor-critic (A3C) algorithm
This is a question I posted here. I am asking it on this StackExchange branch as well, so that more people who could potentially answer get to see the question.
In the A3C algorithm from the original ...
1
vote
2answers
126 views
Can stochastic gradient descent be properly used in any sample based learning algorithm in Reinforcement Learning?
Assuming we use an MSE cost function of the form
$$ \sum_s\mu(s)(V_{\pi}(S_t)-\hat{V}(S_t,\theta_t))^2 = E_{\mu(s)}[(V_{\pi}(S_t)-\hat{V}(S_t,\theta_t))^2])$$
The Stochastic Gradient Descent is used ...
0
votes
0answers
22 views
PPO: sampling next action vs picking the most probable action
According to the original Proximal Policy Optimization paper (PPO paper), we always sample an action from the actor distribution.
According to the link
The overall loss is calculated as
$\text{loss} =...
3
votes
1answer
114 views
Is my proof of equation 0.6 in the book “Reinforcement Learning: Theory and Algorithms” correct?
In Sham Kakade's Reinforcement Learning: Theory and Algorithms, this equation (page 17) is used preceding the proof of performance difference lemma.
I am attempting to prove equation 0.6. Here is my ...
7
votes
2answers
919 views
What are the biggest barriers to get RL in production?
I am studying the state of the art of Reinforcement Learning, and my point is that we see so many applications in the real world using Supervised and Unsupervised learning algorithms in production, ...
1
vote
2answers
68 views
Is there a natural way to define the terminal state from the MDP transition probabilities $p(s',r|s,a)$?
I'm learning the basics of RL and I'm struggling to understand the notion of terminal state in MDPs.
To ask my question straightforwardly: is there a natural way to define the terminal state from the ...
2
votes
1answer
48 views
Can I add expert data to the replay buffer used by the DDPG algorithm in order to make it converge faster?
I am working on a restricted reinforcement learning environment, i.e. the environment breaks very often (i.e.: the communication between the simulator and reinforcement learning agent breaks after ...
1
vote
0answers
20 views
Gradual decrease in performance of a DDPG agent
I'm trying to solve the OpenAI's CarRacing-v0 environment with the DDPG algorithm. I've observed that after a period of learning, the agent's performance starts to deteriorate slowly. For some ...
2
votes
1answer
61 views
How can I go from $R(s)$ to $R(s,a)$ in this specific MDP?
I'm trying to implement a research paper, as explained in this other post, here the author of the paper assumed R as a function of both states and actions, while the code (and the MDP) I'm using to ...
2
votes
0answers
54 views
What is the dimensionality of these derivatives in the paper “Active Learning for Reward Estimation in Inverse Reinforcement Learning”?
I'm trying to implement in code part of the following paper: Active Learning for Reward Estimation in Inverse Reinforcement Learning.
I'm specifically referring to section 2.3 of the paper.
Let's ...
2
votes
2answers
65 views
Reinforcement Learning algorithm with rewards dependent both on previous action and current action
Problem description:
Suppose we have an environment, where a reward at time step $t$ is dependent not only on the current action, but also on previous action in the following way:
if current action ==...
4
votes
1answer
55 views
How should I implement the state transition when it is a Gaussian distribution?
I am reading this paper Anxiety, Avoidance and Sequential Evaluation and is confused about the implementation of a specific lab study. Namely, the authors model what is called the Balloon task using a ...
1
vote
1answer
160 views
What is a “learned policy” in Q-learning?
I am completing an assignment at the moment. One of the assignment questions asks how you identified the learned policy and how you obtained it. The question is a reinforcement learning question, and ...
0
votes
1answer
51 views
Relationship between Rewards and Q Value (Graph between Q(s, a) vs episodes)
I'm employing the Actor-Critic algorithm. The critic network approximates the action-value function, i.e. $Q(s, a)$, which determines how good a particular state is, when provided with an action.
$Q(s,...
2
votes
0answers
45 views
Reinforcement learning and Graph Neural Networks: Entropy drops to zero
I am currently working on an experiment to link reinforcement learning with graph neural networks.
This is my architecture:
Feature Extraction with GCN:
there is a fully meshed topology with ...
