Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

Filter by
Sorted by
Tagged with
-2
votes
0answers
30 views

Can reinforcement learning and evolutionary algorithms be the same? [closed]

https://ai.googleblog.com/2018/03/using-evolutionary-automl-to-discover.html As shown in the table, evolutionary algorithms are superior. Question 1. If reinforcement learning algorithms are rewarded ...
2
votes
0answers
15 views

Are there any known disadvantages of implementing vanilla Q-learning on a discretized-state space environment?

For an RL problem on a continuous state space, the states could be discretized into buckets and these buckets used in implementing the Q-table. I see that is what is done here. However, according to ...
1
vote
1answer
29 views

What is a good convergence criterion for Q-learning in a stochastic environment?

I have a stochastic environment and I'm implementing a Q-table for the learning that happens on the environment. The code is shown below. In short, there are ten states (0, 1, 2,...,9), and three ...
2
votes
1answer
57 views

Does stochasticity of an environment necessarily mean non-stationarity in MDPs?

Is a stochastic environment necessarily also non-stationary? To elaborate, consider a two-state environment ($s_1$ and $s_2$), with two actions $a_1$ and $a_2$. In $s_1$, taking action $a_1$ has a ...
0
votes
0answers
23 views

Control Strategy for Multi-Agent Systems

I am working on a smart grid system which can be modeled with multiple agents interacting with each other. The agents are physically coupled with non-linearities, the action of one agent has a direct ...
1
vote
0answers
41 views

Which RL algorithm would be suitable for this multi-dimensional and continuous action space?

Is there an RL approach/algorithm that would be suited for the following kind of problem? There is a continuous action space with an action value $A_{a,t}$ for each action dimension $a$. The ...
1
vote
2answers
58 views

What is the purpose of storing the action $a$ within an experience tuple?

From what I understand, experience replay works by storing tuples of $(s, a, r, s')$ to be sampled for training. I understand why we store $s$, $r$ and $s'$. However, I do not understand the need for ...
2
votes
1answer
58 views

How would I compute the optimal state-action value for a certain state and action?

I am currently trying to learn reinforcement learning and I started with the basic gridworld application. I tried Q-learning with the following parameters: Learning rate = 0.1 Discount factor = 0.95 ...
3
votes
1answer
43 views

Is there a document with a list of conjectures or research problems regarding reinforcement learning (like the Millennium Prize Problems)?

Is there a document with a list of conjectures or research problems regarding reinforcement learning like the Millennium Prize Problems?
4
votes
1answer
160 views

Why does Q-learning converge under 100% exploration rate?

I am working on this assignment where I made the agent learn state-action values (Q-values) with Q-learning and 100% exploration rate. The environment is the classic gridworld as shown in the ...
1
vote
0answers
41 views

Why is Openai's PPO2 implementation differentiable?

I'm trying to understand the concept behind the implementation of the OpenAI PPO2 algorithm. The loss function that is minimized is as follows: ...
1
vote
0answers
40 views

How should I simulate this Markov Decision Process?

I am working on solving a problem on nodes in a graph communicating with each other. They try to estimate a central state using Kalman consensus filter, with the connections described by the graph's ...
1
vote
0answers
25 views

PPO2: Intuition behind Gumbel Softmax and Exploration?

I'm trying to understand the logic behind the magic of using the gumbel distribution for action sampling inside the PPO2 algorithm. This code snippet implements the action sampling, taken from here: <...
0
votes
0answers
16 views

How DynaQ behaves in stochastic world in comparison with other reinforcement learning algorithms?

I came across of implementations of a bunch of algorithms on stochastic windy gridworld. This is the graph comparing their performance: So clearly, it seems that DynaQ performs better than all other ...
1
vote
0answers
28 views

Clarifying representation of Neural Nerwork input for Chess Alpha Zero

In the Alpha Zero paper (https://arxiv.org/pdf/1712.01815.pdf) page 13, the input for the NN is described. In the beggining of the page, the authors state that: "The input to the Neural Network ...
0
votes
1answer
43 views

Can I train a DQN on the same dataset for multiple epochs?

I am trying to learn about reinforcement learning and chose the stock market to experiment with. I have minute by minute historical data on a particular stock for the past 20 years. I am using a ...
1
vote
0answers
19 views

Is (log-)standard deviation learned in TRPO and PPO or fixed instead?

After having read Williams (1992), where it was suggested that actually both the mean and standard deviation can be learned while training a REINFORCE algorithm on generating continuous output values, ...
1
vote
0answers
30 views

Confusion about computing policy gradient with automatic differentiation ( material from Berkeley CS285)

I am taking Berkeley’s CS285 via self-study. On this particular lecture regarding Policy Gradient, I am very confused about the inconsistency between the concept explanation and the demonstration of ...
4
votes
1answer
1k views

Is there a machine learning model that can be trained with labels that only say how “right” or “wrong” it was?

I'm trying to find the name for a model that is used to output a decision (maybe something like right, left, or ...
5
votes
2answers
137 views

What is the relation between the context in contextual bandits and the state in reinforcement learning?

Conceptually, in general, how is the context being handled in contextual bandits (CB), compared to states in reinforcement learning (RL)? Specifically, in RL, we can use a function approximator (e.g. ...
0
votes
0answers
26 views

Understanding policies in helicopter control in the paper by Andrew Ng et al

I was going through this paper on helicopter flight control using reinforcement learning by Andrew Ng et al. It defines two policy classes to learn two policies, one for hovering the helicopter and ...
2
votes
0answers
33 views

What trait of a planning problem makes reinforcement learning a well suited solution?

Planning problems have been the first problems studied at the dawn of AI (Shakey the robot). Graph search (e.g. A*) and planning (e.g. GraphPlan) algorithms can be very efficient at generating a plan. ...
1
vote
1answer
33 views

Embedding Isolation game states into key values for RL

I'm trying to think of how I can embed a game's state into a unique key value. The game I'm specifically working with is Isolation: https://en.wikipedia.org/wiki/Isolation_(board_game). The game state ...
-1
votes
0answers
29 views

Frequency of State Update in a DQN?

I have built my DQN. The environment is a blank white image. The goal is for the agent to draw a single line on the image consisting of two points for each step. The image resolution is $500 \times ...
0
votes
0answers
13 views

How to create a customized environment in Open AI based on a trained neural network? [migrated]

I am aware of the way to create a custom environment in Open AI.But is it possible to do it based on a trained neural network? I have a trained neural network which specifies the relationship between ...
2
votes
1answer
40 views

What should the input and output of the Q-network be in the case of an ordinal action space?

I recently started looking into implementations of the DQN algorithm (e.g. TensorFlow) in some more detail. All the implementations that I found use a network that gives an output for each possible ...
5
votes
1answer
93 views

How to improve the reward signal when the rewards are sparse?

In cases where the reward is delayed, this can negatively impact a models ability to do proper credit assignment. In the case of a sparse reward, are there ways in which this can be negated? In a ...
1
vote
1answer
38 views

Why is the update in-place faster than the out-of-place one in dynamic programming?

In Barto and Sutton's book, it's written that we have two types of updates in dynamic programming Update out-of-place Update in-place The update in-place is the faster one. Why is that the case? ...
1
vote
2answers
92 views

What is the target output for updating a Deep Q Network

I'm trying to implement Deep Q-Learning for a pet problem having a continuous state space and discretized action space. The algorithm for table-based Q-Learning updates a single entry of the Q table - ...
2
votes
1answer
50 views

Can the rewards be matrices when using DQN?

I have a basic question. I'm working towards developing a reward function for my DQN. I'd like to train an RL agent to edit pixels on an image. I understand that convolutions are ideal for working ...
0
votes
0answers
36 views

What are the popular approaches to Q-value approximation?

I need the q-value for my RL training, there are some approaches: Brute-force the action sequence (this won't work for long sequence) Use classic algorithm to optimise and estimate (this ain't much ...
5
votes
3answers
455 views

In Q-learning, wouldn't it be better to simply iterate through all possible states?

In Q-learning, all resources I've found seem to say that the algorithm to update the Q-table should start at some initial state, and pick actions (which are sometimes random) to explore the state ...
1
vote
0answers
27 views

Understanding loss function gradient in asynchronous advantage actor-critic (A3C) algorithm

This is a question I posted here. I am asking it on this StackExchange branch as well, so that more people who could potentially answer get to see the question. In the A3C algorithm from the original ...
1
vote
2answers
125 views

Can stochastic gradient descent be properly used in any sample based learning algorithm in Reinforcement Learning?

Assuming we use an MSE cost function of the form $$ \sum_s\mu(s)(V_{\pi}(S_t)-\hat{V}(S_t,\theta_t))^2 = E_{\mu(s)}[(V_{\pi}(S_t)-\hat{V}(S_t,\theta_t))^2])$$ The Stochastic Gradient Descent is used ...
0
votes
0answers
22 views

PPO: sampling next action vs picking the most probable action

According to the original Proximal Policy Optimization paper (PPO paper), we always sample an action from the actor distribution. According to the link The overall loss is calculated as $\text{loss} =...
3
votes
1answer
113 views

Is my proof of equation 0.6 in the book “Reinforcement Learning: Theory and Algorithms” correct?

In Sham Kakade's Reinforcement Learning: Theory and Algorithms, this equation (page 17) is used preceding the proof of performance difference lemma. I am attempting to prove equation 0.6. Here is my ...
7
votes
2answers
917 views

What are the biggest barriers to get RL in production?

I am studying the state of the art of Reinforcement Learning, and my point is that we see so many applications in the real world using Supervised and Unsupervised learning algorithms in production, ...
1
vote
2answers
66 views

Is there a natural way to define the terminal state from the MDP transition probabilities $p(s',r|s,a)$?

I'm learning the basics of RL and I'm struggling to understand the notion of terminal state in MDPs. To ask my question straightforwardly: is there a natural way to define the terminal state from the ...
2
votes
1answer
48 views

Can I add expert data to the replay buffer used by the DDPG algorithm in order to make it converge faster?

I am working on a restricted reinforcement learning environment, i.e. the environment breaks very often (i.e.: the communication between the simulator and reinforcement learning agent breaks after ...
1
vote
0answers
20 views

Gradual decrease in performance of a DDPG agent

I'm trying to solve the OpenAI's CarRacing-v0 environment with the DDPG algorithm. I've observed that after a period of learning, the agent's performance starts to deteriorate slowly. For some ...
2
votes
1answer
61 views

How can I go from $R(s)$ to $R(s,a)$ in this specific MDP?

I'm trying to implement a research paper, as explained in this other post, here the author of the paper assumed R as a function of both states and actions, while the code (and the MDP) I'm using to ...
2
votes
0answers
54 views

What is the dimensionality of these derivatives in the paper “Active Learning for Reward Estimation in Inverse Reinforcement Learning”?

I'm trying to implement in code part of the following paper: Active Learning for Reward Estimation in Inverse Reinforcement Learning. I'm specifically referring to section 2.3 of the paper. Let's ...
2
votes
2answers
65 views

Reinforcement Learning algorithm with rewards dependent both on previous action and current action

Problem description: Suppose we have an environment, where a reward at time step $t$ is dependent not only on the current action, but also on previous action in the following way: if current action ==...
4
votes
1answer
54 views

How should I implement the state transition when it is a Gaussian distribution?

I am reading this paper Anxiety, Avoidance and Sequential Evaluation and is confused about the implementation of a specific lab study. Namely, the authors model what is called the Balloon task using a ...
1
vote
1answer
160 views

What is a “learned policy” in Q-learning?

I am completing an assignment at the moment. One of the assignment questions asks how you identified the learned policy and how you obtained it. The question is a reinforcement learning question, and ...
0
votes
1answer
50 views

Relationship between Rewards and Q Value (Graph between Q(s, a) vs episodes)

I'm employing the Actor-Critic algorithm. The critic network approximates the action-value function, i.e. $Q(s, a)$, which determines how good a particular state is, when provided with an action. $Q(s,...
2
votes
0answers
44 views

Reinforcement learning and Graph Neural Networks: Entropy drops to zero

I am currently working on an experiment to link reinforcement learning with graph neural networks. This is my architecture: Feature Extraction with GCN: there is a fully meshed topology with ...
8
votes
2answers
824 views

What is the difference between Q-learning, Deep Q-learning and Deep Q-network?

Q-learning uses a table to store all state-action pairs. Q-learning is a model-free RL algorithm, so how could there be the one called Deep Q-learning, as deep means using DNN; or maybe the state-...
3
votes
1answer
56 views

Does DQN generalise to unseen states in the case of discrete state-spaces?

In my understanding, DQN is useful because it utilises a neural network as a q-value function approximator, which, after the training, can generalise to unseen states. I understand how that would work ...
3
votes
0answers
64 views

Where does this variation of the importance sampling weight come from?

I have seeing a variation in importance sampling (IS) in Prioritized Experience Replay (PER) in some implementations regarding the original paper approach stated as (in section 3.4): $$ w_{i}=\left(\...

1
2 3 4 5
33