Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

Filter by
Sorted by
Tagged with
0
votes
2answers
46 views

Is there a natural way to define the terminal state from the MDP transition probabilities $p(s',r|s,a)$?

I'm learning the basics of RL and I'm struggling to understand the notion of terminal state in MDPs. To ask my question straightforwardly: is there a natural way to define the terminal state from the ...
1
vote
1answer
64 views

How to best make use of learning rate scheduling in reinforcement learning?

How to best make use of learning rate scheduling in reinforcement learning? To me, a low learning rate towards the end to fine-tune what you've learned with subtle updates makes sense. But I don't ...
2
votes
1answer
41 views

How to let the agent choose how to populate a state space matrix in RL (using python)

I have an agent (drone) that has to allocate subchannels for different types of User Equipment. I have represented the subchannel allocation with a 2-dimentional binary matrix, that is initialized to ...
-2
votes
0answers
23 views

Can reinforcement learning and evolutionary algorithms be the same? [closed]

https://ai.googleblog.com/2018/03/using-evolutionary-automl-to-discover.html As shown in the table, evolutionary algorithms are superior. Question 1. If reinforcement learning algorithms are rewarded ...
0
votes
0answers
38 views

Which RL algorithm would be suitable for this multi-dimensional and continuous action space?

Is there an RL approach/algorithm that would be suited for the following kind of problem? There is a continuous action space with an action value $A_{a,t}$ for each action dimension $a$. The ...
2
votes
0answers
14 views

Are there any known disadvantages of implementing vanilla Q-learning on a discretized-state space environment?

For an RL problem on a continuous state space, the states could be discretized into buckets and these buckets used in implementing the Q-table. I see that is what is done here. However, according to ...
1
vote
1answer
25 views

What is a good convergence criterion for Q-learning in a stochastic environment?

I have a stochastic environment and I'm implementing a Q-table for the learning that happens on the environment. The code is shown below. In short, there are ten states (0, 1, 2,...,9), and three ...
0
votes
0answers
23 views

Control Strategy for Multi-Agent Systems

I am working on a smart grid system which can be modeled with multiple agents interacting with each other. The agents are physically coupled with non-linearities, the action of one agent has a direct ...
3
votes
1answer
42 views

Is there a document with a list of conjectures or research problems regarding reinforcement learning (like the Millennium Prize Problems)?

Is there a document with a list of conjectures or research problems regarding reinforcement learning like the Millennium Prize Problems?
2
votes
1answer
57 views

Does stochasticity of an environment necessarily mean non-stationarity in MDPs?

Is a stochastic environment necessarily also non-stationary? To elaborate, consider a two-state environment ($s_1$ and $s_2$), with two actions $a_1$ and $a_2$. In $s_1$, taking action $a_1$ has a ...
4
votes
1answer
2k views

What is the difference between a stationary and a non-stationary policy?

In reinforcement learning, there are deterministic and non-deterministic (or stochastic) policies, but there are also stationary and non-stationary policies. What is the difference between a ...
2
votes
1answer
58 views

How would I compute the optimal state-action value for a certain state and action?

I am currently trying to learn reinforcement learning and I started with the basic gridworld application. I tried Q-learning with the following parameters: Learning rate = 0.1 Discount factor = 0.95 ...
1
vote
0answers
41 views

Why is Openai's PPO2 implementation differentiable?

I'm trying to understand the concept behind the implementation of the OpenAI PPO2 algorithm. The loss function that is minimized is as follows: ...
6
votes
2answers
691 views

What are the state-of-the-art meta-reinforcement learning methods?

This question can seem a little bit too broad, but I am wondering what are the current state-of-the-art works on meta reinforcement learning. Can you provide me with the current state-of-the-art in ...
1
vote
2answers
58 views

What is the purpose of storing the action $a$ within an experience tuple?

From what I understand, experience replay works by storing tuples of $(s, a, r, s')$ to be sampled for training. I understand why we store $s$, $r$ and $s'$. However, I do not understand the need for ...
0
votes
1answer
43 views

Can I train a DQN on the same dataset for multiple epochs?

I am trying to learn about reinforcement learning and chose the stock market to experiment with. I have minute by minute historical data on a particular stock for the past 20 years. I am using a ...
1
vote
1answer
65 views

What is the difference between Bayes-adaptive MDP and a Belief-MDP in Reinforcement Learning?

I have been reading a few papers in this area recently and I keep coming across these two terms. As far as I'm aware, Belief-MDPs are when you cast a POMDP as a regular MDP with a continuous state ...
4
votes
2answers
168 views

What is the difference between return and expected return?

At a time step $t$, for a state $S_{t}$, the return is defined as the discounted cumulative reward from that time step $t$. If an agent is following a policy (which in itself is a probability ...
4
votes
1answer
160 views

Why does Q-learning converge under 100% exploration rate?

I am working on this assignment where I made the agent learn state-action values (Q-values) with Q-learning and 100% exploration rate. The environment is the classic gridworld as shown in the ...
1
vote
1answer
82 views

Understanding GLIE conditions for epsilon greedy approach

I was going through this course on reinforcement learning (the course has two lecture videos and corresponding slides) and I had a doubt. On slide 18 of this pdf, it states following condition for an ...
1
vote
0answers
39 views

How should I simulate this Markov Decision Process?

I am working on solving a problem on nodes in a graph communicating with each other. They try to estimate a central state using Kalman consensus filter, with the connections described by the graph's ...
1
vote
0answers
25 views

PPO2: Intuition behind Gumbel Softmax and Exploration?

I'm trying to understand the logic behind the magic of using the gumbel distribution for action sampling inside the PPO2 algorithm. This code snippet implements the action sampling, taken from here: <...
4
votes
1answer
126 views

How can I design a hierarchy of agents each of which with different goals?

I read some light material earlier about the possibility of building a hierarchy of agents, where the agents at the leaves solve primitive tasks while higher-level agents are optimized for ...
1
vote
1answer
110 views

What is the difference between a fitness function and a reward function?

In reinforcement learning (RL), the reward function (RF), which can be denoted as $r(s)$, $r(s, a)$, $r(s, a, s')$, $r(s, s')$ depending on its specific definition, provides the learning signal, which ...
4
votes
1answer
101 views

How are afterstate value functions mathematically defined?

In this answer, afterstate value functions are mentioned, and that temporal-difference (TD) and Monte Carlo (MC) methods can also use these value functions. Mathematically, how are these value ...
0
votes
0answers
16 views

How DynaQ behaves in stochastic world in comparison with other reinforcement learning algorithms?

I came across of implementations of a bunch of algorithms on stochastic windy gridworld. This is the graph comparing their performance: So clearly, it seems that DynaQ performs better than all other ...
1
vote
0answers
28 views

Clarifying representation of Neural Nerwork input for Chess Alpha Zero

In the Alpha Zero paper (https://arxiv.org/pdf/1712.01815.pdf) page 13, the input for the NN is described. In the beggining of the page, the authors state that: "The input to the Neural Network ...
2
votes
1answer
92 views

Can supervised learning be used to solve the inverted pendulum problem?

I know that reinforcement learning has been used to solve the inverted pendulum problem. Can supervised learning be used to solve the inverted pendulum problem? For example, there could be an ...
1
vote
1answer
75 views

Q table not converging for an arbitrary experiment

This is an experiment in order to understand the working of Q table and Q learning. I have the states as states = [0,1,2,3] I have an arbitrary value for each ...
2
votes
2answers
61 views

Should we feed a greater fraction of terminal states to the value network so that their values are learned first?

The basis of Q-learning is recursive (similar to dynamic programming), where only the absolute value of the terminal state is known. Shouldn't it make sense to feed the model a greater proportion of ...
9
votes
2answers
5k views

What is the difference between reinforcement learning and optimal control?

Coming from a process (optimal) control background, I have begun studying the field of deep reinforcement learning. Sutton & Barto (2015) state that particularly important (to the writing of the ...
8
votes
1answer
788 views

Why is reinforcement learning not the answer to AGI?

I previously asked a question about How can an AI freely make decisions?. I got a great answer about how current algorithms lack agency. The first thing I thought of was reinforcement learning, since ...
8
votes
3answers
2k views

How can an AI freely make decisions?

Suppose a deep neural network is created using Keras or Tensorflow. Usually, when you want to make a prediction, the user would invoke model.predict. However, how ...
4
votes
1answer
1k views

Is there a machine learning model that can be trained with labels that only say how “right” or “wrong” it was?

I'm trying to find the name for a model that is used to output a decision (maybe something like right, left, or ...
2
votes
1answer
120 views

How long should the state-dependent baseline for policy gradient methods be trained at each iteration?

How long should the state-dependent baseline be trained at each iteration? Or what baseline loss should we target at each iteration for use with policy gradient methods? I'm using this equation to ...
1
vote
0answers
19 views

Is (log-)standard deviation learned in TRPO and PPO or fixed instead?

After having read Williams (1992), where it was suggested that actually both the mean and standard deviation can be learned while training a REINFORCE algorithm on generating continuous output values, ...
0
votes
1answer
62 views

Why aren’t heuristics for Connect Four Monte Carlo tree search improving the agent?

I’ve created an agent using MCTS to play Connect Four. It wins against humans pretty well, but I’d like to improve upon it. I decided to add domain knowledge to the MCTS rollout stage. My evaluation ...
1
vote
0answers
30 views

Confusion about computing policy gradient with automatic differentiation ( material from Berkeley CS285)

I am taking Berkeley’s CS285 via self-study. On this particular lecture regarding Policy Gradient, I am very confused about the inconsistency between the concept explanation and the demonstration of ...
2
votes
3answers
78 views

Why aren't exploration techniques, such as UCB or Thompson sampling, used in full RL problems?

Why aren't exploration techniques, such as UCB or Thompson sampling, typically used in bandit problems, used in full RL problems? Monte Carlo Tree Search may use the above-mentioned methods in its ...
5
votes
2answers
136 views

What is the relation between the context in contextual bandits and the state in reinforcement learning?

Conceptually, in general, how is the context being handled in contextual bandits (CB), compared to states in reinforcement learning (RL)? Specifically, in RL, we can use a function approximator (e.g. ...
4
votes
1answer
103 views

Research into social behavior in Prisoner's Dilemma

I've been working on research into reproducing social behavior using multi-agent reinforcement learning. My focus has been on a GridWorld-style game, but I was thinking that maybe a simpler Prisoner's ...
2
votes
1answer
57 views

Are linear approximators better suited to some tasks compared to complex neural net functions?

Model based RL attempts to learn a function $f(s_{t+1}|s_t, a_t)$ representing the environment transitions, otherwise known as a model of the system. I see linear functions are still being used in ...
0
votes
0answers
26 views

Understanding policies in helicopter control in the paper by Andrew Ng et al

I was going through this paper on helicopter flight control using reinforcement learning by Andrew Ng et al. It defines two policy classes to learn two policies, one for hovering the helicopter and ...
3
votes
2answers
100 views

Why is regret so defined in MABs?

Consider a multi-armed bandit(MAB). There are $k$ arms, with reward distributions $R_i$ where $1 \leq i \leq k$. Let $\mu_i$ denote the mean of the $i^{th}$ distribution. If we run the multi-armed ...
0
votes
1answer
32 views

Why would the reward of A3C with LSTM suddenly drop off after many episodes?

I am training an A3C with stacked LSTM. During initial training, my model was giving descent +ve reward. However, after many episodes, its reward just goes to zero and is continuing for a long time. ...
1
vote
1answer
49 views

What are some suitable positive functions as activations of neural networks?

I am working on a deep Q-learning project. My project is different than normal deep Q-learning. The rewards of my neural network must be positive because I need their values to importance sample ...
2
votes
1answer
90 views

NoisyNet DQN with default parameters not exploring

I implemented a DQN algorithm that plays OpenAIs Cartpole environment. The NN architecture consists of 3 normal linear layers that encode the state, and one noisy linear layer, that predicts the Q ...
5
votes
1answer
2k views

How to deal with different actions for different states of the environment?

I'm new to this AI/Machine Learning and was playing around with OpenAI Gym a bit. When looking through the environments, I came across the Blackjack-v0 environment, ...
2
votes
0answers
33 views

What trait of a planning problem makes reinforcement learning a well suited solution?

Planning problems have been the first problems studied at the dawn of AI (Shakey the robot). Graph search (e.g. A*) and planning (e.g. GraphPlan) algorithms can be very efficient at generating a plan. ...
7
votes
2answers
1k views

What algorithms are considered reinforcement learning algorithms?

What are the areas that belong to the Reinforcement Learning? TD(0), Q-Learning and SARSA are all temporal-difference algorithms, which belong to the reinforcement learning area, but is there more to ...

1
2 3 4 5
33