Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

615 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3
votes
0answers
52 views

How to deal with approximate states when doing path planning?

If one is interested in implementing a path planning algorithm that is grid-based, one needs to consider the fact that your grid points will never represent the true state of the robot. How is this ...
3
votes
0answers
319 views

How to define an action space when an agent can take multiple sub-actions in a step?

I'm attempting to design an action space in OpenAI's gym and hitting the following roadblock. I've looked at this post which is closely related but subtly different. The environment I'm writing needs ...
3
votes
0answers
50 views

Evaluation a policy learned using Q - learning

I have been reading literature on reinforcement learning in healthcare. I am slightly confused between the policy evaluation for both SARSA and Q-learning. To my knowledge, I believe that SARSA is ...
3
votes
0answers
54 views

Are there reinforcement learning algorithms not based on Markov decision processes?

Are all RL algorithms based on the MDP? If not, could you give examples of some which aren't? I've looked elsewhere, but I haven't seen it explicitly said.
3
votes
0answers
51 views

How exactly does self-play work, and how does it relate to MCTS?

I am working towards using RL to create an AI for a two-player, hidden-information, a turn-based board game. I have just finished David Silver's RL course and Denny Britz's coding exercises, and so am ...
3
votes
0answers
43 views

How to deal with nonstationary rewards in asymmetric self-play reinforcement learning?

Suppose we're training two agents to play an asymmetric game from scratch using self play (like Zerg vs. Protoss in Starcraft). During training one of the agents can become stronger (discover a good ...
3
votes
0answers
38 views

Reinforcement Learning on quantum circuit

I am trying to teach an agent to make any random 1-qubit state reach uniform superposition. So basically, the full circuit will be ...
3
votes
1answer
114 views

Understanding proof of lemma 1 (policy improvement bound) of the “Trust Region Policy Optimization” paper

In the Trust Region Policy Optimization paper, in Lemma 1 of Appendix A, I did not quite understand the transition from (21) from (20). In going from (20) to (21), $A^\pi(s_t, a_t)$ is substituted ...
3
votes
0answers
51 views

Ideas on a network that can translate image differences into motor commands?

I'd like to design a network that gets two images (an image under construction, and an ideal image), and has to come up with an action vector for a simple motor command which would augment the image ...
3
votes
0answers
140 views

Is there a way to do reinforcement learning in POMDP?

Are there any algorithms to use reinforcement learning to learn optimal policies in partially observable Markov decision process (POMDP) i.e. when the state is not perfectly observed. More ...
3
votes
0answers
71 views

What is the difference between random and sequential sampling from the reply memory?

I was working on an RL problem and I am confused at one specific point. We use replay memory so that the network learns about previous actions and how these actions lead to a success or a failure. ...
3
votes
0answers
67 views

Feasibility of using machine learning to obtain self-consistent solutions

I am a physicist and I don't have much background on machine learning or deep learning except taking a couple of courses on statistics. In physics, we often simulate a model by means of two-way ...
3
votes
0answers
29 views

Should importance sample weighting be compensated for by dynamically increasing learning rate?

I'm using Prioritized Experience Replay (PER) with a DDQN. To compensate for overfitting relatively high-value samples due to the non-uniform selection, I'm training with sample weights provided along ...
3
votes
1answer
94 views

How are the observations stored in the RNN that encodes the state?

I am a bit confused about observations in RL systems which use RNN to encode the state. I read a few papers like this and this. If I were to use a sequence of raw observations (or features) as an ...
3
votes
0answers
100 views

What is the meaning of the words 'bias' and 'variance' in RL?

In reinforcement learning approaches, like temporal-difference (TD) learning or Monte Carlo methods, two of the metrics used to measure their performance are the bias and the variance. What do these ...
3
votes
0answers
90 views

How can I use Q-learning for inventory decision making?

I am trying to model operational decisions in inventory control. The control policy is base stock with a fixed stock level of $S$. That is replenishment orders are placed for every demand arrival to ...
3
votes
0answers
64 views

Designing state representation for board game

I am trying to write self-play RL (NN + MCTS http://web.stanford.edu/~surag/posts/alphazero.html) to "solve" a board game. However, I got stuck in designing boardgame same (input layer for NN). 1) ...
3
votes
1answer
435 views

Deep Reinforcement Learning: Rewards suddenly dip down

I am working on a deep reinforcement learning problem. The policy network has the same architecture as the one Deepmind published in 'Playing Atari with Deep Reinforcement Learning'. I am also using ...
3
votes
0answers
71 views

How to represent action space in reinforcement learning?

I started to learn reinforcement learning a few days ago. And I want to use that to solve resource allocation problem something like given a constant number, find the best way to divide it into ...
3
votes
0answers
29 views

Training a reinforcement learning model with multiple images

I am tentatively trying to train a deep reinforcement learning model the maze escaping task, and each time it takes one image as the input (e.g., a different "maze"). Suppose I have about $10K$ ...
3
votes
0answers
150 views

When using hashing in tile coding, why are memory requirements reduced and there is only a little loss of performance?

In the book "Reinforcement Learning: An Introduction" (2018) Sutton and Barto explain, on page 221, a form of tile coding using hashing, to reduce memory consumption. I have two questions ...
3
votes
0answers
37 views

Can $\Phi$ measure of Integrated Information Theory serve as reward for reinforcement learning system?

Can $\Phi$ measure (computed rigorously or approximately) of Integrated Information Theory serve as reward for self-evolving/learning reinforcement learning system and hence we let this system to ...
3
votes
0answers
194 views

Should noise (such as OU) be decreased over time in actor / critic algorithms?

In most of RL algorithms I saw, there is a coefficient that reduces actions exploration over time, to help convergence. But in Actor-Critic, or other algorithms (A3C, DDPG, ...) used in continuous ...
3
votes
0answers
31 views

Code examples of controlling multiple units with RL

Anyone knows a resources (papers, articles and especially repositories) regarding controlling multiple units with RL. The controlled units should not be fixed, for example in Real Time Strategy the ...
3
votes
0answers
51 views

Deep Q-Learning agent poor performing actions. Need help optimizing

I'm trying to make deep q-learning agent from https://keon.io/deep-q-learning My environment looks like this: https://imgur.com/a/OnbiCtV As you can see my agent is a circle and there is one gray ...
3
votes
0answers
82 views

Feature Selection using Monte Carlo Tree Search

I'm trying to tackle the problem of feature selection as an RL problem, inspired by the paper Feature Selection as a One-Player Game. I know Monte-Carlo tree search (MCTS) is hardly RL. So, I used ...
3
votes
0answers
183 views

What can be considered a deep recurrent neural network?

In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the DRQN is described as DQN with the first post-convolutional fully-connected layer replaced by a recurrent LSTM. I have DQN ...
3
votes
0answers
41 views

Does everyone still use discount rates?

In Section 10.4 of Sutton and Barto's RL book, they argue that the discount rate $\gamma$ has no effect in continuing settings. They show (at least for one objective function) that the average of the ...
3
votes
0answers
55 views

Negative counterfactual regret

I am reading the paper Regret Minimization in Games with Incomplete Information on CFR algorithm. On page 4, the paper defines $R^{T,+}_{i,\text{imm}}=\max\{R^{T}_{i,\text{imm}}, 0\}$ after equation (...
3
votes
1answer
117 views

Can A3C update the policy / critic on a local machine without needing to copy?

To make A2C into A3C you make it asynchronous. From what I understand the 'correct' way to do that is to thread off workers with a copy of the policy and critic, and then return the state/action/...
3
votes
0answers
49 views

Choosing more than one action in a parameterized policy

I would like to implement a variant of policy iteration that can choose one or more actions in each state. An example would be to heal and move in the game of Doom. Parameterizing the power set of ...
3
votes
0answers
103 views

Can I use deterministic policy gradient methods for stochastic policy learning?

Can I treat a stochastic policy (over a finite action space of size $n$) as a deterministic policy (in the set of probability distribution in $\mathbb{R}^n$)? It seems to me that nothing is broken ...
3
votes
0answers
116 views

Why are all the actions converging to the same index?

I am using PPO with an LSTM agent. My agent is performing 10 actions for each episode, one action is corresponding to one LSTM timestep and the action space is discrete. I have only one reward per ...
3
votes
0answers
332 views

Solving equations using reinforcement learning

I was lately curious about a reinforcement learning approach that would solve maths equations. For example, if I have the following equation: $$ f(g(h(w))) = 0 , with \ w = \begin{matrix} a_{11} &...
3
votes
0answers
197 views

Dyna-Q algorithm, having trouble when adding the simulated experiences

I'm trying to create a simple Dyna-Q agent to solve small mazes, in python. For the Q function, Q(s, a), I'm just using a matrix, where each row is for a state value, and each column is for one of the ...
3
votes
0answers
397 views

RL to generate sentences

I want to develop a system to generate grammatically correct sentences. The input would be some words. The output would be a grammatically correct human-like sentence. Eg: Input: capital, Paris, ...
2
votes
1answer
12 views

Are policy and value iteration used only in grid world like scenarios?

I am trying to self learn reinforcement learning. At the moment I am focusing on policy and value iteration, and I am finding several problems and doubts. One of the main doubts is given by the fact ...
2
votes
1answer
30 views

How does sharing parameters between the policy and value functions help in PPO?

The PPO objective may include a value function error term when parameters are shared between the policy and value functions. How does this help, and when to use a neural network architecture that ...
2
votes
0answers
79 views

Update Rule with Deep Q-Learning (DQN) for 2-player games

I am wondering how to correctly implement the DQN algorithm for two-player games such as Tic Tac Toe and Connect 4. While my algorithm is mastering Tic Tac Toe relatively quickly, I cannot get great ...
2
votes
0answers
40 views

Why isn't RL considered a continual learning strategy itself?

I have read about methods that apply continual learning strategies to reinforcement learning. Since reinforcement learning also learns step by step (i.e., task by task, in a sense) during the training ...
2
votes
0answers
33 views

Is there any reasonable notion of regret for infinite horizon discounted MDPs?

I am thinking about episodic MDPs. Usually, in episodic MDPs, it seems that we have a finite fixed horizon per episode and no discount factor. Then, a very intuitive notion of regret after $T$ ...
2
votes
0answers
38 views

Off-policy Bellman Operators: Writing Operator and Weight Update Function for a 2-State System

I am studying for RL on my own and was trying to solve this question I came across. Write an operator function $T(w, \pi, \mu, l, g)$ that takes weights $w$, a target policy $\pi$, a behaviour policy ...
2
votes
0answers
48 views

How do I implement the cross-entropy-method for a RL environment with a continuous action space?

I found many tutorials and posts on how to solve RL environments with discrete action spaces using the cross entropy method (e.g., in this blog post for the OpenAI Gym frozen lake environment). ...
2
votes
0answers
46 views

Train agent to surround a burning fire

I have built a wildfire 'simulation' in unity. And I want to train an RL agent to 'control' this fire. However, I think my task is quite complicated, and I can't work out to get the agent to do what I ...
2
votes
0answers
26 views

What is the name of this algorithm that estimates the gradient with an average by sampling from a distribution?

Consider maximizing the function $R(w)$ with parameter $w$ using gradient ascent. However, we don't know the gradient $\nabla_wR(w)$ formula. Now suppose $w$ is sampled from a probability distribution ...
2
votes
1answer
46 views

How can I compress the states of a reinforcement learning agent?

I'm working on a problem that involves an RL agent with very large states. These states consist of several pieces of information about the agent. The states are not images, so techniques like ...
2
votes
0answers
27 views

How to find good features for a linear function approximation in RL with large discrete state set?

I've recently read much about feature engineering in continuous (uncountable) feature spaces. Now I am interested what methods exist in the setting of large discrete state spaces. For example consider ...
2
votes
0answers
30 views

Where does the hierarchical reinforcement learning framework name “MAXQ” come from?

I've been researching different frameworks for hierarchical RL (mainly options, HAMs, and MAXQ) and noticed that both options and HAMs have names that relate to how they function. I can't seem to find ...
2
votes
0answers
23 views

Are there any known disadvantages of implementing vanilla Q-learning on a discretized-state space environment?

For an RL problem on a continuous state space, the states could be discretized into buckets and these buckets used in implementing the Q-table. I see that is what is done here. However, according to ...
2
votes
0answers
50 views

Is (log-)standard deviation learned in TRPO and PPO or fixed instead?

After having read Williams (1992), where it was suggested that actually both the mean and standard deviation can be learned while training a REINFORCE algorithm on generating continuous output values, ...

1
2
3 4 5
13