Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

571 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3
votes
0answers
52 views

Are there reinforcement learning algorithms not based on Markov decision processes?

Are all RL algorithms based on the MDP? If not, could you give examples of some which aren't? I've looked elsewhere, but I haven't seen it explicitly said.
3
votes
0answers
50 views

How exactly does self-play work, and how does it relate to MCTS?

I am working towards using RL to create an AI for a two-player, hidden-information, a turn-based board game. I have just finished David Silver's RL course and Denny Britz's coding exercises, and so am ...
3
votes
0answers
43 views

How to deal with nonstationary rewards in asymmetric self-play reinforcement learning?

Suppose we're training two agents to play an asymmetric game from scratch using self play (like Zerg vs. Protoss in Starcraft). During training one of the agents can become stronger (discover a good ...
3
votes
0answers
37 views

Reinforcement Learning on quantum circuit

I am trying to teach an agent to make any random 1-qubit state reach uniform superposition. So basically, the full circuit will be ...
3
votes
1answer
105 views

Understanding proof of lemma 1 (policy improvement bound) of the “Trust Region Policy Optimization” paper

In the Trust Region Policy Optimization paper, in Lemma 1 of Appendix A, I did not quite understand the transition from (21) from (20). In going from (20) to (21), $A^\pi(s_t, a_t)$ is substituted ...
3
votes
0answers
51 views

Ideas on a network that can translate image differences into motor commands?

I'd like to design a network that gets two images (an image under construction, and an ideal image), and has to come up with an action vector for a simple motor command which would augment the image ...
3
votes
0answers
121 views

Is there a way to do reinforcement learning in POMDP?

Are there any algorithms to use reinforcement learning to learn optimal policies in partially observable Markov decision process (POMDP) i.e. when the state is not perfectly observed. More ...
3
votes
0answers
49 views

What is the difference between random and sequential sampling from the reply memory?

I was working on an RL problem and I am confused at one specific point. We use replay memory so that the network learns about previous actions and how these actions lead to a success or a failure. ...
3
votes
0answers
57 views

Feasibility of using machine learning to obtain self-consistent solutions

I am a physicist and I don't have much background on machine learning or deep learning except taking a couple of courses on statistics. In physics, we often simulate a model by means of two-way ...
3
votes
0answers
29 views

Should importance sample weighting be compensated for by dynamically increasing learning rate?

I'm using Prioritized Experience Replay (PER) with a DDQN. To compensate for overfitting relatively high-value samples due to the non-uniform selection, I'm training with sample weights provided along ...
3
votes
1answer
79 views

How are the observations stored in the RNN that encodes the state?

I am a bit confused about observations in RL systems which use RNN to encode the state. I read a few papers like this and this. If I were to use a sequence of raw observations (or features) as an ...
3
votes
0answers
88 views

What is the meaning of the words 'bias' and 'variance' in RL?

In reinforcement learning approaches, like temporal-difference (TD) learning or Monte Carlo methods, two of the metrics used to measure their performance are the bias and the variance. What do these ...
3
votes
0answers
90 views

How can I use Q-learning for inventory decision making?

I am trying to model operational decisions in inventory control. The control policy is base stock with a fixed stock level of $S$. That is replenishment orders are placed for every demand arrival to ...
3
votes
0answers
64 views

Designing state representation for board game

I am trying to write self-play RL (NN + MCTS http://web.stanford.edu/~surag/posts/alphazero.html) to "solve" a board game. However, I got stuck in designing boardgame same (input layer for NN). 1) ...
3
votes
0answers
71 views

How to represent action space in reinforcement learning?

I started to learn reinforcement learning a few days ago. And I want to use that to solve resource allocation problem something like given a constant number, find the best way to divide it into ...
3
votes
0answers
29 views

Training a reinforcement learning model with multiple images

I am tentatively trying to train a deep reinforcement learning model the maze escaping task, and each time it takes one image as the input (e.g., a different "maze"). Suppose I have about $10K$ ...
3
votes
0answers
144 views

When using hashing in tile coding, why are memory requirements reduced and there is only a little loss of performance?

In the book "Reinforcement Learning: An Introduction" (2018) Sutton and Barto explain, on page 221, a form of tile coding using hashing, to reduce memory consumption. I have two questions ...
3
votes
0answers
35 views

Can $\Phi$ measure of Integrated Information Theory serve as reward for reinforcement learning system?

Can $\Phi$ measure (computed rigorously or approximately) of Integrated Information Theory serve as reward for self-evolving/learning reinforcement learning system and hence we let this system to ...
3
votes
0answers
156 views

Should noise (such as OU) be decreased over time in actor / critic algorithms?

In most of RL algorithms I saw, there is a coefficient that reduces actions exploration over time, to help convergence. But in Actor-Critic, or other algorithms (A3C, DDPG, ...) used in continuous ...
3
votes
0answers
30 views

Code examples of controlling multiple units with RL

Anyone knows a resources (papers, articles and especially repositories) regarding controlling multiple units with RL. The controlled units should not be fixed, for example in Real Time Strategy the ...
3
votes
0answers
49 views

Deep Q-Learning agent poor performing actions. Need help optimizing

I'm trying to make deep q-learning agent from https://keon.io/deep-q-learning My environment looks like this: https://imgur.com/a/OnbiCtV As you can see my agent is a circle and there is one gray ...
3
votes
0answers
79 views

Feature Selection using Monte Carlo Tree Search

I'm trying to tackle the problem of feature selection as an RL problem, inspired by the paper Feature Selection as a One-Player Game. I know Monte-Carlo tree search (MCTS) is hardly RL. So, I used ...
3
votes
0answers
170 views

What can be considered a deep recurrent neural network?

In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the DRQN is described as DQN with the first post-convolutional fully-connected layer replaced by a recurrent LSTM. I have DQN ...
3
votes
0answers
39 views

Does everyone still use discount rates?

In Section 10.4 of Sutton and Barto's RL book, they argue that the discount rate $\gamma$ has no effect in continuing settings. They show (at least for one objective function) that the average of the ...
3
votes
1answer
112 views

Why Q2 is a more or less independant estimate in Twin Delayed DDPG (TD3)?

Twin Delayed Deep Deterministic (TD3) policy gradient is inspired by both double Q-learning and double DQN. In double Q-learning, I understand that Q1 and Q2 are independent because they are trained ...
3
votes
0answers
51 views

Negative counterfactual regret

I am reading the paper Regret Minimization in Games with Incomplete Information on CFR algorithm. On page 4, the paper defines $R^{T,+}_{i,\text{imm}}=\max\{R^{T}_{i,\text{imm}}, 0\}$ after equation (...
3
votes
1answer
109 views

Can A3C update the policy / critic on a local machine without needing to copy?

To make A2C into A3C you make it asynchronous. From what I understand the 'correct' way to do that is to thread off workers with a copy of the policy and critic, and then return the state/action/...
3
votes
0answers
49 views

Choosing more than one action in a parameterized policy

I would like to implement a variant of policy iteration that can choose one or more actions in each state. An example would be to heal and move in the game of Doom. Parameterizing the power set of ...
3
votes
0answers
100 views

Can I use deterministic policy gradient methods for stochastic policy learning?

Can I treat a stochastic policy (over a finite action space of size $n$) as a deterministic policy (in the set of probability distribution in $\mathbb{R}^n$)? It seems to me that nothing is broken ...
3
votes
0answers
113 views

Why are all the actions converging to the same index?

I am using PPO with an LSTM agent. My agent is performing 10 actions for each episode, one action is corresponding to one LSTM timestep and the action space is discrete. I have only one reward per ...
3
votes
0answers
309 views

Solving equations using reinforcement learning

I was lately curious about a reinforcement learning approach that would solve maths equations. For example, if I have the following equation: $$ f(g(h(w))) = 0 , with \ w = \begin{matrix} a_{11} &...
3
votes
0answers
173 views

Dyna-Q algorithm, having trouble when adding the simulated experiences

I'm trying to create a simple Dyna-Q agent to solve small mazes, in python. For the Q function, Q(s, a), I'm just using a matrix, where each row is for a state value, and each column is for one of the ...
3
votes
0answers
286 views

Snake path finding variant : Algorithm choice

I am working on a project which maps to a variant of path finding problem. I am new to this area and I would be very grateful if you could give suggestions/ point to libraries for relevant algorithms. ...
3
votes
0answers
396 views

RL to generate sentences

I want to develop a system to generate grammatically correct sentences. The input would be some words. The output would be a grammatically correct human-like sentence. Eg: Input: capital, Paris, ...
2
votes
0answers
14 views

Are there any known disadvantages of implementing vanilla Q-learning on a discretized-state space environment?

For an RL problem on a continuous state space, the states could be discretized into buckets and these buckets used in implementing the Q-table. I see that is what is done here. However, according to ...
2
votes
0answers
33 views

What trait of a planning problem makes reinforcement learning a well suited solution?

Planning problems have been the first problems studied at the dawn of AI (Shakey the robot). Graph search (e.g. A*) and planning (e.g. GraphPlan) algorithms can be very efficient at generating a plan. ...
2
votes
0answers
54 views

What is the dimensionality of these derivatives in the paper “Active Learning for Reward Estimation in Inverse Reinforcement Learning”?

I'm trying to implement in code part of the following paper: Active Learning for Reward Estimation in Inverse Reinforcement Learning. I'm specifically referring to section 2.3 of the paper. Let's ...
2
votes
0answers
43 views

Reinforcement learning and Graph Neural Networks: Entropy drops to zero

I am currently working on an experiment to link reinforcement learning with graph neural networks. This is my architecture: Feature Extraction with GCN: there is a fully meshed topology with ...
2
votes
0answers
25 views

Is better to reward short- or long-term progress in Q-learning?

I have been training some kind of agent to reach a target using a Q-learning based approach, and I have tried two different types of rewards: Long-term reward: $\mathrm{reward} = - \mathrm{distance}(\...
2
votes
0answers
32 views

What place do Agent Communications Language have in Multi-Agent Systems nowadays?

I am currently working on implementing a Multi-Agent System for Smart Grids. There's a lot of literature for that and some things confuse me. I have read that there is FIPA, which aimed to create a ...
2
votes
0answers
17 views

Can DQN outperform DoubleDQN?

I found a similar post about this issue, but unfortunately I did not find a proper answer. Are there any references where DQN is better than DoubleDQN, that is DoubleDQN does not improve DQN ?
2
votes
0answers
21 views

Does there necessarily exist “dominated actions” in a MDP?

In a Markov Decision Process, is it possible that there exists no "dominated action"? I define a dominated action the following way: we say that $(s,a)$ is a dominated action, if $\forall \...
2
votes
0answers
44 views

How does bootstrapping work with the offline $\lambda$-return algorithm?

In Barton and Sutton's book, Reinforcement Learning: An Introduction (2nd edition), an expression, on page 289 (equation 12.2), introduced the form of the $\lambda$-return defined as follows $$G_t^{\...
2
votes
0answers
64 views

$\nabla \log \pi$ with respect to some parameters constantly being zero

I am new to reinforcement learning. May I ask a simple (and maybe a bit silly) question here? I am trying to use the "one-step actor-critic" method to train a robot on a gridworld. Let's ...
2
votes
0answers
67 views

Policy gradient: Does it use the Markov property?

To derive the policy gradient, we start by writing the equation for the probability of a certain trajectory (e.g. see spinningup tutorial): $$ \begin{align} P_\theta(\tau) &= P_\theta(s_0, a_0, ...
2
votes
0answers
34 views

How does uniform offset tiling work with function approximation?

I get the fundamental idea of how tilings work, but, in Barton and Sutton's book, Reinforcement Learning: An Introduction (2nd edition), a diagram, on page 219 (figure 9.11), showing the variations of ...
2
votes
0answers
23 views

CNN for a DQN agent with a 2D matrix state and action as a 2D matrix

I have a custom environment, where the state is a 2D matrix of 11 rows (equals to number of users to satisfy) and 3 columns. Each column can take the value of either 0 or 1, and in each row, there can ...
2
votes
0answers
124 views

How should I build this DQN agent?

I have a set of users that can be one of 3 types. They will randomly request a service from the UAV which is a drone used as a Base Station. The UAV (the agent) is tasked with allocating subchannels (...
2
votes
0answers
31 views

Why is TD(0) not converging to the optimal policy?

I am trying to implement the basic RL algorithms to learn on this 10x10 GridWorld (from REINFORCEJS by Kaparthy). Currently I am stuck at TD(0). No matter how many episodes I run, when I am updating ...
2
votes
0answers
45 views

Can $Q$-learning or SARSA be thought of a Markov Chain?

I might just be overthinking a very simple question but nonetheless the following has been bugging me a lot. Given an MDP with non-trivial state and action sets, we can implement the SARSA algorithm ...

1
2
3 4 5
12