Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

504 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3
votes
0answers
70 views

How to represent action space in reinforcement learning?

I started to learn reinforcement learning a few days ago. And I want to use that to solve resource allocation problem something like given a constant number, find the best way to divide it into ...
3
votes
0answers
28 views

Training a reinforcement learning model with multiple images

I am tentatively trying to train a deep reinforcement learning model the maze escaping task, and each time it takes one image as the input (e.g., a different "maze"). Suppose I have about $10K$ ...
3
votes
0answers
32 views

Can $\Phi$ measure of Integrated Information Theory serve as reward for reinforcement learning system?

Can $\Phi$ measure (computed rigorously or approximately) of Integrated Information Theory serve as reward for self-evolving/learning reinforcement learning system and hence we let this system to ...
3
votes
0answers
105 views

Should noise (such as OU) be decreased over time in actor / critic algorithms?

In most of RL algorithms I saw, there is a coefficient that reduces actions exploration over time, to help convergence. But in Actor-Critic, or other algorithms (A3C, DDPG, ...) used in continuous ...
3
votes
0answers
30 views

Code examples of controlling multiple units with RL

Anyone knows a resources (papers, articles and especially repositories) regarding controlling multiple units with RL. The controlled units should not be fixed, for example in Real Time Strategy the ...
3
votes
0answers
49 views

Deep Q-Learning agent poor performing actions. Need help optimizing

I'm trying to make deep q-learning agent from https://keon.io/deep-q-learning My environment looks like this: https://imgur.com/a/OnbiCtV As you can see my agent is a circle and there is one gray ...
3
votes
0answers
75 views

Feature Selection using Monte Carlo Tree Search

I'm trying to tackle the problem of feature selection as an RL problem, inspired by the paper Feature Selection as a One-Player Game. I know Monte-Carlo tree search (MCTS) is hardly RL. So, I used ...
3
votes
0answers
158 views

What can be considered a deep recurrent neural network?

In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the DRQN is described as DQN with the first post-convolutional fully-connected layer replaced by a recurrent LSTM. I have DQN ...
3
votes
0answers
39 views

Does everyone still use discount rates?

In Section 10.4 of Sutton and Barto's RL book, they argue that the discount rate $\gamma$ has no effect in continuing settings. They show (at least for one objective function) that the average of the ...
3
votes
1answer
97 views

Why Q2 is a more or less independant estimate in Twin Delayed DDPG (TD3)?

Twin Delayed Deep Deterministic (TD3) policy gradient is inspired by both double Q-learning and double DQN. In double Q-learning, I understand that Q1 and Q2 are independent because they are trained ...
3
votes
0answers
85 views

What is a generalized MDP?

What is a generalized MDP? How is it different than a "regular" MDP? How does it generalise the notion of an MDP? Why do we need a generalised MDP? Do generalised MDPs have some practical usefulness ...
3
votes
0answers
49 views

Negative counterfactual regret

I am reading the paper Regret Minimization in Games with Incomplete Information on CFR algorithm. On page 4, the paper defines $R^{T,+}_{i,\text{imm}}=\max\{R^{T}_{i,\text{imm}}, 0\}$ after equation (...
3
votes
1answer
101 views

Can A3C update the policy / critic on a local machine without needing to copy?

To make A2C into A3C you make it asynchronous. From what I understand the 'correct' way to do that is to thread off workers with a copy of the policy and critic, and then return the state/action/...
3
votes
0answers
49 views

Choosing more than one action in a parameterized policy

I would like to implement a variant of policy iteration that can choose one or more actions in each state. An example would be to heal and move in the game of Doom. Parameterizing the power set of ...
3
votes
0answers
94 views

Can I use deterministic policy gradient methods for stochastic policy learning?

Can I treat a stochastic policy (over a finite action space of size $n$) as a deterministic policy (in the set of probability distribution in $\mathbb{R}^n$)? It seems to me that nothing is broken ...
3
votes
0answers
112 views

Why are all the actions converging to the same index?

I am using PPO with an LSTM agent. My agent is performing 10 actions for each episode, one action is corresponding to one LSTM timestep and the action space is discrete. I have only one reward per ...
3
votes
0answers
162 views

Dyna-Q algorithm, having trouble when adding the simulated experiences

I'm trying to create a simple Dyna-Q agent to solve small mazes, in python. For the Q function, Q(s, a), I'm just using a matrix, where each row is for a state value, and each column is for one of the ...
3
votes
0answers
198 views

Understanding multi iteration update of model in Policy Gradient PPO algorithm

I have a general question about the updating of the network/model in the PPO algorithm. If I understand it correctly, there are multiple iterations of weight updates done on the model with data that ...
3
votes
1answer
110 views

How can I design a hierarchy of agents each of which with different goals?

I read some light material earlier about the possibility of building a hierarchy of agents, where the agents at the leaves solve primitive tasks while higher-level agents are optimized for ...
3
votes
0answers
391 views

RL to generate sentences

I want to develop a system to generate grammatically correct sentences. The input would be some words. The output would be a grammatically correct human-like sentence. Eg: Input: capital, Paris, ...
2
votes
0answers
26 views

Problems with gradient-biased actor critic methods

To my knowledge, there are at least 6 different variants of Actor Critic: \begin{array}{l l l l} \text{actor gradient} & \text{critic gradient} & \text{actor gradient biased} & \text{name} ...
2
votes
0answers
27 views

What exactly does meta-learning in reinforcement learning setting mean?

We can use DDPG to train agents to stack objects. And stacking objects can be viewed as first grasping followed by pick and place. In this context, how does meta-reinforcement learning fit? Does it ...
2
votes
0answers
30 views

What is the return-to-go in reinforcement learning?

In reinforcement learning, the return is defined as some function of the rewards. For example, you can have the discounted return, where you multiply the rewards received at later time steps by ...
2
votes
0answers
32 views

Why is it necessary to divide the priority range according to the batch size in Prioritized Experience Replay?

According to DeepMinds's paper Prioritized Experience Replay (2016), one should equally divide the priority range $[0, p_\text{total}]$ into $k$ ranges, where $k$ is the size of the batch, and sample ...
2
votes
0answers
13 views

Mapping given probabilities to empirical probabilities

Consider following problem statement: You have given $n$ actions. You can perform any of them. Each action gives you success with some probability. The challenge is to perform given finite number of ...
2
votes
1answer
49 views

What are proxy reward functions?

The understanding I have is that they somehow adjust the objective to make it easier to meet, without changing the reward function. ... the observed proxy reward function is the approximate solution ...
2
votes
0answers
23 views

Is there a UCB type algorithm for linear stochastic bandit with lasso regression?

Why is there no upper confidence bound algorithm for linear stochastic bandits that uses lasso regression in the case that the regression parameters are sparse in the features? In particular, I don't ...
2
votes
1answer
46 views

Are linear approximators better suited to some tasks compared to complex neural net functions?

Model based RL attempts to learn a function $f(s_{t+1}|s_t, a_t)$ representing the environment transitions, otherwise known as a model of the system. I see linear functions are still being used in ...
2
votes
0answers
29 views

Handling a Large Discrete Action Space in Deep Q Learning

I am attempting to solve a timetabling problem using deep Q learning. It could be thought of as a resource allocation problem to obtain some certificate of 'optimality'. However, how to define and ...
2
votes
0answers
32 views

What's an example of a simple policy but a complex value function?

Hado van Hasselt, a researcher at DeepMind, mentioned in one of his videos (from 7:20 to 8:20) on Youtube (about policy gradient methods) that there are cases when the policy is very simple compared ...
2
votes
0answers
24 views

Combine DQN with the Average Reward setting

I have to deal with a non-episodic task, where there is addittionally a continuous state space and more specifically in each time step there is always a new state that has never been seen before. I ...
2
votes
0answers
24 views

Do we assume the policy to be deterministic when proving the optimality?

In reinforcement learning, when we talk about the principle of optimality, do we assume the policy to be deterministic?
2
votes
1answer
71 views

How to compute the target for double Q-learning update step?

I've already read the original paper about double DQN but I do not find a clear and practical explanation of how the target $y$ is computed, so here's how I interpreted the method (let's say I have 3 ...
2
votes
0answers
54 views

Should I use the discounted average reward as objective in a finite-horizon problem?

I am new to reinforcement learning, but, for a finite horizon application problem, I am considering using the average reward instead of the sum of rewards as the objective. Specifically, there are a ...
2
votes
0answers
15 views

Alternatives to Hierarchical RL for centralized control tasks?

Consider a problem where the agent must learn to control a hierarchy of agents acting against another such agent in a competitive environment. The agents on each team need to learn cooperate in order ...
2
votes
0answers
62 views

Most of state-action pairs remain unvisited in the q-table

In building my first Q-learning algorithm for OpenAI gym's CartPole problem, many of my states remain unvisited. I believe it is the reason that my agent does not learn. Can I be told of the reasons I ...
2
votes
0answers
56 views

If the performance of an RL agent in a partially observable environment is “good”, is this likely only accidental?

In my research, I remember to have read that, in case of an environment which can be modeled by partially observable MDP, there are no convergence guarantees (unfortunately, I do not find the paper ...
2
votes
0answers
150 views

Why is DDPG not learning and it does not converge?

I have used a different setting, but DDPG is not learning and it does not converge. I have used these codes 1,2, and 3 and I used different optimizers, activation functions, and learning rate but ...
2
votes
0answers
48 views

What kind of policy evaluation and policy improvement AlphaGo, AlphaGo Zero and AlphaZero are using

I'm trying to find out what kind of policy improvement and policy evaluation AlphaGo, AlphaGo Zero, and AlphaZero are using. By looking into their respective paper and SI, I can conclude that it is a ...
2
votes
0answers
50 views

Prioritised Remembering in Experience Replay (Q-Learning)

I'm using Experience Replay based on the original Prioritized Experience Replay (PER) paper. In the paper authors show ~ an order of magnitude increase in data efficiency from prioritized sampling. ...
2
votes
0answers
43 views

How can I formulate a prediction problem (given labeled data) as an RL problem and solve it with Q-learning?

One of my friends sent me a problem he was working on lately, and I couldn't help but I wonder how could it be solved using Q-learning. The statement is as follows: Given the following datasets, the ...
2
votes
1answer
50 views

Should forecasting with neural networks only be treated as a supervised learning (regression) problem?

I have recently made a work about the application of neural networks to time series forecasting, and I treated this as a supervised learning (regression) problem. I have come across the suggestion of ...
2
votes
1answer
57 views

Why does shifting all the rewards have a different impact on the performance of the agent?

I am new to reinforcement learning. For my application, I have found out that if my reward function contains some negative and positive values, my model does not give the optimal solution, but the ...
2
votes
0answers
27 views

Why is it the case that off - policy evaluation using importance sampling suffers from high variance?

The average return for trajectories, $V^{\pi_e}$(s) is often computed via the importance sampling estimate $$V^{\pi_e}(s) = \frac{1}{n}\sum_{i=1}^n\prod_{t=0}^{H}\frac{\pi_e(a_t | s_t)}{\pi_b(a_t|s_t)}...
2
votes
0answers
70 views

Proof of Maximization Bias in Q-learning?

In the textbook "Reinforcement Learning: An Introduction" by Richard Sutton and Andrew Barto, the concept of Maximization Bias is introduced in section 6.7, and how Q-learning "over-estimates" action-...
2
votes
0answers
80 views

What is the proof that “reward-to-go” reduces variance of policy gradient?

I am following the OpenAI's spinning up tutorial Part 3: Intro to Policy Optimization. It is mentioned there that the reward-to-go reduces the variance of the policy gradient. While I understand the ...
2
votes
0answers
41 views

If the minimum Q value is decreasing and the maximum Q value increasing, is this a sign that dueling double DQN is diverging?

I'm training a dueling double DQN agent with prioritized replay buffer and notice that the min Q values are decreasing, while the max Q values are increasing. Is this a sign that it is diverging? ...
2
votes
0answers
37 views

Are the final states not being updated in this $n$-step Q-Learning algorithm?

I am reading this paper and in algorithm 3 they describe an $n$-step Q-Learning algorithm. Below is the pseudo-code. From this pseudo-code, it looks as though the final tuples that they would ...
2
votes
0answers
31 views

Can you find another reason for sample inefficiency of model-free on-policy Deep Reinforcement Learning?

The following mindmap gives an overview of multiple reasons for sample inefficiency. The list is definitely not complete. Can you see another reason not mentioned so far? Some related links: ...
2
votes
0answers
47 views

Can weighted importance sampling be applied to off-policy evaluation for continuous state space MDPs?

Can weighted importance sampling (WIS) and importance sampling (IS) be applied to off-policy evaluation for continuous state spaces MDPs? Given that I have trajectories of $(s_t,a_t)$ pairs and the ...

1
2
3 4 5
11