Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

639 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
2
votes
0answers
24 views

Are there any known disadvantages of implementing vanilla Q-learning on a discretized-state space environment?

For an RL problem on a continuous state space, the states could be discretized into buckets and these buckets used in implementing the Q-table. I see that is what is done here. However, according to ...
2
votes
0answers
74 views

Is (log-)standard deviation learned in TRPO and PPO or fixed instead?

After having read Williams (1992), where it was suggested that actually both the mean and standard deviation can be learned while training a REINFORCE algorithm on generating continuous output values, ...
2
votes
0answers
39 views

What trait of a planning problem makes reinforcement learning a well suited solution?

Planning problems have been the first problems studied at the dawn of AI (Shakey the robot). Graph search (e.g. A*) and planning (e.g. GraphPlan) algorithms can be very efficient at generating a plan. ...
2
votes
0answers
57 views

What is the dimensionality of these derivatives in the paper “Active Learning for Reward Estimation in Inverse Reinforcement Learning”?

I'm trying to implement in code part of the following paper: Active Learning for Reward Estimation in Inverse Reinforcement Learning. I'm specifically referring to section 2.3 of the paper. Let's ...
2
votes
0answers
51 views

Reinforcement learning and Graph Neural Networks: Entropy drops to zero

I am currently working on an experiment to link reinforcement learning with graph neural networks. This is my architecture: Feature Extraction with GCN: there is a fully meshed topology with ...
2
votes
0answers
28 views

Is better to reward short- or long-term progress in Q-learning?

I have been training some kind of agent to reach a target using a Q-learning based approach, and I have tried two different types of rewards: Long-term reward: $\mathrm{reward} = - \mathrm{distance}(\...
2
votes
0answers
58 views

What place do Agent Communications Language have in Multi-Agent Systems nowadays?

I am currently working on implementing a Multi-Agent System for Smart Grids. There's a lot of literature for that and some things confuse me. I have read that there is FIPA, which aimed to create a ...
2
votes
0answers
20 views

Can DQN outperform DoubleDQN?

I found a similar post about this issue, but unfortunately I did not find a proper answer. Are there any references where DQN is better than DoubleDQN, that is DoubleDQN does not improve DQN ?
2
votes
0answers
27 views

Does there necessarily exist “dominated actions” in a MDP?

In a Markov Decision Process, is it possible that there exists no "dominated action"? I define a dominated action the following way: we say that $(s,a)$ is a dominated action, if $\forall \...
2
votes
0answers
48 views

How does bootstrapping work with the offline $\lambda$-return algorithm?

In Barton and Sutton's book, Reinforcement Learning: An Introduction (2nd edition), an expression, on page 289 (equation 12.2), introduced the form of the $\lambda$-return defined as follows $$G_t^{\...
2
votes
0answers
69 views

$\nabla \log \pi$ with respect to some parameters constantly being zero

I am new to reinforcement learning. May I ask a simple (and maybe a bit silly) question here? I am trying to use the "one-step actor-critic" method to train a robot on a gridworld. Let's ...
2
votes
0answers
78 views

Policy gradient: Does it use the Markov property?

To derive the policy gradient, we start by writing the equation for the probability of a certain trajectory (e.g. see spinningup tutorial): $$ \begin{align} P_\theta(\tau) &= P_\theta(s_0, a_0, ...
2
votes
0answers
37 views

How does uniform offset tiling work with function approximation?

I get the fundamental idea of how tilings work, but, in Barton and Sutton's book, Reinforcement Learning: An Introduction (2nd edition), a diagram, on page 219 (figure 9.11), showing the variations of ...
2
votes
0answers
50 views

Why is TD(0) not converging to the optimal policy?

I am trying to implement the basic RL algorithms to learn on this 10x10 GridWorld (from REINFORCEJS by Kaparthy). Currently I am stuck at TD(0). No matter how many episodes I run, when I am updating ...
2
votes
0answers
51 views

Can $Q$-learning or SARSA be thought of a Markov Chain?

I might just be overthinking a very simple question but nonetheless the following has been bugging me a lot. Given an MDP with non-trivial state and action sets, we can implement the SARSA algorithm ...
2
votes
0answers
20 views

Understanding neural network achitectures in policy gradient reinforcement learning for continuous state and action space

I am trying to train a neural network using reinforcement learning / policy gradient methods. The states, i.e. the inputs, as well as the actions I am trying to sample are vectors with each element ...
2
votes
0answers
22 views

How can I discourage the RL agent from drawing in a zero-sum game?

My agent receives $1, 0, -1$ rewards for winning, drawing, and losing the game, respectively. What would be the consequences of setting reward to $-1$ for draws? Would that encourage the agent to win ...
2
votes
0answers
28 views

What is the appropriate way of passing a list of integers that represents the environment to a neural network's dense layer?

I'm training an RL agent using the DQN algorithm to do a specific task. The environment is represented by a list of $10$ integer numbers from $0$ to $20$. An example would be $[5, 15, 8, 8, 0, \dots]$....
2
votes
0answers
42 views

Is Deep SARSA learning a feasible approach?

I noticed that SARSA has been rarely used in the deep RL setting. Usually, the training for DQN is done off-policy. I think one of the major reasons for this is due to greater sample efficiency in ...
2
votes
0answers
49 views

Problems with gradient-biased actor critic methods

To my knowledge, there are at least 6 different variants of Actor Critic: \begin{array}{l l l l} \text{actor gradient} & \text{critic gradient} & \text{actor gradient biased} & \text{name} ...
2
votes
0answers
36 views

What exactly does meta-learning in reinforcement learning setting mean?

We can use DDPG to train agents to stack objects. And stacking objects can be viewed as first grasping followed by pick and place. In this context, how does meta-reinforcement learning fit? Does it ...
2
votes
0answers
92 views

What is the return-to-go in reinforcement learning?

In reinforcement learning, the return is defined as some function of the rewards. For example, you can have the discounted return, where you multiply the rewards received at later time steps by ...
2
votes
1answer
49 views

What are the strategies for computationally heavy environments or long-time waiting environments?

I have an environment that is computationally heavy (takes several seconds to get a reward and next state). This limits reinforcement capability, due to poor sampling of the problem. There is any ...
2
votes
0answers
25 views

Is there a UCB type algorithm for linear stochastic bandit with lasso regression?

Why is there no upper confidence bound algorithm for linear stochastic bandits that uses lasso regression in the case that the regression parameters are sparse in the features? In particular, I don't ...
2
votes
0answers
102 views

Use of virtual worlds (e.g. Second Life) for training Artificial General Intelligence agents?

There is emerging effort for Third Wave Artificial Intelligence (Artificial General Intelligence) (http://hlc.doc.ic.ac.uk/3AI_HLC_2019.html and https://www.darpa.mil/work-with-us/ai-next-campaign) ...
2
votes
0answers
48 views

Handling a Large Discrete Action Space in Deep Q Learning

I am attempting to solve a timetabling problem using deep Q learning. It could be thought of as a resource allocation problem to obtain some certificate of 'optimality'. However, how to define and ...
2
votes
0answers
34 views

What's an example of a simple policy but a complex value function?

Hado van Hasselt, a researcher at DeepMind, mentioned in one of his videos (from 7:20 to 8:20) on Youtube (about policy gradient methods) that there are cases when the policy is very simple compared ...
2
votes
0answers
37 views

Combine DQN with the Average Reward setting

I have to deal with a non-episodic task, where there is addittionally a continuous state space and more specifically in each time step there is always a new state that has never been seen before. I ...
2
votes
0answers
25 views

Do we assume the policy to be deterministic when proving the optimality?

In reinforcement learning, when we talk about the principle of optimality, do we assume the policy to be deterministic?
2
votes
0answers
63 views

Should I use the discounted average reward as objective in a finite-horizon problem?

I am new to reinforcement learning, but, for a finite horizon application problem, I am considering using the average reward instead of the sum of rewards as the objective. Specifically, there are a ...
2
votes
0answers
18 views

Alternatives to Hierarchical RL for centralized control tasks?

Consider a problem where the agent must learn to control a hierarchy of agents acting against another such agent in a competitive environment. The agents on each team need to learn cooperate in order ...
2
votes
0answers
40 views

How to combine two differently equally important signals into the reward function, that have different scales?

I have two signals that I want to use to model my reward. The first one is the CPU TIME: running mean from this diagram: The second one is the MAX RESIDUAL from this diagram: Since they are both ...
2
votes
0answers
70 views

Most of state-action pairs remain unvisited in the q-table

In building my first Q-learning algorithm for OpenAI gym's CartPole problem, many of my states remain unvisited. I believe it is the reason that my agent does not learn. Can I be told of the reasons I ...
2
votes
0answers
56 views

If the performance of an RL agent in a partially observable environment is “good”, is this likely only accidental?

In my research, I remember to have read that, in case of an environment which can be modeled by partially observable MDP, there are no convergence guarantees (unfortunately, I do not find the paper ...
2
votes
0answers
389 views

Why is DDPG not learning and it does not converge?

I have used a different setting, but DDPG is not learning and it does not converge. I have used these codes 1,2, and 3 and I used different optimizers, activation functions, and learning rate but ...
2
votes
0answers
68 views

What kind of policy evaluation and policy improvement AlphaGo, AlphaGo Zero and AlphaZero are using

I'm trying to find out what kind of policy improvement and policy evaluation AlphaGo, AlphaGo Zero, and AlphaZero are using. By looking into their respective paper and SI, I can conclude that it is a ...
2
votes
0answers
52 views

Prioritised Remembering in Experience Replay (Q-Learning)

I'm using Experience Replay based on the original Prioritized Experience Replay (PER) paper. In the paper authors show ~ an order of magnitude increase in data efficiency from prioritized sampling. ...
2
votes
0answers
43 views

How can I formulate a prediction problem (given labeled data) as an RL problem and solve it with Q-learning?

One of my friends sent me a problem he was working on lately, and I couldn't help but I wonder how could it be solved using Q-learning. The statement is as follows: Given the following datasets, the ...
2
votes
0answers
139 views

What is the proof that “reward-to-go” reduces variance of policy gradient?

I am following the OpenAI's spinning up tutorial Part 3: Intro to Policy Optimization. It is mentioned there that the reward-to-go reduces the variance of the policy gradient. While I understand the ...
2
votes
0answers
100 views

If the minimum Q value is decreasing and the maximum Q value increasing, is this a sign that dueling double DQN is diverging?

I'm training a dueling double DQN agent with prioritized replay buffer and notice that the min Q values are decreasing, while the max Q values are increasing. Is this a sign that it is diverging? ...
2
votes
0answers
78 views

How can the target rely on untrained parameters?

I'm trying to understand DQN. I understand where the loss function comes from. I'm just unsure about why the target function works in practice. Given the loss function $$ L_i(\theta_i) = [(y_i - Q(s,a;...
2
votes
0answers
48 views

Are the final states not being updated in this $n$-step Q-Learning algorithm?

I am reading this paper and in algorithm 3 they describe an $n$-step Q-Learning algorithm. Below is the pseudo-code. From this pseudo-code, it looks as though the final tuples that they would ...
2
votes
0answers
81 views

How to prevent deep Q-learning algorithms to overfit?

I have recently solved the Cartpole problem using double deep Q-learning. When I saw how the agent was doing, it used to go right every time, never left, and it did similar actions all the time. Did ...
2
votes
0answers
39 views

Can you find another reason for sample inefficiency of model-free on-policy Deep Reinforcement Learning?

The following mindmap gives an overview of multiple reasons for sample inefficiency. The list is definitely not complete. Can you see another reason not mentioned so far? Some related links: ...
2
votes
0answers
49 views

Can weighted importance sampling be applied to off-policy evaluation for continuous state space MDPs?

Can weighted importance sampling (WIS) and importance sampling (IS) be applied to off-policy evaluation for continuous state spaces MDPs? Given that I have trajectories of $(s_t,a_t)$ pairs and the ...
2
votes
0answers
69 views

Does the concept of validation loss apply to training deep Q networks?

In deep learning, the concept of validation loss is to ensure that the model being trained is not currently overfitting the data. Is there a similar concept of overfitting in deep q learning? Given ...
2
votes
0answers
32 views

Is there an online RL algorithm that receives as input a camera frame and produces an action as output?

I want to build a reinforcement learning model, which takes a camera picture as input, that learns online (in terms of machine learning). Based on the position of an object on the camera, I want the ...
2
votes
0answers
66 views

Can we use a Gaussian process to approximate the belief distribution at every instant in a POMDP?

Suppose $x_{t+1} \sim \mathbb{P}(\cdot | x_t, a_t)$ denotes the state transition dynamics in a reinforcement learning (RL) problem. Let $y_{t+1} = \mathbb{P}(\cdot | x_{t+1})$ denote the noisy ...
2
votes
1answer
86 views

Do smaller loss values during DQN training produce better policies?

During the training of DQN, I noticed that the model with prioritized experience replay (PER) had a smaller loss in general compared to a DQN without PER. The mean squared loss was an order of ...
2
votes
0answers
45 views

When to do discretization to decrease the state/action space in RL?

When to do discretization to decrease the state/action space in RL? Can you give me some references that such a technique is used?

1 2
3
4 5
13