Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

Filter by
Sorted by
Tagged with
2
votes
1answer
53 views

What are preferences and preference functions in multi-objective reinforcement learning?

In RL (reinforcement learning) or MARL (multi-agent reinforcement learning), we have the usual tuple: (state, action, transition_probabilities, reward, next_state) ...
6
votes
2answers
121 views

Why is the reward in reinforcement learning always a scalar?

I'm reading Reinforcement Learning by Sutton & Barto, and in section 3.2 they state that the reward in a Markov decision process is always a scalar real number. At the same time, I've heard about ...
1
vote
1answer
59 views

What is the relationship between the reward function and the value function?

To clarify it in my head, the value function calculates how 'good' it is to be in a certain state by summing all future (discounted) rewards, while the reward function is what the value function uses ...
2
votes
1answer
211 views

What happens to the optimal value function if the reward is multiplied by a constant?

What happens to the optimal state-action value function, $q_*$ if the reward function is multiplied by a constant $c$? Is the optimal state-action value function also multiplied by such a constant?
3
votes
1answer
55 views

Why is the reward function $\text{reward} = 1/{(\text{cost}+1)^2}$ better than $\text{reward} =1/(\text{cost}+1)$?

I have implemented a simple Q-learning algorithm to minimize a cost function by setting the reward to the inverse of the cost of the action taken by the agent. The algorithm converges nicely, but ...
2
votes
1answer
24 views

How do we derive the expression for average reward setting in continuing tasks?

In the average reward setting we have: $$r(\pi)\doteq \lim_{h\rightarrow\infty}\frac{1}{h}\sum_{t=1}^{h}\mathbb{E}[R_{t}|S_0,A_{0:t-1}\sim\pi]$$ $$r(\pi)\doteq \lim_{t\rightarrow\infty}\mathbb{E}[R_{t}...
2
votes
3answers
1k views

Has anyone been able to solve OpenAI's hardcore bipedal walker with their implementation of DDPG?

As the question suggests, I'm trying to see if I can solve OpenAI's hardcore version of their gym's bipedal walker using OpenAI's DDPG algorithm. Below is a performance graph from my latest attempt, ...
2
votes
2answers
112 views

How to define reward function in POMDPs?

How do I define a reward function for my POMDP model? In the literature, it is common to use one simple number as a reward, but I am not sure if this is really how you define a function. Because this ...
2
votes
1answer
40 views

How do I design the rewards and penalties for an agent whose goal it is to explore a map

I am trying to train an agent to explore an unknown two-dimensional map while avoiding circular obstacles (with varying radii). The agent has control over its steering angle and its speed. The ...
3
votes
1answer
71 views

How should I design the reward function for racing game (where the goal is to reach finishing line before the opponent)?

I'm building an agent for a racing game. In this game, there is a randomized map where there are speed boosts for the player to pick up and obstacles that act to slow the player down. The goal of the ...
2
votes
1answer
310 views

How should I define the reward function in the case of Connect Four?

I'm using RL to train a Network on the game Connect4. It learns quickly that 4 connected pieces is good. It gets a reward of 1 for this. A zero is rewarded for all other moves. It takes quite a time ...
3
votes
2answers
73 views

Why do all states appear identical under the function approximation in the Short Corridor task?

This is the Short Corridor problem taken from the Sutton & Barto book. Here it's written: The problem is difficult because all the states appear identical under the function approximation But ...
3
votes
1answer
49 views

Are there any reliable ways of modifying the reward function to make the rewards less sparse?

If I am training an agent to try and navigate a maze as fast as possible, a simple reward would be something like \begin{align} R(\text{terminal}) &= N - \text{time}\ \ , \ \ N \gg \text{...
3
votes
1answer
60 views

Expressing Arbitrary Reward Functions as Potential-Based Advice (PBA)

I am trying to reproduce the results for the simple grid-world environment in [1]. But it turns out that using a dynamically learned PBA makes the performance worse and I cannot obtain the results ...
3
votes
2answers
1k views

What should I do when the potential value of a state is too high?

I'm working on a Reinforcement Learning task where I use reward shaping as proposed in the paper Policy invariance under reward transformations: Theory and application to reward shaping (1999) by ...
4
votes
0answers
45 views

How define a reward function for a humanoid agent whose goal is to stand up from the ground?

I'm trying to teach a humanoid agent how to stand up after falling. The episode starts with the agent lying on the floor with its back touching the ground, and its goal is to stand up in the shortest ...
6
votes
2answers
368 views

How do we define the reward function for an environment?

How do you actually decide what reward value to give for each action in a given state for an environment? Is this purely experimental and down to the programmer of the environment? So, is it a ...
3
votes
1answer
95 views

How to avoid rapid actuator movements in favor of smooth movements in a continuous space and action space problem?

I'm working on a continuous state / continuous action controller. It shall control a certain roll angle of an aircraft by issuing the correct aileron commands (in $[-1, 1]$). To this end, I use a ...
1
vote
0answers
39 views

Designing a reward function for my reinforcement learning problem

I'm working on a project lately and I'm trying to solve a problem with reinforcement learning and I have serious issues with shaping the reward function. The problem is designing a device with maximum ...
3
votes
2answers
144 views

What are some best practices when trying to design a reward function?

Generally speaking, is there a best-practice procedure to follow when trying to define a reward function for a reinforcement-learning agent? What common pitfalls are there when defining the reward ...
3
votes
0answers
61 views

Simplification of expected reward under the limit in continuous tasks

I was reading the average reward setting for continuous tasks from rich sutton's book (page 202, 2nd edition). There he perform a simplification over the expected reward under the limit approaching to ...
3
votes
1answer
81 views

What is the loss for policy gradients with continuous actions?

I know with policy gradients used in an environment with a discrete action space are updated with $$ \Delta \theta_{t}=\alpha \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) v_{t} $$ ...
7
votes
2answers
4k views

What is the difference between reinforcement learning and optimal control?

Coming from a process (optimal) control background, I have begun studying the field of deep reinforcement learning. Sutton & Barto (2015) state that particularly important (to the writing of the ...
0
votes
0answers
36 views

Variance of the Gaussian policy is not decreasing while training the agent using Soft Actor-Critic method

I've written my own version of SAC(v2) for a problem with continuous action space. While training, the losses for the value network and both q functions steadily decrease down to 0.02-0.03. The loss ...
1
vote
1answer
57 views

Connection between the Bellman equation for the action value function $q_\pi(s,a)$ and expressing $q_\pi(s,a) = q_\pi(s, a,v_\pi(s'))$

When deriving the Bellman equation for $q_\pi(s,a)$, we have $q_\pi(s,a) = E_\pi[G_t | S_t = s, A_t = a] = E_\pi[R_{t+1} + \gamma G_{t+1} | S_t = s, A_t = a]$ (1) This is what is confusing me, at this ...
2
votes
1answer
50 views

What's the difference between estimation and approximation error?

I'm unable to find online, or understand from context - the difference between estimation error and approximation error in the context of machine learning (and, specifically, reinforcement learning). ...
2
votes
0answers
31 views

Why is it necessary to divide the priority range according to the batch size in Prioritized Experience Replay?

According to DeepMinds's paper Prioritized Experience Replay (2016), one should equally divide the priority range $[0, p_\text{total}]$ into $k$ ranges, where $k$ is the size of the batch, and sample ...
5
votes
2answers
792 views

What are the available exploration strategies for continuous action space scenarios in RL?

I'm building a deep neural network to serve as the policy estimator in an actor-critic reinforcement learning algorithm for a continuing (not episodic) case. I'm trying to determine how to explore ...
3
votes
1answer
52 views

Can we stop training as soon as epsilon is small?

I'm new to reinforcement learning. As it is common in RL, $\epsilon$-greedy search for the behavior/exploration is used. So, at the beginning of the training, $\epsilon$ is high, and therefore a lot ...
1
vote
0answers
39 views

What are the disadvantages of actor-only methods with respect to value-based ones?

While the advantages of actor-only algorithms, the ones that search directly the policy without the use of the value function, are clear (possibility of having a continuous action space, a stochastic ...
2
votes
2answers
126 views

What is the gradient of the objective function in the Soft Actor-Critic paper?

In the paper "Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor", they define the loss function for the policy network as $$ J_\pi(\phi)=\mathbb E_{...
1
vote
1answer
49 views

How to best make use of learning rate scheduling in reinforcement learning?

How to best make use of learning rate scheduling in reinforcement learning? To me, a low learning rate towards the end to fine-tune what you've learned with subtle updates makes sense. But I don't ...
0
votes
1answer
29 views

How to let the agent choose how to populate a state space matrix in RL (using python)

I have an agent (drone) that has to allocate subchannels for different types of User Equipment. I have represented the subchannel allocation with a 2-dimentional binary matrix, that is initialized to ...
1
vote
1answer
44 views

Is this ML task possible?

What I want to do is from an Internet challenge to transform any given image into the Polish flag using the available filters and crop tool on the iPhone camera app. Here's an example. There aren't ...
2
votes
1answer
45 views

Adversarial Q Learning should use the same Q Table?

I'm creating a RF Q-Learning agent for a two player fully-observable board game and wondered, if I was to train the Q Table using adversarial training, should I let both 'players' use, and update, the ...
1
vote
1answer
47 views

Why do my rewards reduce after extensive training using D3QN?

I am running a drone simulator for collision avoidance using a slight variant of D3QN. The training is usually costly (runs for at least a week) and I have observed that reward function gradually ...
2
votes
1answer
61 views

How should I take into consideration the number of steps in the reward function?

I am currently implementing the paper Active Object Localization with Deep Reinforcement Learning in Python. While reading about the reward scheme I came across the following: Finally, the proposed ...
0
votes
0answers
13 views

How to design the step() method in a custom-built environment when the action space is a 2-dimentional matrix?

I have an action space that is a matrix. I am struggling with finding how to choose how to take the action when the number of possible actions is huge.
4
votes
1answer
186 views

Why can't we apply value iteration when we do not know the reward and transition functions, and how does Q-learning solve this issue?

I don't understand why we can't apply value iteration when don't know the reward and transition probabilities. In this lecture, the lecturer says it has to do with not being able to take max with ...
3
votes
1answer
312 views

What are the pros and cons of using standard deviation or entropy for exploration in PPO?

When trying to implement my own PPO (Proximal Policy Optimizer), I came across two different implementations : Exploration with std Collect trajectories on $N$ timesteps, by using a policy-centered ...
2
votes
1answer
31 views

Why is regret so defined in MABs?

Consider a multi-armed bandit(MAB). There are $k$ arms, with reward distributions $R_i$ where $1 \leq i \leq k$. Let $\mu_i$ denote the mean of the $i^{th}$ distribution. If we run the multi-armed ...
1
vote
1answer
168 views

How can we estimate the transition model and reward function?

In reinforcement learning (RL), there are model-based and model-free algorithms. In short, model-based algorithms use a transition model (e.g. a probability distribution) and the reward function, even ...
1
vote
2answers
77 views

How does one know that a problem is “model-free” in reinforcement learning?

Consider this slide from a Stanford lecture on reinforcement learning. It states that a model is the agent's representation of how the world changes in response to the agent's action. I've been ...
3
votes
0answers
31 views

Why weighting by lambda that sums to 1 ensures convergence in eligibility trace?

In Sutton and Barto's Book in chapter 12, they state that if weights sum to 1, then an equation's updates have "guaranteed convergence properties". Actually why it ensures convergence? There ...
1
vote
1answer
452 views

How to use the LSTM layer in PPO architecture?

What is the best way of using the LSTM layer in PPO architecture? Should I use them in the first layer of both actor and critic, or use them just before the final layer of these networks? Should I ...
1
vote
0answers
9 views

Relation between a value function of an MDP and a value function of the corresponding latent MDP

In paper "DeepMDP: Learning Continuous Latent Space Models for Representation Learning", Gelada et al. state in the beginning of section 2.4 The degree to which a value function of $\bar{\...
1
vote
1answer
63 views

Reinforcement learning with industrial continuous process

I am new to RL and wish to realize a RL control for an industrial process. The goal is to control the temperature and humidity in a vegetal food production chamber. States: External temperature and ...
2
votes
1answer
53 views

How can I sample the output distribution multiple times when pruning the filters with reinforcement learning?

I was reading the paper Learning to Prune Filters in Convolutional Neural Networks, which is about pruning the CNN filters using reinforcement learning (policy gradient). The paper says that the input ...
2
votes
0answers
13 views

Mapping given probabilities to empirical probabilities

Consider following problem statement: You have given $n$ actions. You can perform any of them. Each action gives you success with some probability. The challenge is to perform given finite number of ...
0
votes
1answer
51 views

What is the search depth of AlphaGo and AlphaGo Zero?

I cannot find reliable sources but someone says it is 40 moves and someone else says it is 50+ moves. I read their papers and they use value function (NN) and policy function to trim the tree, so more ...

1
2
3 4 5
29