Questions tagged [rewards]

For questions related to the concept of reward and reward functions (e.g. in the context of reinforcement learning and Markov decision processes).

Filter by
Sorted by
Tagged with
2
votes
0answers
48 views

Should I use the discounted average reward as objective in a finite-horizon problem?

I am new to reinforcement learning, but, for a finite horizon application problem, I am considering using the average reward instead of the sum of rewards as the objective. Specifically, there are a ...
4
votes
2answers
79 views

How can we prevent AGI from doing drugs?

I recently read some introductions to AI alignment, AIXI and decision theory things. As far as I understood, one of the main problems in AI alignment is how to define a utility function well, not ...
0
votes
0answers
24 views

How to combine two differently equally important signals into the reward function, that have different scales?

I have two signals that I want to use to model my reward. The first one is the CPU TIME: running mean from this diagram: The second one is the MAX RESIDUAL from this diagram: Since they are both ...
5
votes
1answer
55 views

Why is the reward in reinforcement learning always a scalar?

I'm reading Reinforcement Learning by Sutton & Barto, and in section 3.2 they state that the reward in a Markov decision process is always a scalar real number. At the same time, I've heard about ...
3
votes
2answers
96 views

What are some best practices when trying to design a reward function?

Generally speaking, is there a best-practice procedure to follow when trying to define a reward function for a reinforcement-learning agent? What common pitfalls are there when defining the reward ...
2
votes
1answer
68 views

Is reinforcement learning reward set for step by step, or the whole sequence until failure?

Reinforcement Learning may start with no data, and the agent receives rewards for correct actions. Are the rewards given out step by step, or only until the agent fails then the reward is a negative ...
4
votes
3answers
243 views

Upper limit to the maximum cumulative reward in a deep reinforcement learning problem

Is there an upper limit to the maximum cumulative reward in a deep reinforcement learning problem? For example you want to train an DQN agent in an environment and you want to know what is the highest ...
2
votes
1answer
89 views

Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards?

Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards? Would it not make more sense to compute $\mathbb{E}(R \mid s, a)$ (the expected return for taking ...
1
vote
1answer
54 views

Can rewards be decomposed into components?

I'm training a robot to walk to a specific $(x, y)$ point using TD3, and, for simplicity, I have something like ...
2
votes
1answer
80 views

How is the reward in reinforcement learning different from the label in supervised learning problems?

How is the notion of immediate reward used in the reinforcement learning different from the notion of a label we find in the supervised learning problems?
2
votes
1answer
50 views

Why does shifting all the rewards have a different impact on the performance of the agent?

I am new to reinforcement learning. For my application, I have found out that if my reward function contains some negative and positive values, my model does not give the optimal solution, but the ...
1
vote
2answers
53 views

How do I calculate the return given the discount factor and a sequence of rewards?

I know that $G_t = R_{t+1} + G_{t+1}$. Suppose $\gamma = 0.9$ and the reward sequence is $R_1 = 2$ followed by an infinite sequence of $7$s. What is the value of $G_0$? As it's infinite, how can we ...
2
votes
1answer
110 views

Non-differentiable reward function to update a neural network

In Reinforcement Learning, when reward function is not differentiable, a policy gradient algorithm is used to update the weights of a network. In the paper Neural Architecture Search with ...
1
vote
1answer
43 views

When discounted MAB is useful?

Many of multi-armed bandit algorithms are used when the total reward is the sum of all rewards. However, in RL, the discounted reward is mainly used. Why the discounted reward is not prevailing in MAB ...
3
votes
2answers
68 views

Is there any difference between reward and return in reinforcement learning?

I am reading Sutton and Barto's book on reinforcement learning. I thought that reward and return were the same things. However, in Section 5.6 of the book, 3rd line, first paragraph, it is written: ...
2
votes
1answer
42 views

Can the agent wait until the end of the episode to determine the reward in SARSA?

From Sutton and Barto's book Reinforcement Learning (Adaptive Computation and Machine Learning series) (p. 99), the following definition for first-visit MC prediction, for estimating $V \sim V_\pi$ is ...
4
votes
1answer
95 views

How do I convert an MDP with the reward function in the form $R(s,a,s')$ to and an MDP with a reward function in the form $R(s,a)$?

The AIMA book has an exercise about showing that an MDP with rewards of the form $r(s, a, s')$ can be converted to an MDP with rewards $r(s, a)$, and to an MDP with rewards $r(s)$ with equivalent ...
1
vote
0answers
33 views

State-of-the-art algorithms not working on a custom RL environment

I'm trying to train a RL agent on a custom, highly stochastic environment (MDP). In order to do so I'm using existing implementations of state-of-the-art RL algorithms as provided by Stable Baselines. ...
1
vote
1answer
53 views

How do you know if an agent has learnt its environment in reinforcement learning?

I'm new to reinforcement learning and trying to understand it. If you train an agent using a reinforcement learning algorithm (discrete or continuous) on an environment (real or simulated), then how ...
2
votes
1answer
38 views

Shouldn't expected return be calculated for some faraway time in the future $t+n$ instead of current time $t$?

I am learning RL for the first time. It may be naive, but it is a bit odd to grasp this idea that, if the goal of RL is to maximize the expected return, then shouldn't the expected return be ...
1
vote
1answer
46 views

Which reward function works for recommendation systems using knowledge graphs?

I've been reading this paper on recommendation systems using reinforcement learning (RL) and knowledge graphs (KGs). To give some background, the graph has several (finitely many) entities, of which ...
3
votes
1answer
58 views

Reward Function for Racing Game

I'm busy working on a project where I'm building an agent for a racing game. In this game is a randomised map where there are speed boosts for the player to pick up and obstacles that act to slow the ...
3
votes
1answer
48 views

Appropriate algorithm for RL problem with sparse rewards, continuous actions and significant stochasticity

I'm working on a RL problem with the following properties: The rewards are extremely sparse i.e. all rewards are 0 except the terminal non-zero reward. Ideally I would not use any reward engineering ...
1
vote
1answer
38 views

Can optimizing for immediate reward result in a policy maximizing the return?

The goal of a reinforcement learning agent is to maximize the expected return which is often a discounted sum of future rewards. The return indeed is a very noisy random variable as future rewards ...
2
votes
0answers
29 views

How should I design a reward function for a NLP problem where two models interoperate?

I would like to design a reward function. I am training two models from the first model that classify set of texts (paragraphs and keywords) and I also got some hidden states. The second model is ...
0
votes
1answer
39 views

What is the best measurement for how good an action of a reinforcement learning agent really is?

Even when we get a valuable reward signal after every single action, this immediate reward only approximates the short term goodness of the action. To consider the long term effect of an action, we ...
3
votes
2answers
158 views

Formula for expected rewards for state–action–next-state triples as a three-argument function

While reading about reinforcement learning, I have come across the following expression for expected rewards in terms of a summation, the denominator of which I am not able to account for. The ...
3
votes
0answers
31 views

How does normalization of the inputs work in the context of PPO?

What does the normalization of the inputs mean in the context of PPO? At each time step of an episode, I only know the values of this time step and of the previous ones, if I take track of them. This ...
2
votes
0answers
33 views

How was the DQN trained to play many games?

Some people claim that DQN was used to play many Atari games. But what actually happened? Was DQN trained only once (with some data from all games) or was it trained separately for each game? What was ...
1
vote
1answer
99 views

In RL, if I assign the rewards for better positional play, the algorithm is learning nothing?

I'm creating an RL application for the game Connect Four. If I tell the algorithm which moves/token positions will receive greater rewards, surely it's not actually learning anything; it's just a ...
1
vote
1answer
50 views

What is the relationship between the reward function and the value function?

To clarify it in my head, the value function calculates how 'good' it is to be in a certain state by summing all future (discounted) rewards, while the reward function is what the value function uses ...
2
votes
0answers
24 views

How should I define the reward function for the Connect Four game?

I'm creating an RL application for the game Connect Four. I've researched the different strategies for the game and which positions are more favourable to lead to a win. Should I be assigning ...
1
vote
0answers
29 views

How can I find the appropriate reward value for my reinforcement learning problem?

I am wondering how can I find the appropriate reward value for each specific problem. I know this is a highly empirical process, but I am sure that the value is not set totally at random. I want to ...
3
votes
1answer
46 views

Do all expert trajectories have the same starting state in apprenticeship learning?

In the apprenticeship learning algorithm described by Ng et al. in Apprenticeship Learning via Inverse Reinforcement Learning, they mention that expert trajectories come in the form of $\{s_0^i, s_1^i\...
3
votes
0answers
75 views

What are the guidelines for defining a reward function in reinforcement learning (bandit problem)?

I'm working currently on a problem and I'm using RL (bandit problem). In my system, I have an agent that chooses an action among $k$ possible actions, and a user that decides whether the agent ...
3
votes
0answers
43 views

Is this a good approach to solving Atari's “Montezuma's Revenge”?

I'm new to Reinforcement Learning. For an internship, I am currently training Atari's "Montezuma's Revenge" using a double Deep Q-Network with Hindsight Experience Replay (HER). HER is supposed to ...
3
votes
1answer
51 views

How does the optimization process in hindsight experience replay exactly work?

I was reading the following research paper Hindsight Experience Replay. This is the paper that introduces a concept called Hindsight Experience Replay (HER), which basically attempts to alleviate the ...
0
votes
1answer
42 views

How to incentivise snake to go straight to apple?

I have made a Deep Q Network for the game snake but unfortunately, the snake exhibits some unwanted behavior. It generally does quite well but sometimes it gets stuck in an infinite loop that it can't ...
1
vote
0answers
48 views

Which model should I choose to maximise reward of having chosen two numbers from a list?

I am looking for a technique to train a machine learning model to choose two items from a list. So, given a list $x=[x_1, x_2, x_3, x_4, \dots, x_n]$, the model needs to choose two elements $(x_i, ...
1
vote
0answers
23 views

Does apprenticeship learning require prospective data?

I am thinking of applying apprenticeship learning on retrospective data. From looking at this paper by Ng https://ai.stanford.edu/~ang/papers/icml04-apprentice.pdf which talks about apprenticeship ...
2
votes
2answers
89 views

Is there a good ratio between the positive and negative rewards in reinforcement learning?

Is there an ideal ratio in reinforcement learning between the positive and negative rewards? Suppose I have the scenario of moving a robot across the river. There are two options, walk across the ...
2
votes
1answer
67 views

Can recovering a reward function using IRL lead to better policies compared to reward shaping?

I am working on a research project about the different reward functions being used in the RL domain. I have read up on Inverse Reinforcement Learning (IRL) and Reward Shaping (RS). I would like to ...
0
votes
0answers
23 views

Curiosity Driven Learning affect optimal policy

I am trying to understand some of the different approaches used to overcome sparse rewards in a reinforcement learning setting for a research project. Particularly, I have looked at curiosity driven ...
2
votes
1answer
54 views

Immediate reward received in Atari game using DQN

I am trying to understand the different reward functions modelled in a reinforcement learning problem. I want to be able to know how the temporal credit assignment problem, (where the reward is ...
2
votes
1answer
74 views

Reinforcement Learning Continuous Control (DDPG): How to avoid thrashing of issued actions? How to reward smooth output over flittering?

Currently I'm working on a continuous state / continuous action controller. It shall control a certain roll angle of an aircraft by issuing the correct aileron commands (between -1...1 continuous). ...
2
votes
2answers
192 views

Simulating successful trajectories in Montezuma's Revenge turns out to be unsuccessful

I have written code in OpenAI's gym to simulate a random playing in Montezuma's Revenge where the agent randomly samples actions from the action space and tries to play the game. A success for me is ...
6
votes
1answer
88 views

Why cannot an AI agent adjust the reward function directly?

In standard Reinforcement Learning the reward function is specified by an AI designer and is external to the AI agent. The agent attempts to find a behaviour that collects higher cumulative discounted ...
1
vote
0answers
35 views

Should an RL agent directly observe the reward?

I am training an A2C reinforcement learning agent in a dense reward environment (where rewards are known and explicit at every timestep). Is it redundant to include the previous reward in the current ...
6
votes
2answers
213 views

Counterexamples to the reward hypothesis

On Sutton and Barto's RL book, the reward hypothesis is stated as that all of what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative ...
4
votes
1answer
107 views

How to assign rewards in a non-Markovian environment?

I am quite new to the Reinforcement Learning domain and I am curious about something. It seems to be the case that the majority of current research assumes Markovian environments, that is, future ...