Skip to main content

Questions tagged [reward-functions]

For questions about rewards functions (e.g. in the context of reinforcement learning, which may be denoted as $R(s, a)$).

Filter by
Sorted by
Tagged with
7 votes
2 answers
2k views

How are the reward functions $R(s)$, $R(s, a)$ and $R(s, a, s')$ equivalent?

In this video, the lecturer states that $R(s)$, $R(s, a)$ and $R(s, a, s')$ are equivalent representations of the reward function. Intuitively, this is the case, according to the same lecturer, ...
nbro's user avatar
  • 40.9k
8 votes
2 answers
3k views

How do we define the reward function for an environment?

How do you actually decide what reward value to give for each action in a given state for an environment? Is this purely experimental and down to the programmer of the environment? So, is it a ...
Hazzaldo's user avatar
  • 289
8 votes
2 answers
11k views

What are some best practices when trying to design a reward function?

Generally speaking, is there a best-practice procedure to follow when trying to define a reward function for a reinforcement-learning agent? What common pitfalls are there when defining the reward ...
12 rhombi in grid w no corners's user avatar
2 votes
2 answers
564 views

How should I define the reward function to solve the Wumpus game with deep Q-learning?

I'm writing a DQN agent for the Wumpus game. Is the reward function to train the Q-networks (target network and policy) the same as the score of the game, i.e. +1000 for picking up gold, -1000 for ...
Edwin Carlsson's user avatar
12 votes
4 answers
2k views

Counterexamples to the reward hypothesis

On Sutton and Barto's RL book, the reward hypothesis is stated as that all of what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative ...
Bananin's user avatar
  • 221
3 votes
1 answer
398 views

How can I implement the reward function for an 8-DOF robot arm with TRPO?

I need to get an 8-DOF (degrees of freedom) robot arm to move a specified point. I need to implement the TRPO RL code using OpenAI gym. I already have the gazebo environment. But I am unsure of how to ...
user1690356's user avatar
7 votes
1 answer
4k views

Why does a negative reward for every step really encourage the agent to reach the goal as quickly as possible?

If we shift the rewards by any constant (which is a type of reward shaping), the optimal state-action value function (and so optimal policy) does not change. The proof of this fact can be found here. ...
nbro's user avatar
  • 40.9k
5 votes
2 answers
1k views

Why does the definition of the reward function $r(s, a, s')$ involve the term $p(s' \mid s, a)$?

Sutton and Barto define the state–action–next-state reward function, $r(s, a, s')$, as follows (equation 3.6, p. 49) $$ r(s, a, s^{\prime}) \doteq \mathbb{E}\left[R_{t} \mid S_{t-1}=s, A_{t-1}=a, S_{t}...
SAGALPREET SINGH's user avatar
2 votes
1 answer
192 views

Why is the equation $r(s', a, s') =\sum_{r \in \mathcal{R}} r \frac{p\left(s^{\prime}, r \mid s, a\right)}{p\left(s^{\prime} \mid s, a\right)}$true?

I am referring to eq. 3.6 (page 49) based on Sutton's online book and can be found in an image below. I could not make sense of the final derivation of the equation $r(s, a, s')$. My question is ...
alfa_80's user avatar
  • 123
0 votes
1 answer
304 views

How would you shape a reward function if there was four quantities to optimize?

I found this article quite useful on how to shape a reward function in RL. However, the example they gave is quite simple, where the goal is to minimize only two quantities (velocity and distance). ...
BAKYAC's user avatar
  • 36
13 votes
3 answers
2k views

Why is the reward in reinforcement learning always a scalar?

I'm reading Reinforcement Learning by Sutton & Barto, and in section 3.2 they state that the reward in a Markov decision process is always a scalar real number. At the same time, I've heard about ...
Sid Mani's user avatar
  • 233
8 votes
1 answer
3k views

What are other ways of handling invalid actions in scenarios where all rewards are either 0 (best reward) or negative?

I created an OpenAI Gym environment, and I would like to check the performance of the agent from OpenAI Baselines DQN approach on it. In my environment, the best possible outcome for the agent is 0 - ...
AlexGuevara's user avatar
8 votes
1 answer
7k views

Suitable reward function for trading buy and sell orders

I am working to build a deep reinforcement learning agent which can place orders (i.e. limit buy and limit sell orders). The actions are ...
fgauth's user avatar
  • 189
6 votes
2 answers
2k views

Why does shifting all the rewards have a different impact on the performance of the agent?

I am new to reinforcement learning. For my application, I have found out that if my reward function contains some negative and positive values, my model does not give the optimal solution, but the ...
Fishfish's user avatar
4 votes
1 answer
2k views

Is a reward given at every step or only given when the RL agent fails or succeeds?

In reinforcement learning, an agent can receive a positive reward for correct actions and a negative reward for wrong actions, but does the agent also receive rewards for every other step/action?
Dan D.'s user avatar
  • 1,293
4 votes
2 answers
1k views

How to apply Q-learning when rewards is only available at the last state?

I have a scheduling problem in which there are $n$ slots and $m$ clients. I am trying to solve the problem using Q-learning so I have made the following state-action model. A state $s_t$ is given by ...
zdm's user avatar
  • 301
2 votes
0 answers
271 views

How to solve a reinforcement learning problem with changing rewards?

I'm working on a problem with non-stationary environments. The state space is discrete and limited. The action is limited too. But the reward for the same action $a$ can change. Even the reward for ...
Zhenzhen Gong's user avatar
1 vote
1 answer
94 views

In addition to the reward function, which other functions do I need to implement Q-learning?

In general, $Q$ function is defined as $$Q : S \times A \rightarrow \mathbb{R}$$ $$Q(s_t,a_t) = Q(s_t,a_t) + \alpha[r_{t+1} + \gamma \max\limits_{a} Q(s_{t+1},a) - Q(s_t,a_t)] $$ $\alpha$ and $\gamma$...
satya's user avatar
  • 187