4

I believe the claim is true. Here is my attempt at a proof. Let us consider the optimal infinite horizon value function $V_\alpha^*$ of $\mathcal{M}_\alpha$ at an arbitrary state $s \in S$. The value $V_\alpha^*(s)$ is the expected sum of discounted rewards under an optimal policy $\pi_\alpha^*$, i.e., \begin{equation} V_\alpha^*(s) = \mathbb{E}_{\rho_\...


4

If your objective is for the agent to attain some goal (say, reaching a target), then a valid reward function is to assign a reward of 1 when the goal is attained and 0 otherwise. The problem with this reward function is that it's too sparse, meaning the agent has little guidance on how to modify their behavior to become better at attaining said goal, ...


4

Designing reward functions Designing a reward function is sometimes straightforward, if you have knowledge of the problem. For example, consider the game of chess. You know that you have three outcomes: win (good), loss (bad), or draw (neutral). So, you could reward the agent with $+1$ if it wins the game, $-1$ if it loses, and $0$ if it draws (or for any ...


3

Doing something like the dense, distance-based reward signal you propose is possible... but you have to do it very carefully. If you're not careful, and do it in a naive manner, you are likely to reinforce unwanted behaviour. For example, the way I read that reward function you propose, it provide a positive reward for any steps taken by the agent, with ...


3

In Reinforcement Learning (RL), a reward function is part of the problem definition and should: Be based primarily on the goals of the agent. Take into account any combination of starting state $s$, action taken $a$, resulting state $s'$ and/or a random amount (a constant amount is just a random amount with a fixed value having probability 1). You should ...


3

Dennis Soemers provides an important point that from a theoretical standpoint, this can be seen as a non-issue. However, what you bring up is an important practical issue of potential-based reward shaping (PBRS). The issue is actually worse than you describe---it's more general than $s = s'$. In particular, the issue presents itself differently based on the ...


3

I don't think the situation you're sketching should be a problem at all. If $P(s)$ is high (e.g. $P(s) = 1000$), this means (according to your shaping / "heuristic") that it's valuable to be in the state $s$, that you expect to be able to get high future returns from that state. If you then continuously take actions that keep you in the same state $s$, it ...


2

The Bellman optimality equation is given by $$q_*(s,a) = \sum_{s' \in \mathcal{S}, r \in \mathcal{R}}p(s',r \mid s,a)(r + \gamma \max_{a'\in\mathcal{A}(s')}q_*(s',a')) \tag{1}\label{1}.$$ If the reward is multiplied by a constant $c > 0 \in \mathbb{R}$, then the new optimal action-value function is given by $cq_*(s, a)$. To prove this, we just need to ...


2

The same $\gamma = 0.9$ that you use in the definition $F \doteq \gamma \Phi(s') - \Phi(s)$ should also be used as the discount factor in computing returns for multi-step trajectories. So, rather than simply adding up all the rewards for your different time-steps for the different trajectories, you should discount them by $\gamma$ for every time step that ...


2

Yes, you are right. It is somehow an arbitrary choice, although you should consider the reasonable numerical ranges of your activation functions if you decide to go beyond the values +/- 1. You can also have a think about whether you want to add a small reward for the agent reaching states that are near the goal, if you have an environment where such states ...


2

In reinforcement learning (RL), an immediate reward value must be returned after each action, along with the next state. This value can be zero though, which will have no direct impact on optimality or setting goals. Unless you are modifying the reward scheme to try and make an environment easier to learn (called reward shaping), then you should be aiming ...


2

Sutton and Barto state, "The reward signal is your way of communicating to the robot [agent] what you want it to achieve, not how you want it achieved." Since you stated that the goal is to reach the finish line first, then a reward of $1$ for winning, $0$ for losing, and $0$ at all other time steps seems to fit that narrative. If a draw is identical to a ...


1

Inverse Reinforcement Learning (IRL) is a technique that attempts to recover the reward function that the expert is implicitly maximising based on expert demonstrations. When solving reinforcement learning problems, the agent maximises a reward function specified by the designer, and in the process of reward maximisation, accomplishes some task that it had ...


1

Your examples are equivalent. But it is possible to find a constant yielding a different optimal policy. Your examples are absolutely equivalent. The agent maximizes the reward, and only way to do so is by reaching $s^*$. Consider $r_3$ : $$ r_3(s, a)= \begin{cases} 1, & \text{ if } s \neq s^*\\ 2, & \text{ otherwise} \end{cases} $$ With a ...


1

Theorem The optimal state-action value function of $r'(s, a) \triangleq r(s, a) + c$, for $c \in \mathbb{R}$, would be \begin{align} q_*(s, a) + c + c\gamma + c \gamma^2 + c \gamma^3 + \dots &=q_*(s, a) + c \left( 1 + \gamma + \gamma^2 + \gamma^3 + \dots \right) \\ &= q_*(s, a) + c \left( \sum_{k=0}^{\infty} \gamma^{k} \right) \\ &=q_*(s, a) + c\...


1

I think you should try to reason in terms of total "area" explored by the agent rather than "how far" it moves from the initial point, and also you should add some reward terms to push the agent steering more often. I think that the problem with your setting is more or less this: The agent go as straight as it can because you're rewarding ...


1

You have some freedom to re-define reward schemes, whilst still describing the same goals for an agent. How this works depends to some degree on whether you are dealing with an episodic or continuing problem. Episodic problems An episodic problem ends, and once an agent reaches the terminal state, it is guaranteed zero rewards from that point on. The optimal ...


1

Is the method itself defective or anything wrong with my code? There does indeed appear to be an issue with the code, the publications are fine (I know most of those authors and would very much trust their writing too :) ). The first issue I see, and likely the most important, is that the update() calls of DynamicPBA frequently update the contents of self....


Only top voted, non community-wiki answers of a minimum length are eligible