People who code: we want your input. Take the Survey

Hot answers tagged

6

Rather than the survey by Liu et al. recommended above, I'd suggest you read the following survey paper for an overview of MORL (disclaimer - I was a co-author on this, but I genuinely think it is a much more useful introduction to this area) Roijers, D. M., Vamplew, P., Whiteson, S., & Dazeley, R. (2013). A survey of multi-objective sequential decision-...


5

Markov decision problems are usually defined with a reward function $r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$, and in these cases the rewards are expected to be scalar real values. This makes reinforcement learning (RL) easier, for example when defining a policy $\pi(s,a)=\arg\max_a Q(s,a)$, it is clear what is the maximum of the Q-factors in ...


4

If you have multiple types of rewards (say, R1 and R2), then it is no longer clear what would be the optimal way to act: it can happen that one way of acting would maximize R1 and another way would maximize R2. Therefore, optimal policies, value functions, etc., would all be undefined. Of course, you could say that you want to maximize, for example, R1+R2, ...


4

I believe the claim is true. Here is my attempt at a proof. Let us consider the optimal infinite horizon value function $V_\alpha^*$ of $\mathcal{M}_\alpha$ at an arbitrary state $s \in S$. The value $V_\alpha^*(s)$ is the expected sum of discounted rewards under an optimal policy $\pi_\alpha^*$, i.e., \begin{equation} V_\alpha^*(s) = \mathbb{E}_{\rho_\...


3

In short, you don't regret your bad luck that you could do nothing about, you regret your bad choices that you could have done something about if only you knew. The point of regret as a metric therefore is to compare your choices with the ideal choices. This makes sense in MABs, because although the primary goal is to gain the most reward, the learning part ...


3

Reward in reinforcement learning (RL) is entirely different from a supervised learning (SL) label, but can be related to it indirectly. In a RL control setting, you can imagine that you had a data oracle that gave you SL training example and label pairs $x_i, y_i$ where $x_i$ represents a state and $y_i$ represents the correct action to take in that state in ...


2

Lets assume $\sup_{s,a} r(s,a)<b$. Then for continuing problems the upper bound can be obtained by \begin{align} \sum_{t=0}^{\infty} \gamma^{t}r(s_t,a_t) &\le \sum_{t=0}^{\infty} \gamma^{t} \sup_{s,a}r(s,a) \nonumber \\ &=\sum_{t=0}^{\infty} \gamma^{t} b = \frac{b}{1-\gamma}. \end{align} We can use the same bound for episodic tasks with ...


2

My answer to:Is there an upper limit to the maximum cumulative reward in a deep reinforcement learning problem? Yes but depending on the environment, if dealing with the theoretical environment, where there are infinite number of time steps. Calculating the upper bound In reinforcement learning (deep RL inclusive), we want to maximize the discounted ...


2

In any reinforcement learning problem, not just Deep RL, then there is an upper bound for the cumulative reward, provided that the problem is episodic and not continuing. If the problem is episodic and the rewards are designed such that the problem has a natural ending, i.e. the episode will end regardless of how well the agent does in the environment, then ...


2

Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards? That is the definition of return. In fact when applying a discount factor this should formally be called discounted return, and not simply "return". Usually the same symbol is used for both ($R$ in your case, $G$ in e.g. Sutton & Barto). There ...


2

You know all the rewards. They're 5, 7, 7, 7, and 7s forever. The problem now boils down to essentially a geometric series computation. $$ G_0 = R_0 + \gamma G_1 $$ $$ G_0 = 5 + \gamma\sum_{k=0}^\infty 7\gamma^k $$ $$ G_0 = 5 + 7\gamma\sum_{k=0}^\infty\gamma^k $$ $$ G_0 = 5 + \frac{7\gamma}{1-\gamma} = \frac{5 + 2\gamma}{1-\gamma} $$


2

We assume that our MDP is ergodic. Loosely speaking, this means that wherever the MDP starts (i.e. no matter which state we start in) or any actions the agent takes early on can only have a limited effect on the MDP and in the limit (as $t \rightarrow \infty$) the expectation of being in a given state depends only on the policy $\pi$ and the transition ...


1

Epsilon greedy is unaffected by scaling of rewards, it always selects a random action with a probability of epsilon. On the other hand, if we look at the formulation of UCB (Section 2.7 of Reinforcement Learning, Sutton and Barto): $$A_t \doteq \underset{a}{\operatorname{argmax}} [\mathcal{Q}_t(a) + c \sqrt{\frac{\ln t}{N_t(a)}}]$$ Where $Q_t(a)= \frac{R_1 +...


1

This is mostly an implementation architecture problem, and the thing is that basically you can implement anything in the traditional setting. To do so instead of having Env<->Agent1<->Agent2, you should have Agent1<->SuperEnv<->Agent2 where SuperEnv contains Env, and simply uses the reward given to SuperEnv by Agent1 and passes it to ...


1

It is not 100% clear, but this seems like an instance of catastrophic forgetting. This is something that often impacts reinforcement learning. I have answered a very similar question on Data Science stack exchange, and reproduce the same answer here. This is called "catastrophic forgetting" and can be a serious problem in many RL scenarios. If you ...


1

I think you should try to reason in terms of total "area" explored by the agent rather than "how far" it moves from the initial point, and also you should add some reward terms to push the agent steering more often. I think that the problem with your setting is more or less this: The agent go as straight as it can because you're rewarding ...


1

Measure what you want to achieve as directly as possible, and reward that. Later you can add more sophisticated incentives for the type of motion etc, but the key to a good reward signal is that it measures the quality of a solution at a high level, without specifying how to achieve that solution. If you want your simulated car to explore, you will want to ...


1

This is known as reward hacking in the literature; see, e.g., https://medium.com/@deepmindsafetyresearch/designing-agent-incentives-to-avoid-reward-tampering-4380c1bb6cd for discussion and further links.


1

You make a lot of assumptions about AGI, namely that 'we need a computer to compute the utility and reward AGI'. It not clear to me that (1) we can achieve AGI, (2) AGI will be on a computer as we know it and (3) AGI will work with a utility / reward function as we know them. One thing I am sure though is that ML is known for "cheating" (see for ...


1

You have some freedom to re-define reward schemes, whilst still describing the same goals for an agent. How this works depends to some degree on whether you are dealing with an episodic or continuing problem. Episodic problems An episodic problem ends, and once an agent reaches the terminal state, it is guaranteed zero rewards from that point on. The optimal ...


1

There are a few ways to resolve values of infinite sums. In this case, we can use a simple technique of self-reference to create a solvable equation. I will show how to do it for the generic case here of an MDP with same reward $r$ on each timestep: $$G_t = \sum_{k=0}^{\infty} \gamma^k r$$ We can "pop off" the first item: $$G_t = r + \sum_{k=1}^{\...


1

Inverse Reinforcement Learning (IRL) is a technique that attempts to recover the reward function that the expert is implicitly maximising based on expert demonstrations. When solving reinforcement learning problems, the agent maximises a reward function specified by the designer, and in the process of reward maximisation, accomplishes some task that it had ...


1

I think we give ourselves too much credit by already referring to our algorithms and machines as actually thinking and acting on motivations. In my opinion we still have a bit to go before we can actually refer to a human creation as thinking or being able to have motivations more then basic physical ones. By that I would say that a Machines' or AI ...


Only top voted, non community-wiki answers of a minimum length are eligible