Hot answers tagged

7

Generally researchers (Ghandar et al, Michalewicz, Lam) have used the profit or return on investment (ROI) as a reward (fitness) function. $ROI = \frac{ \left[\sum_{t=1}^T (Price_t - sc) \times I_s(t) \right] - \left[ \sum_{t=1}^T (Price_t + bc) \times I_b(t) \right] }{ \left[ \sum_{t=1}^T (Price_t + bc) \times I_b(t) \right] }$ where $I_b(t)$ and $I_s(t)$ ...


7

An important thing we're going to need is what is called the "Expected Grad-Log-Prob Lemma here" (proof included on that page), which says that (for any $t$): $$\mathbb{E}_{\tau \sim \pi_{\theta}(\tau)} \left[ \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t) \right] = 0.$$ Taking the analytical expression of the gradient (from, for example, slide 9) ...


6

It depends on your loss function, but you probably need to tweak it. If you are using an update rule like loss = -log(probabilities) * reward, then your loss is high when you unexpectedly got a large reward—the policy will update to make that action more likely to realize that gain. Conversely, if you get a negative reward with high probability, this will ...


5

This is an interesting question actually. There's a quite realistic idea about "where can the curiosity originate from" in the book "On intelligence" written by Jeff Hawkins and Sandra Blakeslee. It's based on such statements: Mind creates its own model of the world it exists in. It makes predictions about everything all the time (actually Jeff Hawkins ...


5

The current method to implement motivation is some kind of artificial reward. Deepmind's DQN for example is driven by the score of the game. The higher the score, the better. The AI learns to adjust its actions to get the most points and therefore the most reward. This is called reinforcement learing. The reward motivates the AI to adapt its actions, so to ...


5

RL agents - implemented correctly - do not take previous rewards into account when making decisions. For instance value functions only assess potential future reward. The state value or expected return (aka utility) $G$ from a starting state $s$ may be defined like this: $$v(s) = \mathbb{E}_{\pi}[G_t|S_t=s] = \mathbb{E}_{\pi}[\sum_{k=0}^{\infty} \gamma^kR_{...


5

Why do both approaches prevent the AI agent from changing its reward function at will? In RL for optimal control, the reward function is part of the problem formulation. That is, it describes the goals of the agent. Sometimes this is obviously something that should not be under the agent's control, if the reward is a real-world quantity that it should ...


5

Expectation of reward after taking action $a$ in state $s$ and ending up in state $s'$ would simply be \begin{equation} r(s, a, s') = \sum_{r \in R} r \cdot p(r|s, a, s') \end{equation} The problem with this is that they do not define probability distribution for rewards separately, they use joint distribution $p(s', r|s, a)$ which represents probability ...


4

What you are proposing is closer to a heuristic for searching than a reward for RL. This is a blurred line, but generally if you start analysing the problem yourself, breaking it down into components and feeding that knowledge into the algorithm, then you place more emphasis on your understanding of the problem, and less on any learning that an agent might ...


4

Markov decision problems are usually defined with a reward function $r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$, and in these cases the rewards are expected to be scalar real values. This makes reinforcement learning (RL) easier, for example when defining a policy $\pi(s,a)=\arg\max_a Q(s,a)$, it is clear what is the maximum of the Q-factors in ...


3

Doing something like the dense, distance-based reward signal you propose is possible... but you have to do it very carefully. If you're not careful, and do it in a naive manner, you are likely to reinforce unwanted behaviour. For example, the way I read that reward function you propose, it provide a positive reward for any steps taken by the agent, with ...


3

In Reinforcement Learning (RL), a reward function is part of the problem definition and should: Be based primarily on the goals of the agent. Take into account any combination of starting state $s$, action taken $a$, resulting state $s'$ and/or a random amount (a constant amount is just a random amount with a fixed value having probability 1). You should ...


3

Your table is almost correct. It is a minor difference, you should not have a $R_0$, the top row, leftmost column of numbers should be empty. That is because the first reward is $R_1$ (a result of taking action $A_0$ in state $S_0$). The alignment of the columns on the right hand side is correct though. It might help to add the time step number at the top. ...


3

How would you implement this "Number of Steps" cost? What the paper is referring to is the reward discounting process which is a standard way of formulating RL problems, either continuous ones, or episodic ones where the goal is to complete a task in the least time (in the episodic version, a fixed cost per time step will also achieve this). As ...


3

I asked professor Richard Sutton a similar question, in the first lecture of the reinforcement learning course. It seems that there are different ways to motivate the machine. In fact, machine motivation seems to me like a dedicated field of research. Typically, machines are motivated by what we call an objective function or a cost function or a loss ...


3

All right, I figured it out. trajectories need not have the same starting state because the distribution of $s_0$ is drawn from a distribution D (mentioned in the paper). Had been confused because many of the code implementations on github focus on trajectories starting from the same state. Hope this helps everyone !


3

Reinforcement learning (RL) control maximises the expected sum of rewards. If you change the reward metric, it will change what counts as optimal. Your reward functions are not the same, so will in some cases change the priority of solutions. As a simple example, consider a choice between trajectories with costs A(0,4,4,4) and B(1,1,1,1). In the original ...


3

Reward in reinforcement learning (RL) is entirely different from a supervised learning (SL) label, but can be related to it indirectly. In a RL control setting, you can imagine that you had a data oracle that gave you SL training example and label pairs $x_i, y_i$ where $x_i$ represents a state and $y_i$ represents the correct action to take in that state in ...


3

If you have multiple types of rewards (say, R1 and R2), then it is no longer clear what would be the optimal way to act: it can happen that one way of acting would maximize R1 and another way would maximize R2. Therefore, optimal policies, value functions, etc., would all be undefined. Of course, you could say that you want to maximize, for example, R1+R2, ...


2

The cross-entropy loss will always be positive because the probability is in the range $[0, 1]$, so $-ln(p)$ will always be positive.


2

The classic working reward scheme for two player zero sum games (i.e. if I win, you lose and vice versa) is simply: +1 for a win 0 for a draw -1 for a loss These rewards should be associated with the last move made by the player before the game is resolved. I thought about giving a negativ reward for the move played before the winning move. That is ...


2

This is a common problem in reward shaping. You want a certain behavior from you agent but its challenging to describe it completely in terms of rewards. This situation you are describing is challenging specifically because as the grid world grows, the chance of randomly stumbling onto the goal state becomes less likely AKA the problem of exploration. There ...


2

I want to maximize the profit inside a trading day and avoid to place the pair (limit buy order, limit sell order) if the profit on that transaction is less than 100$. Be aware that I thought using the "Profit & Loss" as the reward. To me this implies that your profit per transaction is not the true reward function that you should be using. You don't ...


2

You don't need to have a reward on every single timestep, reward at the end is enough. Reinforcement learning can deal with temporal credit assignment problem, all algorithms are designed to work with it. Its enough to define a reward at the end where you, for example, give a reward of 1 if sentence is satisfactory or -1 if it isn't. Regular tabular Q-...


2

Yes, you are right. It is somehow an arbitrary choice, although you should consider the reasonable numerical ranges of your activation functions if you decide to go beyond the values +/- 1. You can also have a think about whether you want to add a small reward for the agent reaching states that are near the goal, if you have an environment where such states ...


2

In this case, the word "system" refers to a Markov decision process (MDP), which is the mathematical model used to represent the reinforcement learning (RL) problem or, in general, a decision making problem. Recall that, in RL, the problem consists in finding an (optimal) policy, which is a policy that allows the agent to collect the highest amount of reward ...


2

Yes you can use RL for this. The trick is to include the location of the cheese as part of the state description. So as well as up to 400 states for the mouse location, you have (very roughly) $400^{10}$ possible cheese locations, meaning you have $400^{11}$ states in total. So you are going to want some function approximation if you want to use RL - you ...


2

The good news is that: Your MDP appears valid, with well-defined states, actions. It has state transition and reward functions (which you have implemented as matrices). There is nothing else to add, it's a full MDP. You could use this MDP to evaluate a policy, using a variety of reinforcement learning (RL) methods suitable for finite discrete MDPS. For ...


2

The reward function can be a function of the current state, current action, and next state: $R(s_t, a_t, s_{t+1})$. It's valid to use the Bellman operator in this setting because it's still a contraction and will yield the optimal value function. NOTE: I'm assuming that you will be solving the MDP with the Bellman equation.


2

Dealing with a Non-Markovian process is unusual in Reinforcement Learning. Although some explicit attempts have been made, the most common approach when confronted with a non-Markovian environment is to try and make the agent's representation of it Markovian. After reducing Agent's model of the dynamics to a Markovian process, rewards are assigned from the ...


Only top voted, non community-wiki answers of a minimum length are eligible