Questions tagged [rewards]

For questions related to the concept of reward, for example, in the context of reinforcement learning and Markov decision processes.

Filter by
Sorted by
Tagged with
1
vote
0answers
49 views

Which model should I choose to maximise reward of having chosen two numbers from a list?

I am looking for a technique to train a machine learning model to choose two items from a list. So, given a list $x=[x_1, x_2, x_3, x_4, \dots, x_n]$, the model needs to choose two elements $(x_i, ...
1
vote
0answers
27 views

Does apprenticeship learning require prospective data?

I am thinking of applying apprenticeship learning on retrospective data. From looking at this paper by Ng https://ai.stanford.edu/~ang/papers/icml04-apprentice.pdf which talks about apprenticeship ...
2
votes
2answers
314 views

Is there a good ratio between the positive and negative rewards in reinforcement learning?

Is there an ideal ratio in reinforcement learning between the positive and negative rewards? Suppose I have the scenario of moving a robot across the river. There are two options, walk across the ...
3
votes
1answer
117 views

Can recovering a reward function using IRL lead to better policies compared to reward shaping?

I am working on a research project about the different reward functions being used in the RL domain. I have read up on Inverse Reinforcement Learning (IRL) and Reward Shaping (RS). I would like to ...
0
votes
0answers
26 views

Curiosity Driven Learning affect optimal policy

I am trying to understand some of the different approaches used to overcome sparse rewards in a reinforcement learning setting for a research project. Particularly, I have looked at curiosity driven ...
2
votes
1answer
98 views

Immediate reward received in Atari game using DQN

I am trying to understand the different reward functions modelled in a reinforcement learning problem. I want to be able to know how the temporal credit assignment problem, (where the reward is ...
3
votes
1answer
149 views

How to avoid rapid actuator movements in favor of smooth movements in a continuous space and action space problem?

I'm working on a continuous state / continuous action controller. It shall control a certain roll angle of an aircraft by issuing the correct aileron commands (in $[-1, 1]$). To this end, I use a ...
2
votes
2answers
250 views

Simulating successful trajectories in Montezuma's Revenge turns out to be unsuccessful

I have written code in OpenAI's gym to simulate a random playing in Montezuma's Revenge where the agent randomly samples actions from the action space and tries to play the game. A success for me is ...
6
votes
1answer
147 views

Why cannot an AI agent adjust the reward function directly?

In standard Reinforcement Learning the reward function is specified by an AI designer and is external to the AI agent. The agent attempts to find a behaviour that collects higher cumulative discounted ...
1
vote
0answers
40 views

Should an RL agent directly observe the reward?

I am training an A2C reinforcement learning agent in a dense reward environment (where rewards are known and explicit at every timestep). Is it redundant to include the previous reward in the current ...
4
votes
1answer
167 views

How to assign rewards in a non-Markovian environment?

I am quite new to the Reinforcement Learning domain and I am curious about something. It seems to be the case that the majority of current research assumes Markovian environments, that is, future ...
1
vote
1answer
93 views

Neural network for reinforcement learning

I’m using a simple neural network to solve a reinforcement learning problem. The configuration is: X-inputs: The current state Y-outputs: The possible actions Whenever the network yields a “good” ...
4
votes
1answer
126 views

What could be the cause of the drop in the reward in A3C?

The mean episodic reward is generally increasing, but it has spontaneous drops, and I'm not sure of their cause. The problem has a sparse reward, batch size=2000, <...
3
votes
1answer
168 views

Can someone please help me validate my MDP?

Problem Statement : I have a system with four states - S1 through S4 where S1 is the beginning state and S4 is the end/terminal state. The next state is always better than the previous state i.e if ...
1
vote
0answers
49 views

Finding optimal Value function and Policy for an MDP

I am solving an RL MDP problem which is model based. I have an MDP which has four possible states S1-S4 and four different actions A1-A4, with S4 being terminal state and S1 is the beginning state. ...
3
votes
1answer
60 views

Can I have different rewards for a single action based on which state it transitions to?

I am working on an MDP where there are four states and ten actions. I am supposed to derive the optimal policy to reach the desired state. At any state, a particular action can take you to any of the ...
2
votes
0answers
29 views

Developmental systems that try to explain or understand the reward value in the reinforcement learning?

Are there methods (possibly logical or (how they are called in the literature) relational) that allows for the developmental systems to understand or explain the value of the received reward during ...
2
votes
1answer
641 views

Should RL rewards diminish over time?

Should a reward be cumulative or diminish over time? For example, say an agent performed a good action at time $t$ and received a positive reward $R$. If reward is cumulative, $R$ is carried on ...
4
votes
1answer
164 views

How to evaluate an RL algorithm when used in a game?

I'm planning to create a web-based RL board game, and I wondered how I would evaluate the performance of the RL agent. How would I be able to say, "Version X performed better than version Y, as ...
2
votes
0answers
85 views

Is it possible to use Reward Function of type R(s, a, s') if more than one action is applied?

I am applying a reinforcement learning agent (PPO2, stable baselines implementation) to a custom built environment using OpenAI Gym. One reward function (formualted as loss function, that is, all ...
1
vote
0answers
119 views

Reward problem in A2C with multiple simultaneous discrete actions

I've built an A2C model whose actor's network has two different kinds of discrete actions, so the critic would take state and action (note that critic takes 2 actions because in each timestep we will ...
2
votes
0answers
62 views

Will the RL agent implemented as a neural network fine-tune itself?

Normally, when you develop a neural network, train it for object recognition (on normal objects like bike, car, plane, dog, cloud, etc.), and it turns out to perform very well, you would like to fine-...
3
votes
1answer
69 views

Are there any reliable ways of modifying the reward function to make the rewards less sparse?

If I am training an agent to try and navigate a maze as fast as possible, a simple reward would be something like \begin{align} R(\text{terminal}) &= N - \text{time}\ \ , \ \ N \gg \text{...
4
votes
0answers
54 views

How define a reward function for a humanoid agent whose goal is to stand up from the ground?

I'm trying to teach a humanoid agent how to stand up after falling. The episode starts with the agent lying on the floor with its back touching the ground, and its goal is to stand up in the shortest ...
1
vote
1answer
869 views

What is the reward system of reinforcement learning?

Can you describe this reward system in more detail? I understand that the environment sends a signal indicating whether or not the action taken by the agent was 'good' or not, but it seems too simple. ...
2
votes
0answers
18 views

How is GARB implemented in PGRD-DL to calculate gradients w.r.t. internal rewards?

In section 3 of this paper the author outlines how GARB was adapted to reduce the variance in updating parameters to an internal reward function estimator. I have read it a number of times and ...
0
votes
1answer
60 views

Encourage Deep Q to seek short-term reward

I understand that gamma is an important factor in determining the rewards for a deep Q agent, however during testing of my network I am noticing that the agent is outputting more actions to "do ...
1
vote
0answers
48 views

Deciding the rewards for different actions in Pong for a DQN agent

I am attempting to implement an agent that learns to play in the Pong environment, the environment was created in PyGame and I return the pixel data and score at each frame. I use a CNN to take a ...
4
votes
0answers
42 views

What research has been done on learning non-Markovian reward functions?

Recently, some work has been done planning and learning in Non-Markovian Decision Processes, that is, decision-making with temporally extended rewards. In these settings, a particular reward is ...
2
votes
1answer
63 views

Do we have to consider the feasability of an action when defining the reward function of a MDP?

Do we have to consider if (s is given) an action a can lead to s' when defining a reward function? To be more specific: Let's say I have a 1D Map like: |A|B|C|D| ...
2
votes
2answers
204 views

Can Reinforcement Learning solve problems, where certain elements in the environement are randomly located?

I want to solve a problem using Reinforcement Learning on a 20x20 board. An agent (a mouse) has to get the highest possible rewards as fast as possible by collecting cheese, which there are 10 in ...
5
votes
2answers
1k views

Reinforcement Learning with long term rewards and fixed states and actions

I have read a lot about RL algorithms, that update the action-value function at each step with the currently gained reward. The requirement here is, that the reward is obtained after each step. I ...
3
votes
1answer
99 views

Given specific rewards, how can I calculate the returns for each time step?

Let's use Excercise 3.8 from Sutton, Barto - Introduction to RL: Suppose $\gamma = 0.5$ and following sequence of rewards is received $R_1=-1$ , $R_2=2$ , $R_3=6$ , $R_4=3$ , $R_5=2$ , with $T=5$ ...
2
votes
1answer
51 views

How do I avoid an agent to tend to terminate in a negative state when time needs to be taken into account?

In an unknown environment, how do I avoid an agent to tend to terminate its trajectory in a negative state when time needs to be taken into account? Suppose the following example to make my question ...
6
votes
1answer
3k views

Suitable reward function for trading buy and sell orders

I am working to build an deep reinforcement learning agent which can place orders (i.e. limit buy and limit sell orders). The actions are ...
0
votes
3answers
187 views

For some reasons, a reward becomes a penalty if

I am working to build a reinforcement agent with DQN. The agent would be able to place buy and sell orders for a day trading purpose. I am facing a little problem with that project. The question is "...
5
votes
1answer
2k views

Why does the “reward to go” trick in policy gradient methods work?

In the policy gradient method, there's a trick to reduce the variance of policy gradient. We use causality, and remove part of the sum over rewards so that only actions happened after the reward are ...
2
votes
1answer
399 views

How should I define the reward function in the case of Connect Four?

I'm using RL to train a Network on the game Connect4. It learns quickly that 4 connected pieces is good. It gets a reward of 1 for this. A zero is rewarded for all other moves. It takes quite a time ...
3
votes
2answers
579 views

What is the main difference between additive rewards and discounted rewards?

What is the difference between additive and discounted rewards?
1
vote
1answer
110 views

Why is it ok to calculate the reward based on a hidden state?

I'm looking at this source code, where the reward is calculated with reward = cmp(score(self.player), score(self.dealer)) Why is it ok to calculate the reward ...
3
votes
1answer
72 views

Is my interpretation of the return correct?

Sutton and Barto 2018 define the discounted return $G_t$ the following way (p 55): Is my interpretation correct? Or should all "1" be in the same column?
2
votes
1answer
64 views

How should I take into consideration the number of steps in the reward function?

I am currently implementing the paper Active Object Localization with Deep Reinforcement Learning in Python. While reading about the reward scheme I came across the following: Finally, the proposed ...
10
votes
1answer
3k views

What is the difference between expected return and value function?

I've seen numerous mathematical explanations of reward, value functions $V(s)$, and return functions. The reward provides an immediate return for being in a specific state. The better the reward, the ...
12
votes
6answers
2k views

What would motivate a machine?

Currently, within the AI development field, the main focus seems to be on pattern recognition and machine learning. Learning is about adjusting internal variables based on a feedback loop. Maslow's ...
9
votes
2answers
9k views

How do I handle negative rewards in policy gradients with the cross-entropy loss function?

I am using policy gradients in my reinforcement learning algorithm, and occasionally my environment provides a severe penalty (i.e. negative reward) when a wrong move is made. I'm using a neural ...

1
2