Questions tagged [rewards]

For questions related to the concept of reward and reward functions (e.g. in the context of reinforcement learning and Markov decision processes).

Filter by
Sorted by
Tagged with
2
votes
1answer
25 views

Can the agent wait until the end of the episode to determine the reward in SARSA?

From Sutton and Barto's book Reinforcement Learning (Adaptive Computation and Machine Learning series) (p. 99), the following definition for first-visit MC prediction, for estimating $V \sim V_\pi$ is ...
3
votes
1answer
68 views

How do I convert an MDP with the reward function in the form $R(s,a,s')$ to and an MDP with a reward function in the form $R(s,a)$?

The AIMA book has an exercise about showing that an MDP with rewards of the form $r(s, a, s')$ can be converted to an MDP with rewards $r(s, a)$, and to an MDP with rewards $r(s)$ with equivalent ...
1
vote
0answers
24 views

State-of-the-art algorithms not working on a custom RL environment

I'm trying to train a RL agent on a custom, highly stochastic environment (MDP). In order to do so I'm using existing implementations of state-of-the-art RL algorithms as provided by Stable Baselines. ...
1
vote
1answer
44 views

How do you know if an agent has learnt its environment in reinforcement learning?

I'm new to reinforcement learning and trying to understand it. If you train an agent using a reinforcement learning algorithm (discrete or continuous) on an environment (real or simulated), then how ...
2
votes
1answer
29 views

Shouldn't expected return be calculated for some faraway time in the future $t+n$ instead of current time $t$?

I am learning RL for the first time. It may be naive, but it is a bit odd to grasp this idea that, if the goal of RL is to maximize the expected return, then shouldn't the expected return be ...
1
vote
1answer
28 views

Which reward function works for recommendation systems using knowledge graphs?

I've been reading this paper on recommendation systems using reinforcement learning (RL) and knowledge graphs (KGs). To give some background, the graph has several (finitely many) entities, of which ...
2
votes
0answers
23 views

Reward Function for Racing Game

I'm busy working on a project where I'm building an agent for a racing game. In this game is a randomised map where there are speed boosts for the player to pick up and obstacles that act to slow the ...
3
votes
1answer
37 views

Appropriate algorithm for RL problem with sparse rewards, continuous actions and significant stochasticity

I'm working on a RL problem with the following properties: The rewards are extremely sparse i.e. all rewards are 0 except the terminal non-zero reward. Ideally I would not use any reward engineering ...
1
vote
1answer
27 views

Can optimizing for immediate reward result in a policy maximizing the return?

The goal of a reinforcement learning agent is to maximize the expected return which is often a discounted sum of future rewards. The return indeed is a very noisy random variable as future rewards ...
2
votes
0answers
23 views

How should I design a reward function for a NLP problem where two models interoperate?

I would like to design a reward function. I am training two models from the first model that classify set of texts (paragraphs and keywords) and I also got some hidden states. The second model is ...
0
votes
1answer
30 views

What is the best measurement for how good an action of a reinforcement learning agent really is?

Even when we get a valuable reward signal after every single action, this immediate reward only approximates the short term goodness of the action. To consider the long term effect of an action, we ...
3
votes
2answers
122 views

Formula for expected rewards for state–action–next-state triples as a three-argument function

While reading about reinforcement learning, I have come across the following expression for expected rewards in terms of a summation, the denominator of which I am not able to account for. The ...
3
votes
0answers
20 views

How does normalization of the inputs work in the context of PPO?

What does the normalization of the inputs mean in the context of PPO? At each time step of an episode, I only know the values of this time step and of the previous ones, if I take track of them. This ...
2
votes
0answers
28 views

How was the DQN trained to play many games?

Some people claim that DQN was used to play many Atari games. But what actually happened? Was DQN trained only once (with some data from all games) or was it trained separately for each game? What was ...
1
vote
1answer
94 views

In RL, if I assign the rewards for better positional play, the algorithm is learning nothing?

I'm creating an RL application for the game Connect Four. If I tell the algorithm which moves/token positions will receive greater rewards, surely it's not actually learning anything; it's just a ...
1
vote
1answer
38 views

What is the relationship between the reward function and the value function?

To clarify it in my head, the value function calculates how 'good' it is to be in a certain state by summing all future (discounted) rewards, while the reward function is what the value function uses ...
2
votes
0answers
17 views

How should I define the reward function for the Connect Four game?

I'm creating an RL application for the game Connect Four. I've researched the different strategies for the game and which positions are more favourable to lead to a win. Should I be assigning ...
1
vote
0answers
21 views

How can I find the appropriate reward value for my reinforcement learning problem?

I am wondering how can I find the appropriate reward value for each specific problem. I know this is a highly empirical process, but I am sure that the value is not set totally at random. I want to ...
3
votes
1answer
38 views

Do all expert trajectories have the same starting state in apprenticeship learning?

In the apprenticeship learning algorithm described by Ng et al. in Apprenticeship Learning via Inverse Reinforcement Learning, they mention that expert trajectories come in the form of $\{s_0^i, s_1^i\...
3
votes
0answers
61 views

What are the guidelines for defining a reward function in reinforcement learning (bandit problem)?

I'm working currently on a problem and I'm using RL (bandit problem). In my system, I have an agent that chooses an action among $k$ possible actions, and a user that decides whether the agent ...
3
votes
0answers
33 views

Is this a good approach to solving Atari's “Montezuma's Revenge”?

I'm new to Reinforcement Learning. For an internship, I am currently training Atari's "Montezuma's Revenge" using a double Deep Q-Network with Hindsight Experience Replay (HER). HER is supposed to ...
2
votes
1answer
35 views

How does the optimization process in hindsight experience replay exactly work?

I was reading the following research paper Hindsight Experience Replay. This is the paper that introduces a concept called Hindsight Experience Replay (HER), which basically attempts to alleviate the ...
0
votes
1answer
35 views

How to incentivise snake to go straight to apple?

I have made a Deep Q Network for the game snake but unfortunately, the snake exhibits some unwanted behavior. It generally does quite well but sometimes it gets stuck in an infinite loop that it can't ...
1
vote
0answers
41 views

Which model should I choose to maximise reward of having chosen two numbers from a list?

I am looking for a technique to train a machine learning model to choose two items from a list. So, given a list $x=[x_1, x_2, x_3, x_4, \dots, x_n]$, the model needs to choose two elements $(x_i, ...
1
vote
0answers
20 views

Does apprenticeship learning require prospective data?

I am thinking of applying apprenticeship learning on retrospective data. From looking at this paper by Ng https://ai.stanford.edu/~ang/papers/icml04-apprentice.pdf which talks about apprenticeship ...
2
votes
2answers
48 views

Is there a good ratio between the positive and negative rewards in reinforcement learning?

Is there an ideal ratio in reinforcement learning between the positive and negative rewards? Suppose I have the scenario of moving a robot across the river. There are two options, walk across the ...
2
votes
0answers
32 views

Policy $\pi$ obtained from inverse reinforcement learning vs reward shaping

I am working on a research project about the different reward functions being used in RL domain. I have read up on Inverse Reinforcement Learning (IRL) and Reward Shaping (RS). I would like to clarify ...
0
votes
0answers
19 views

Curiosity Driven Learning affect optimal policy

I am trying to understand some of the different approaches used to overcome sparse rewards in a reinforcement learning setting for a research project. Particularly, I have looked at curiosity driven ...
2
votes
1answer
46 views

Immediate reward received in Atari game using DQN

I am trying to understand the different reward functions modelled in a reinforcement learning problem. I want to be able to know how the temporal credit assignment problem, (where the reward is ...
2
votes
0answers
44 views

Reinforcement Learning Continuous Control (DDPG): How to avoid thrashing of issued actions? How to reward smooth output over flittering?

Currently I'm working on a continuous state / continuous action controller. It shall control a certain roll angle of an aircraft by issuing the correct aileron commands (between -1...1 continuous). ...
2
votes
2answers
183 views

Simulating successful trajectories in Montezuma's Revenge turns out to be unsuccessful

I have written code in OpenAI's gym to simulate a random playing in Montezuma's Revenge where the agent randomly samples actions from the action space and tries to play the game. A success for me is ...
6
votes
1answer
83 views

Why cannot an AI agent adjust the reward function directly?

In standard Reinforcement Learning the reward function is specified by an AI designer and is external to the AI agent. The agent attempts to find a behaviour that collects higher cumulative discounted ...
1
vote
0answers
32 views

Should an RL agent directly observe the reward?

I am training an A2C reinforcement learning agent in a dense reward environment (where rewards are known and explicit at every timestep). Is it redundant to include the previous reward in the current ...
5
votes
0answers
69 views

Counterexamples to the reward hypothesis

On Sutton and Barto's RL book, the reward hypothesis is stated as that all of what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative ...
4
votes
1answer
80 views

How to assign rewards in a non-Markovian environment?

I am quite new to the Reinforcement Learning domain and I am curious about something. It seems to be the case that the majority of current research assumes Markovian environments, that is, future ...
1
vote
1answer
73 views

Neural network for reinforcement learning

I’m using a simple neural network to solve a reinforcement learning problem. The configuration is: X-inputs: The current state Y-outputs: The possible actions Whenever the network yields a “good” ...
3
votes
1answer
67 views

What could be the cause of the drop in the reward in A3C?

The mean episodic reward is generally increasing, but it has spontaneous drops, and I'm not sure of their cause. The problem has a sparse reward, batch size=2000, <...
2
votes
1answer
98 views

Doubt in Deep-Q learning with sparse rewards

I am working on a deep reinforcement learning problem, when I got stuck at the following questions. They are rather general and not specific to my specific problem. The solution uses a sparse reward ...
3
votes
1answer
88 views

Can someone please help me validate my MDP?

Problem Statement : I have a system with four states - S1 through S4 where S1 is the beginning state and S4 is the end/terminal state. The next state is always better than the previous state i.e if ...
1
vote
0answers
38 views

Finding optimal Value function and Policy for an MDP

I am solving an RL MDP problem which is model based. I have an MDP which has four possible states S1-S4 and four different actions A1-A4, with S4 being terminal state and S1 is the beginning state. ...
3
votes
1answer
48 views

Can I have different rewards for a single action based on which state it transitions to?

I am working on an MDP where there are four states and ten actions. I am supposed to derive the optimal policy to reach the desired state. At any state, a particular action can take you to any of the ...
2
votes
0answers
26 views

Developmental systems that try to explain or understand the reward value in the reinforcement learning?

Are there methods (possibly logical or (how they are called in the literature) relational) that allows for the developmental systems to understand or explain the value of the received reward during ...
2
votes
1answer
590 views

Should RL rewards diminish over time?

Should a reward be cumulative or diminish over time? For example, say an agent performed a good action at time $t$ and received a positive reward $R$. If reward is cumulative, $R$ is carried on ...
2
votes
0answers
59 views

Is it possible to use Reward Function of type R(s, a, s') if more than one action is applied?

I am applying a reinforcement learning agent (PPO2, stable baselines implementation) to a custom built environment using OpenAI Gym. One reward function (formualted as loss function, that is, all ...
1
vote
0answers
55 views

Reward problem in A2C with multiple simultaneous discrete actions

I've built an A2C model whose actor's network has two different kinds of discrete actions, so the critic would take state and action (note that critic takes 2 actions because in each timestep we will ...
2
votes
0answers
34 views

Will the RL agent implemented as a neural network fine-tune itself?

Normally, when you develop a neural network, train it for object recognition (on normal objects like bike, car, plane, dog, cloud, etc.), and it turns out to perform very well, you would like to fine-...
2
votes
1answer
37 views

Using heuristic dense rewards in a sparse problem

If I am training an agent to try and navigate a maze as fast as possible, a simple reward would be something like \begin{align} R(\text{terminal}) &= N - \text{time}\ \ , \ \ N \gg \text{...
1
vote
1answer
354 views

What is the reward system of reinforcement learning?

Can you describe this system in more detail? I understand that the environment sends a signal indicating whether or not the action taken by the agent was 'good' or not, but it seems too simple. ...
2
votes
0answers
14 views

How is GARB implemented in PGRD-DL to calculate gradients w.r.t. internal rewards?

In section 3 of this paper the author outlines how GARB was adapted to reduce the variance in updating parameters to an internal reward function estimator. I have read it a number of times and ...
0
votes
1answer
56 views

Encourage Deep Q to seek short-term reward

I understand that gamma is an important factor in determining the rewards for a deep Q agent, however during testing of my network I am noticing that the agent is outputting more actions to "do ...