Questions tagged [rewards]

For questions related to the concept of reward, for example, in the context of reinforcement learning and Markov decision processes.

30 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4
votes
1answer
126 views

What could be the cause of the drop in the reward in A3C?

The mean episodic reward is generally increasing, but it has spontaneous drops, and I'm not sure of their cause. The problem has a sparse reward, batch size=2000, <...
4
votes
0answers
54 views

How define a reward function for a humanoid agent whose goal is to stand up from the ground?

I'm trying to teach a humanoid agent how to stand up after falling. The episode starts with the agent lying on the floor with its back touching the ground, and its goal is to stand up in the shortest ...
4
votes
0answers
42 views

What research has been done on learning non-Markovian reward functions?

Recently, some work has been done planning and learning in Non-Markovian Decision Processes, that is, decision-making with temporally extended rewards. In these settings, a particular reward is ...
3
votes
0answers
55 views

How does normalization of the inputs work in the context of PPO?

What does the normalization of the inputs mean in the context of PPO? At each time step of an episode, I only know the values of this time step and of the previous ones, if I take track of them. This ...
3
votes
0answers
85 views

What are the guidelines for defining a reward function in reinforcement learning (bandit problem)?

I'm working currently on a problem and I'm using RL (bandit problem). In my system, I have an agent that chooses an action among $k$ possible actions, and a user that decides whether the agent ...
2
votes
0answers
57 views

What is the dimensionality of these derivatives in the paper “Active Learning for Reward Estimation in Inverse Reinforcement Learning”?

I'm trying to implement in code part of the following paper: Active Learning for Reward Estimation in Inverse Reinforcement Learning. I'm specifically referring to section 2.3 of the paper. Let's ...
2
votes
0answers
63 views

Should I use the discounted average reward as objective in a finite-horizon problem?

I am new to reinforcement learning, but, for a finite horizon application problem, I am considering using the average reward instead of the sum of rewards as the objective. Specifically, there are a ...
2
votes
0answers
31 views

How should I design a reward function for a NLP problem where two models interoperate?

I would like to design a reward function. I am training two models from the first model that classify set of texts (paragraphs and keywords) and I also got some hidden states. The second model is ...
2
votes
0answers
34 views

How was the DQN trained to play many games?

Some people claim that DQN was used to play many Atari games. But what actually happened? Was DQN trained only once (with some data from all games) or was it trained separately for each game? What was ...
2
votes
0answers
28 views

How should I define the reward function for the Connect Four game?

I'm creating an RL application for the game Connect Four. I've researched the different strategies for the game and which positions are more favourable to lead to a win. Should I be assigning ...
2
votes
0answers
29 views

Developmental systems that try to explain or understand the reward value in the reinforcement learning?

Are there methods (possibly logical or (how they are called in the literature) relational) that allows for the developmental systems to understand or explain the value of the received reward during ...
2
votes
0answers
85 views

Is it possible to use Reward Function of type R(s, a, s') if more than one action is applied?

I am applying a reinforcement learning agent (PPO2, stable baselines implementation) to a custom built environment using OpenAI Gym. One reward function (formualted as loss function, that is, all ...
2
votes
0answers
62 views

Will the RL agent implemented as a neural network fine-tune itself?

Normally, when you develop a neural network, train it for object recognition (on normal objects like bike, car, plane, dog, cloud, etc.), and it turns out to perform very well, you would like to fine-...
2
votes
0answers
18 views

How is GARB implemented in PGRD-DL to calculate gradients w.r.t. internal rewards?

In section 3 of this paper the author outlines how GARB was adapted to reduce the variance in updating parameters to an internal reward function estimator. I have read it a number of times and ...
1
vote
1answer
30 views

How do I represent sample efficiency of RL rewards in mathematical notation?

I define sample efficiency as the area under the curve/graph, where $x$-axis is the number of episodes while y-axis is the cumulative reward for that episode. I would like to formally define it with a ...
1
vote
0answers
60 views

Why would DDPG with Hindsight Experience Replay not converge?

I am trying to train a DDPG agent augmented with Hindsight Experience Replay (HER) to solve the KukaGymEnv environment. The actor and critic are simple neural networks with two hidden layers (as in ...
1
vote
0answers
38 views

Does importance sampling for off-policy estimation also apply to the case of negative rewards?

Importance sampling is a common method for calculating off-policy estimates in RL. I have been reading through some of the original documentation (D.G. Horvitz and D.J. Thompson, Powell, M.J. and ...
1
vote
0answers
39 views

State-of-the-art algorithms not working on a custom RL environment

I'm trying to train a RL agent on a custom, highly stochastic environment (MDP). In order to do so I'm using existing implementations of state-of-the-art RL algorithms as provided by Stable Baselines. ...
1
vote
0answers
33 views

How can I find the appropriate reward value for my reinforcement learning problem?

I am wondering how can I find the appropriate reward value for each specific problem. I know this is a highly empirical process, but I am sure that the value is not set totally at random. I want to ...
1
vote
0answers
49 views

Which model should I choose to maximise reward of having chosen two numbers from a list?

I am looking for a technique to train a machine learning model to choose two items from a list. So, given a list $x=[x_1, x_2, x_3, x_4, \dots, x_n]$, the model needs to choose two elements $(x_i, ...
1
vote
0answers
27 views

Does apprenticeship learning require prospective data?

I am thinking of applying apprenticeship learning on retrospective data. From looking at this paper by Ng https://ai.stanford.edu/~ang/papers/icml04-apprentice.pdf which talks about apprenticeship ...
1
vote
0answers
40 views

Should an RL agent directly observe the reward?

I am training an A2C reinforcement learning agent in a dense reward environment (where rewards are known and explicit at every timestep). Is it redundant to include the previous reward in the current ...
1
vote
0answers
50 views

Finding optimal Value function and Policy for an MDP

I am solving an RL MDP problem which is model based. I have an MDP which has four possible states S1-S4 and four different actions A1-A4, with S4 being terminal state and S1 is the beginning state. ...
1
vote
0answers
119 views

Reward problem in A2C with multiple simultaneous discrete actions

I've built an A2C model whose actor's network has two different kinds of discrete actions, so the critic would take state and action (note that critic takes 2 actions because in each timestep we will ...
1
vote
0answers
48 views

Deciding the rewards for different actions in Pong for a DQN agent

I am attempting to implement an agent that learns to play in the Pong environment, the environment was created in PyGame and I return the pixel data and score at each frame. I use a CNN to take a ...
0
votes
0answers
37 views

How to normalize rewards in REINFORCE?

I'm trying to solve a reinforcement learning problem using a Monte Carlo policy gradient algorithm and, more specifically, REINFORCE, with rewards attributed to individual moves instead of applied to ...
0
votes
0answers
16 views

Multi-agent policy gradient, 1 total reward instead of reward in each step, 2 changing action space

I am new in reinforcement learning and not sure I have the right understanding of multi-agent policy gradient. 1, in my question, each agent has its own action space. When doing the sampling, for each ...
0
votes
0answers
37 views

Double DQN backpropagation of negative final rewards?

My problem is that in my Double DQN model, negative final rewards are not being backpropagated into action Q-values, and so some Q-values are positive, when they should be negative, and hence ...
0
votes
0answers
26 views

Curiosity Driven Learning affect optimal policy

I am trying to understand some of the different approaches used to overcome sparse rewards in a reinforcement learning setting for a research project. Particularly, I have looked at curiosity driven ...
0
votes
1answer
32 views

Is is not possible to achieve average reward of more than 20-40 with simple Q-Learning

I have implemented the simple Q-Learning based solution for AI-gym's Cartpole-v0. However, despite changing hyper-parameters, and rechecking my code, I cannot get an average reward (N-running reward) ...