Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

604 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
2
votes
0answers
31 views

How should I design a reward function for a NLP problem where two models interoperate?

I would like to design a reward function. I am training two models from the first model that classify set of texts (paragraphs and keywords) and I also got some hidden states. The second model is ...
2
votes
0answers
62 views

Action masking for on policy algorithm like PPO

I have an environment, in which my agent learns according to PPO. The environment has a maximum of 80 actions, however not all of them are always allowed. My idea was to mask them, by setting the ...
2
votes
0answers
31 views

How can I formalise a non-zero-sum game of $N$ agent as Markov game?

I coded a non-zero-sum game of $N$ agents in a discrete dynamic environment to RL with Q-learning and DQN agents. It's like a marathon. Only two actions are available per agent: $\{ G \text{ (move ...
2
votes
0answers
45 views

Is Mean Squared Error Loss function a good loss function for continuous variables $0 < x < 1$

Suppose I am utilising a neural network to predict the next state, $s'$ based on the current $(s, a)$ pairs. all my neural network inputs are between 0 and 1 and the loss function for this network ...
2
votes
0answers
34 views

How was the DQN trained to play many games?

Some people claim that DQN was used to play many Atari games. But what actually happened? Was DQN trained only once (with some data from all games) or was it trained separately for each game? What was ...
2
votes
0answers
139 views

How to formulate normalization/probability conditions on state-action spaces in Gym?

I intend to develop a custom environment for open-ai's gym. My goal is for an agent to learn (among additional objectives) dividing a certain quantity drawn from a continous action space (i.e. spaces....
2
votes
0answers
27 views

How should I define the reward function for the Connect Four game?

I'm creating an RL application for the game Connect Four. I've researched the different strategies for the game and which positions are more favourable to lead to a win. Should I be assigning ...
2
votes
0answers
107 views

Whats the correct loss function to use during deep Q-learning (discrete action space)

After playing around with normal Q-learning I have decided to switch to deep Q-learning and I have encountered this problem. As I understand, for a task with discrete action space, where there are 4 ...
2
votes
0answers
46 views

Why is my DQN model not getting better?

I've created a deep Q network. My model does not get better, and can't see what I'm doing wrong. I'm new to RL. Replay Memory ...
2
votes
0answers
87 views

Policy Gradient Reward Oscillation in MATLAB

I'm trying to train a Policy Gradient Agent with Baseline for my RL research. I'm using the in-built RL toolbox from MATLAB (https://www.mathworks.com/help/reinforcement-learning/ug/pg-agents.html) ...
2
votes
0answers
33 views

How to draw backup diagram when reward is in expectation but next state is iterated?

I am working through Sutton and Barto's RL book. So far in the text, when backup diagrams are drawn, the reward and next state are iterated together (i.e. the equations always have $\sum_{s',r}$), ...
2
votes
0answers
46 views

Should the RL agent be trained in an environment with real-world data or with a synthetic model?

I want to train a reinforcement learning agent in an environment with parameters (for example, the wind speed, sun irradiation, etc.) that change over time. I have recorded a limited amount of data ...
2
votes
1answer
91 views

How can I implement the reward function for an 8-DOF robot arm with TRPO?

I need to get an 8-DOF (degrees of freedom) robot arm to move a specified point. I need to implement the TRPO RL code using OpenAI gym. I already have the gazebo environment. But I am unsure of how to ...
2
votes
0answers
130 views

How does adding noise to the action in DDPG help in learning?

I can't understand how playing with the action generated by the actor network in DDPG by adding the noise term helps in exploration.
2
votes
1answer
222 views

Why isn't my implementation of DQN using TensorFlow on the FrozenWorld environment working?

I am trying to test DQN on FrozenWorld environment in gym using TensorFlow 2.x. The update rule is (off policy) $$Q(s,a) \leftarrow Q(s,a)+\alpha (r+\gamma~ max_{a'}Q(s',a')-Q(s,a))$$ I am using an ...
2
votes
0answers
73 views

Formulation of a Markov Decision Process Problem

Given a list of $N$ questions. If question $i$ is answered correctly (given probability $p_i$), we receive reward $R_i$; if not the quiz terminates. Find the optimal order of questions to maximize ...
2
votes
0answers
20 views

What effect does increasing the actions in RL have?

Consider a 2D snake game, where the snake has to eat food to become longer. It must avoid hitting walls and biting into her tail. Such a game could have a different amount of actions: 3 actions: go ...
2
votes
0answers
139 views

How does the update rule for the one-step actor-critic method work?

Can you please elucidate the math behind the update rule for the critic? I've seen in other places that just a squared distance of $R + \hat{v}(S', w) - \hat{v}(S,w)$ is used, but Sutton suggests an ...
2
votes
0answers
48 views

Can the importance sampling estimator have a non-stationary behaviour policy even if the target policy is stationary?

The inverse propensity score (IPS) estimator, which is used for off-policy evaluation in a contextual bandit problem, is well explained in the paper Doubly Robust Policy Evaluation and Optimization. ...
2
votes
0answers
39 views

Why is this Monte Carlo approach scalable for a growing number of states variables and action variables?

I am reading a research paper on the formulation of MDP problems to ICU treatment decision making: Treatment Recommendation in Critical Care: A Scalable and Interpretable Approach in Partially ...
2
votes
1answer
100 views

NoisyNet DQN with default parameters not exploring

I implemented a DQN algorithm that plays OpenAIs Cartpole environment. The NN architecture consists of 3 normal linear layers that encode the state, and one noisy linear layer, that predicts the Q ...
2
votes
0answers
42 views

Can we use imitation learning for on-policy algorithms?

Imitation learning uses experiences of an (expert) agent to train another agent, in my understanding. If I want to use an on-policy algorithm, for example, Proximal Policy Optimization, because of it'...
2
votes
0answers
36 views

What are some approaches to estimate the transition and observation probabilities in POMDP?

What are some common approaches to estimate the transition or observation probabilities, when the probabilities are not exactly known? When realizing a POMDP model, the state model needs additional ...
2
votes
0answers
41 views

Should I consider mean or sampled value for action selection in ppo algorithm?

When considering the policy network in PPO algorithm, we need to fit a Gaussian distribution to the neural network output (for a continuous action space problem). When I use this network to obtain ...
2
votes
0answers
48 views

What form of output would be needed to train a model on a connect 4 AI?

I've had a big interest in machine learning for a while, and I've followed along a few tutorials, but have never made my own project. After losing many games of connect 4 with my friends, I decided to ...
2
votes
0answers
67 views

How would one develop an action space for a game that is proprietary?

I'm currently trying to develop an RL that will teach itself to play the popular fighting game "Tekken 7". I initially had the idea of teaching it to play generally- against actual opponents with ...
2
votes
0answers
40 views

How to set the multiple continuous actions with constraints

I want to build a Deep Reinforcement Learning Model for Asset allocation. Background: I have 7 stock indexes from different markets, and I want to build a policy to produce the action (likes whether ...
2
votes
0answers
32 views

What could be the cause of the drop of the total reward when using DQN to solve the cart-pole environment?

I'm trying to use DQN to solve the cart-pole environment. I have 2 networks (target and behavior). Both of them have 3 hidden layers with 24 neurons, using the ReLU activation. The loss is MSE and the ...
2
votes
0answers
24 views

Finding total number of states in a POMDP

I've been working on a question that is posed in a document I've been reading, that models qualifying for a job as a POMDP. In this model, a person takes 3 exams, and must pass all of them in order to ...
2
votes
0answers
38 views

How to learn using DDPG in python solely using a timeseries datasets

I have a lengthy timeseries datasets which contains several variables (from sensors etc) to be classified as actions or states. Providing they are successfully done, I want to learn a control policy ...
2
votes
0answers
54 views

Is there any open source implementation of the SBEED learning algorithm?

Are there are any openly available implementations of the SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation paper?
2
votes
0answers
34 views

Best approach for online Machine Translation with few hundred of samples?

I want to implement a model that improves itself with the passage of time. My main task is to build a machine translator (from English to Urdu).. The problem I am facing is that I have very little ...
2
votes
0answers
175 views

how to use Softmax action selection algorithm in atari-like game

I'm currently writing a program using keras (python 3) to play a game similar to Atari games, only in this one there are objects moving in the screen in different angles and directions (in most of ...
2
votes
0answers
25 views

Loss reduction, but constant performance with CNN

I made a CNN with a reasonable loss curve, but the performance of the model does not improve. I have tried making the model larger, I am using three convolutional layers with batch norms. Thanks for ...
2
votes
0answers
53 views

Torch CNN not training

I am completely new to CNN's, and I do not quite know how to design or use them efficiently. That being said, I am attempting to build a CNN that learns to play Pac-man with reinforcement learning. I ...
2
votes
0answers
29 views

Developmental systems that try to explain or understand the reward value in the reinforcement learning?

Are there methods (possibly logical or (how they are called in the literature) relational) that allows for the developmental systems to understand or explain the value of the received reward during ...
2
votes
0answers
43 views

How to show Monte Carlo methods converge to an estimate which minimizes mean squared error?

In chapter six of Sutton and Barto (p.128), they claim Monte Carlo methods converge to an estimate minimizing the mean squared error. How can this be shown formally? Bump
2
votes
0answers
80 views

Is it possible to use Reward Function of type R(s, a, s') if more than one action is applied?

I am applying a reinforcement learning agent (PPO2, stable baselines implementation) to a custom built environment using OpenAI Gym. One reward function (formualted as loss function, that is, all ...
2
votes
0answers
81 views

What is a high performing network architecture to use in a PPO2 MlpLnLstmPolicy RL model?

I am playing around with creating custom architectures in stable-baselines. Specifically I am training an agent using a PPO2 model. My question is, are there some rules of thumb or best practices in ...
2
votes
0answers
21 views

Why do these reward functions give different training curves?

Let's say our task is to pick and place a block, like: https://gym.openai.com/envs/FetchPickAndPlace-v0/ Reward function 1: -1 for block not placed, 0 for block placed Reward function 2: 0 for block ...
2
votes
0answers
45 views

How to solve optimal control problem with reinforcement learning

The problem I am trying to attack is a predator-prey pursuit problem. There are multiple predators that pursue multiple preys and preys tried to evade predators. I am trying to solve a simplified ...
2
votes
0answers
148 views

Coloring graphs with reinforcement learning

I am trying to build an RL agent to solve the NP-hard problem graph coloring. The problem is quite challenging. This how I addressed it. The environment To preserve the scalability of the ...
2
votes
0answers
108 views

What is ratio of the objective function in the case of continuous action spaces?

I'm trying to implement the proximal policy optimization (PPO) algorithm. I'm confused on how to make it work with continuous action space. For discrete action space, the output of the network is the ...
2
votes
0answers
57 views

Will the RL agent implemented as a neural network fine-tune itself?

Normally, when you develop a neural network, train it for object recognition (on normal objects like bike, car, plane, dog, cloud, etc.), and it turns out to perform very well, you would like to fine-...
2
votes
0answers
91 views

How does Friend-or-Foe Q-learning intuitively work?

I read about Q-Learning and was reading about multi-agent environments. I tried to read the paper Friend-or-Foe Q-learning, but could not understand anything, except for a very vague idea. What does ...
2
votes
0answers
44 views

Does coarse coding with radial basis function generate fewer features?

I am learning about discretization of the state space when applying reinforcement learning to continuous state space. In this video the instructor, at 2:02, the instructor says that one benefit of ...
2
votes
0answers
329 views

Gym dict space as keras DQN agent input

I'm trying to make an AI to play my own card game. I have an OpenAI gym for the game with Dict as an observation space. It is nested dict, so I can't easily replace it with a tuple. I want to pass ...
2
votes
0answers
47 views

Inverse Reinforcement Learning for Markov Games

This is an Inverse Reinforcement Learning (IRL) problem. I have data (observations) on actions taken by a (real) agent. Given this data I want to estimate the likelihood of the observed actions in a Q-...
2
votes
0answers
87 views

Infinite horizon in Reinforcement Learning

I read this article: "Towards Autonomous Data Ferry Route Design through Reinforcement Learning" by Daniel Henkel and Timothy X Brown. It specifies an infinite horizon problem where they use as a ...
2
votes
0answers
53 views

Reinforcement Learning in Real Life/Practical Terms

In every day life, it seems that we all have various habits and actions that we perform. For example, we wake up and check our email/facebook etc. on our phones. We don't look at are current state ...

1 2 3
4
5
13