Questions tagged [reinforcement-learning]

For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.

569 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
2
votes
0answers
52 views

How are the classical MDP and the object-oriented MDP views different?

I've been reading the attached paper - which aims to model entities in the world as objects, including the learning agent itself! To say the least, the goal is to navigate through what seems like a ...
2
votes
1answer
59 views

How can I sample the output distribution multiple times when pruning the filters with reinforcement learning?

I was reading the paper Learning to Prune Filters in Convolutional Neural Networks, which is about pruning the CNN filters using reinforcement learning (policy gradient). The paper says that the input ...
2
votes
0answers
31 views

How should I design a reward function for a NLP problem where two models interoperate?

I would like to design a reward function. I am training two models from the first model that classify set of texts (paragraphs and keywords) and I also got some hidden states. The second model is ...
2
votes
0answers
48 views

Action masking for on policy algorithm like PPO

I have an environment, in which my agent learns according to PPO. The environment has a maximum of 80 actions, however not all of them are always allowed. My idea was to mask them, by setting the ...
2
votes
0answers
30 views

How can I formalise a non-zero-sum game of $N$ agent as Markov game?

I coded a non-zero-sum game of $N$ agents in a discrete dynamic environment to RL with Q-learning and DQN agents. It's like a marathon. Only two actions are available per agent: $\{ G \text{ (move ...
2
votes
0answers
45 views

Is Mean Squared Error Loss function a good loss function for continuous variables $0 < x < 1$

Suppose I am utilising a neural network to predict the next state, $s'$ based on the current $(s, a)$ pairs. all my neural network inputs are between 0 and 1 and the loss function for this network ...
2
votes
0answers
34 views

How was the DQN trained to play many games?

Some people claim that DQN was used to play many Atari games. But what actually happened? Was DQN trained only once (with some data from all games) or was it trained separately for each game? What was ...
2
votes
0answers
93 views

How to formulate normalization/probability conditions on state-action spaces in Gym?

I intend to develop a custom environment for open-ai's gym. My goal is for an agent to learn (among additional objectives) dividing a certain quantity drawn from a continous action space (i.e. spaces....
2
votes
0answers
26 views

How should I define the reward function for the Connect Four game?

I'm creating an RL application for the game Connect Four. I've researched the different strategies for the game and which positions are more favourable to lead to a win. Should I be assigning ...
2
votes
0answers
22 views

What are the most common non-Markov RL paradigms?

I am interested in doing model-free RL but not using the Markov assumptions typical for MDPs or POMDPs. What are alternative paradigms that don't rely on the Markov assumptions? Are there any common ...
2
votes
0answers
349 views

How much time does it take to train DQN on Atari environment?

I am trying to build a DQN model for the Atari Pong game, but I am not sure whether the model is learning at all. I am using the architecture described in the paper Playing Atari with Deep ...
2
votes
0answers
49 views

Whats the correct loss function to use during deep Q-learning (discrete action space)

After playing around with normal Q-learning I have decided to switch to deep Q-learning and I have encountered this problem. As I understand, for a task with discrete action space, where there are 4 ...
2
votes
0answers
41 views

Why is my DQN model not getting better?

I've created a deep Q network. My model does not get better, and can't see what I'm doing wrong. I'm new to RL. Replay Memory ...
2
votes
0answers
80 views

Policy Gradient Reward Oscillation in MATLAB

I'm trying to train a Policy Gradient Agent with Baseline for my RL research. I'm using the in-built RL toolbox from MATLAB (https://www.mathworks.com/help/reinforcement-learning/ug/pg-agents.html) ...
2
votes
0answers
29 views

How to draw backup diagram when reward is in expectation but next state is iterated?

I am working through Sutton and Barto's RL book. So far in the text, when backup diagrams are drawn, the reward and next state are iterated together (i.e. the equations always have $\sum_{s',r}$), ...
2
votes
0answers
44 views

Should the RL agent be trained in an environment with real-world data or with a synthetic model?

I want to train a reinforcement learning agent in an environment with parameters (for example, the wind speed, sun irradiation, etc.) that change over time. I have recorded a limited amount of data ...
2
votes
1answer
78 views

How can I implement the reward function for an 8-DOF robot arm with TRPO?

I need to get an 8-DOF (degrees of freedom) robot arm to move a specified point. I need to implement the TRPO RL code using OpenAI gym. I already have the gazebo environment. But I am unsure of how to ...
2
votes
0answers
75 views

How does adding noise to the action in DDPG help in learning?

I can't understand how playing with the action generated by the actor network in DDPG by adding the noise term helps in exploration.
2
votes
1answer
169 views

Why isn't my implementation of DQN using TensorFlow on the FrozenWorld environment working?

I am trying to test DQN on FrozenWorld environment in gym using TensorFlow 2.x. The update rule is (off policy) $$Q(s,a) \leftarrow Q(s,a)+\alpha (r+\gamma~ max_{a'}Q(s',a')-Q(s,a))$$ I am using an ...
2
votes
0answers
66 views

Formulation of a Markov Decision Process Problem

Given a list of $N$ questions. If question $i$ is answered correctly (given probability $p_i$), we receive reward $R_i$; if not the quiz terminates. Find the optimal order of questions to maximize ...
2
votes
0answers
20 views

What effect does increasing the actions in RL have?

Consider a 2D snake game, where the snake has to eat food to become longer. It must avoid hitting walls and biting into her tail. Such a game could have a different amount of actions: 3 actions: go ...
2
votes
0answers
105 views

How does the update rule for the one-step actor-critic method work?

Can you please elucidate the math behind the update rule for the critic? I've seen in other places that just a squared distance of $R + \hat{v}(S', w) - \hat{v}(S,w)$ is used, but Sutton suggests an ...
2
votes
0answers
44 views

Can the importance sampling estimator have a non-stationary behaviour policy even if the target policy is stationary?

The inverse propensity score (IPS) estimator, which is used for off-policy evaluation in a contextual bandit problem, is well explained in the paper Doubly Robust Policy Evaluation and Optimization. ...
2
votes
0answers
39 views

Why is this Monte Carlo approach scalable for a growing number of states variables and action variables?

I am reading a research paper on the formulation of MDP problems to ICU treatment decision making: Treatment Recommendation in Critical Care: A Scalable and Interpretable Approach in Partially ...
2
votes
0answers
67 views

Why I got the same action when I train A2C when I increase the number of episodes?

I'm working on an advantage actor-critic (A2C) reinforcement learning model but the problem when I trained the system for 3500 episodes, I start to get the same action for all my testing results. ...
2
votes
0answers
43 views

Reinforcement learning CNN input weakness

I'm trying to train a network to navigate a 48x48 2D grid, and switch pixels from on to off or off to on. The agent receives a small reward if correct, and small punishment if incorrect pixel plotted. ...
2
votes
0answers
23 views

Why we multiply probabilities with support to obtain Q-values in Distributional C51 algorithm?

In 'Deep Reinforcement Learning Hands-On' book and chapter about Distributional C51 algorithm I'm reading, that to obtain Q-values from the distribution I need to calculate the weighted sum of the ...
2
votes
1answer
88 views

NoisyNet DQN with default parameters not exploring

I implemented a DQN algorithm that plays OpenAIs Cartpole environment. The NN architecture consists of 3 normal linear layers that encode the state, and one noisy linear layer, that predicts the Q ...
2
votes
0answers
41 views

Can we use imitation learning for on-policy algorithms?

Imitation learning uses experiences of an (expert) agent to train another agent, in my understanding. If I want to use an on-policy algorithm, for example, Proximal Policy Optimization, because of it'...
2
votes
0answers
35 views

What are some approaches to estimate the transition and observation probabilities in POMDP?

What are some common approaches to estimate the transition or observation probabilities, when the probabilities are not exactly known? When realizing a POMDP model, the state model needs additional ...
2
votes
0answers
26 views

Flattened vector observation or convolutional neural network input?

This is more of a general question of how to model/preprocess 'visual' state-observations to an Agent in Reinforcement Learning that I'll illustrate with an example. Say you have a reinforcement ...
2
votes
0answers
61 views

Reinforcement learning possible with big action space?

I’m experimenting with reinforcement learning for a 2D pixel plotting task, and am running into an issue that (I think) has to do with the big action space. It goes like this: The Agent gets two ...
2
votes
0answers
37 views

Should I consider mean or sampled value for action selection in ppo algorithm?

When considering the policy network in PPO algorithm, we need to fit a Gaussian distribution to the neural network output (for a continuous action space problem). When I use this network to obtain ...
2
votes
0answers
48 views

What form of output would be needed to train a model on a connect 4 AI?

I've had a big interest in machine learning for a while, and I've followed along a few tutorials, but have never made my own project. After losing many games of connect 4 with my friends, I decided to ...
2
votes
0answers
62 views

How would one develop an action space for a game that is proprietary?

I'm currently trying to develop an RL that will teach itself to play the popular fighting game "Tekken 7". I initially had the idea of teaching it to play generally- against actual opponents with ...
2
votes
1answer
86 views

Can supervised learning be used to solve the inverted pendulum problem?

I know that reinforcement learning has been used to solve the inverted pendulum problem. Can supervised learning be used to solve the inverted pendulum problem? For example, there could be an ...
2
votes
0answers
37 views

How to set the multiple continuous actions with constraints

I want to build a Deep Reinforcement Learning Model for Asset allocation. Background: I have 7 stock indexes from different markets, and I want to build a policy to produce the action (likes whether ...
2
votes
0answers
30 views

What could be the cause of the drop of the total reward when using DQN to solve the cart-pole environment?

I'm trying to use DQN to solve the cart-pole environment. I have 2 networks (target and behavior). Both of them have 3 hidden layers with 24 neurons, using the ReLU activation. The loss is MSE and the ...
2
votes
0answers
21 views

Finding total number of states in a POMDP

I've been working on a question that is posed in a document I've been reading, that models qualifying for a job as a POMDP. In this model, a person takes 3 exams, and must pass all of them in order to ...
2
votes
0answers
28 views

Same implementation, but agent is not learning in Retro Pong Environment

I tried to implement the exact same python coding by Andrej Karpathy to train RL agent to play Pong, except that I migrated the environment from Gym to Retro. Everything is the same except the action ...
2
votes
0answers
31 views

How to learn using DDPG in python solely using a timeseries datasets

I have a lengthy timeseries datasets which contains several variables (from sensors etc) to be classified as actions or states. Providing they are successfully done, I want to learn a control policy ...
2
votes
0answers
34 views

Best approach for online Machine Translation with few hundred of samples?

I want to implement a model that improves itself with the passage of time. My main task is to build a machine translator (from English to Urdu).. The problem I am facing is that I have very little ...
2
votes
0answers
137 views

how to use Softmax action selection algorithm in atari-like game

I'm currently writing a program using keras (python 3) to play a game similar to Atari games, only in this one there are objects moving in the screen in different angles and directions (in most of ...
2
votes
0answers
25 views

Loss reduction, but constant performance with CNN

I made a CNN with a reasonable loss curve, but the performance of the model does not improve. I have tried making the model larger, I am using three convolutional layers with batch norms. Thanks for ...
2
votes
0answers
52 views

Torch CNN not training

I am completely new to CNN's, and I do not quite know how to design or use them efficiently. That being said, I am attempting to build a CNN that learns to play Pac-man with reinforcement learning. I ...
2
votes
0answers
29 views

Developmental systems that try to explain or understand the reward value in the reinforcement learning?

Are there methods (possibly logical or (how they are called in the literature) relational) that allows for the developmental systems to understand or explain the value of the received reward during ...
2
votes
0answers
43 views

How to show Monte Carlo methods converge to an estimate which minimizes mean squared error?

In chapter six of Sutton and Barto (p.128), they claim Monte Carlo methods converge to an estimate minimizing the mean squared error. How can this be shown formally? Bump
2
votes
0answers
73 views

Is it possible to use Reward Function of type R(s, a, s') if more than one action is applied?

I am applying a reinforcement learning agent (PPO2, stable baselines implementation) to a custom built environment using OpenAI Gym. One reward function (formualted as loss function, that is, all ...
2
votes
0answers
73 views

What is a high performing network architecture to use in a PPO2 MlpLnLstmPolicy RL model?

I am playing around with creating custom architectures in stable-baselines. Specifically I am training an agent using a PPO2 model. My question is, are there some rules of thumb or best practices in ...
2
votes
0answers
21 views

Why do these reward functions give different training curves?

Let's say our task is to pick and place a block, like: https://gym.openai.com/envs/FetchPickAndPlace-v0/ Reward function 1: -1 for block not placed, 0 for block placed Reward function 2: 0 for block ...

1 2 3
4
5
12