Questions tagged [reinforcement-learning]
For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.
504
questions with no upvoted or accepted answers
0
votes
0answers
41 views
What does the notation “for t=T to 1,−1 do” in terms of time steps, in deep recurrent q network?
In looking at an algorithm in the paper Learning to Communicate with Deep Multi-Agent Reinforcement Learning.
Here is the full algorithm:
What does the notation ...
0
votes
0answers
31 views
DQN Tic-Tac-Toe does not quite become optimal
I am trying to implement a DQN agent for playing standard 3x3 Tic-Tac-Toe (it is a double DQN with experience replay, and using a target network). I got the hyperparameters to the point
where the ...
0
votes
0answers
46 views
Why is my DDPG agent (implemented in TensorFlow) not learning?
I am trying to implement a Reinforcement Learning algorithm called DDPG in TensorFlow 2.x on a custom gym environment. I am new to TF. So, I started with the DDPG TF 1.x implementation from pemami4911....
0
votes
0answers
31 views
Generalizability of Generative Adversarial Imitation Learning (GAIL) method
I have something would like to clarify regarding Generative Adversarial Imitation Learning (GAIL). Is the original GAIL applicable if the experts trajectories (sample data) are for the same task but ...
0
votes
0answers
34 views
Why doesn't my double deep Q network trained with the same training set give consistent performance?
I've written a Double DQN which can do either 1-step or multi-step learning. It also has a prioritised reply buffer. The internal network is an LSTM.
My input data is a series of time series data and ...
0
votes
1answer
43 views
Is there a good website where I can learn about Deep Deterministic Policy Gradient?
Is there a good website where I can learn about Deep Deterministic Policy Gradient?
0
votes
0answers
31 views
How is centralised training and decentralised execution in multi agent reinforcement learning implemented?
In the paper Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learning, it is written
We allow centralised training but require decentralised execution, from which follows that the policies ...
0
votes
0answers
20 views
Optimal RL agent's representation of 3D-grid data: 2D Slices and CNN encoding. Suggestions?
environment
My agent needs to navigate in a 64x64x64 discrete 3d grid environment, and remove certain voxels. Voxels can be in a number of states: should-remove, should-not-remove, empty. It can move ...
0
votes
0answers
31 views
Reinforcement Learning Diagnostic: Total reward doesn't converge
I'm implementing DDQN in my toy scenario. During training, I'm surprised to see that the total reward doesn't converge and have a tendency to degrade. What could be the problem?
Here's the picture: ...
0
votes
0answers
47 views
Why care about the value of the action which I'm not gonna take in policy iteration?
In this article, there is an explanation (with an example) of how policy iteration works.
It seems that, if we replace all the probabilities of moves in the example by new probabilities where the best ...
0
votes
1answer
85 views
How do I test an LSTM-based reinforcement learning model using any Atari games in OpenAI gym?
I am writing a couple of different reinforcement learning models based on Rainbow DQN or some PG models. All of them internally use an LSTM network because my project is using time series data.
I ...
0
votes
1answer
51 views
In Deep Deterministic Policy Gradient, are all weights of the policy network updated with the same or different value?
I'm trying to understand the DDPG algorithm shown at this page. I don't know what should the result of the gradient at step 14 be.
Is it a scalar that I have to use to update all the weights (so all ...
0
votes
0answers
24 views
How to deal with GAE ineffectiveness because of critic value adaptation?
I've noticed if you have a small negative reward (e.g.,-0.01) per step for idling and a series of idle steps, an agent seems to learn to trick GAE by learning a ...
0
votes
0answers
29 views
Patient PPO: how to handle imbalanced discrete action space?
PPO agent. The action space includes 3 actions:
0: do nothing
1: act (start)
2: stop
The agent has to perform thousands of steps doing nothing, then perform step 1 only once (act), then do nothing ...
0
votes
1answer
65 views
How can I model this problem of delivering assets by choosing a route with reinforcement learning?
I would like to build a model based on reinforcement learning (RL) for the following scenario
Recommend the best route (of cities listed for a given country) that satisfies the required criteria (...
0
votes
0answers
40 views
Using deep deterministic policy gradient in OpenAI Gym to solve problems with continuous actions
I am trying to do the following:
Install the OpenAI baseline algorithms from the following GitHub repository: github.com/openai/baselines by following the instructions in the readme file.
Train an ...
0
votes
0answers
44 views
Model Based rl and cross entropy method with nonlinear function approximators
Pseudo code for Cross entropy method according to youtube lecture 32:55
Initialize $\mu \in R^{d}, \sigma \in R^{d}$
iteration 1,2,...
Collect n samples of $\theta_{i} \sim N(\mu,diag(\sigma))$
...
0
votes
0answers
84 views
Problem in understanding equation given for convergence of TD(n) algorithm
Given equation 7.3 of Sutton and Barto's book for convergence of TD(n):
$\max_s|\mathbb{E}_\pi[G_{t:t+n}|S_t = s] - v_\pi(s)| \leqslant \gamma^n \max_s|V_{t+n-1}(s) - v_\pi(s)|$
$\textbf{PROBLEM ...
0
votes
0answers
70 views
TD-Leaf struggles at learning chess
I am currently working on implementing Giraffe chess algorithm. Following this paper, I designed a neural network similar to the one proposed by the author which I trained using TD-Leaf(lambda). The ...
0
votes
0answers
30 views
My Double DQN with Experience Replay produces a no-action decision most of the time. Why?
I've written a Double DQN-based stock trading bot using mainly time series stock data. The internal network of the Double DQN is a LSTM which handles the time series data. An Experience Replay buffer ...
0
votes
0answers
32 views
Can Reinforcement Learning be used for UAV waypoint control?
I want to make a drone which can follow static and dynamic waypoints. I am a total beginner in the drone field so I can't figure out that should I use Reinforcement Learning or any other learning ...
0
votes
0answers
35 views
Why in RL function approximators with recurrent structures can learn planning?
In the paper An Investigation of Model-Free Planning the authors use ConvLSTM to learn a planning function. In particular, for each input x_t at time-step ...
0
votes
0answers
82 views
Should the network weights converge when training Deep Q networks?
I have two sets of data, one training and one test set. I use the train set to train the deep q network model variant. I also continuously evaluate the agent Q values obtained on the test set every ...
0
votes
0answers
61 views
How to choose hyperparameters in double DQN?
I'm looking for some indications about the tuning of hyperparameters in building my double DQN. I have a time series problem (with about 2000 observations and no terminal state, I have to max the ...
0
votes
0answers
35 views
Is there any advantage to using a non-diagonal covariance matrix for a policy distribution?
For reinforcement learning implementations with a gym.spaces.Box action space, which is the product of $k$ real closed intervals, it is common (actually more like ...
0
votes
1answer
47 views
Why aren’t heuristics for Connect Four Monte Carlo tree search improving the agent?
I’ve created an agent using MCTS to play Connect Four. It wins against humans pretty well, but I’d like to improve upon it. I decided to add domain knowledge to the MCTS rollout stage. My evaluation ...
0
votes
0answers
18 views
How is the parameterised server updated in distributed DQN?
In this paper about Massively Parallel Methods for Deep Reinforcement Learning, the parallelisation of DQN is done via separating the actors and learners. Multiple actors carry out the $\epsilon$ ...
0
votes
0answers
30 views
How to observe or measure convergence of Monte Carlo Tree Search?
As above: how does one observe/measure a Monte Carlo Tree Search to be able to update the algorithm and compare results?
0
votes
0answers
112 views
OpenAI Gym: Multiple actions in one step
I'm trying to design an OpenAI Gym environment in which multiple users/players perform actions over time. It's round based and each user needs to take an action before the round is evaluated and the ...
0
votes
0answers
20 views
When has RL helped in building Recommender Systems?
I was wondering if it'd be possible to list some or all the instances wherein Reinforcement Learning has been used to build Recommender Systems - here's one paper I've already come across, and I found ...
0
votes
0answers
52 views
Reinforcement learning random agent always performing the same few actions
I have a DQN model which has 3 features as inputs (properly normalized) and should output q-values for each of the 100 possible actions. However, prior to any training, I would like to examine the ...
0
votes
0answers
56 views
Why is diversity of reasoning paths important in recommender systems using knowledge graphs?
This is a continuation of the discussion that originates on this StackExchange post, about recommender systems using knowledge graphs(KGs). For those who might not prefer reading the original post, I ...
0
votes
0answers
17 views
Won't the copy of the weights of the worker model to the global model erase the work of other workers in A3C?
I was reading the article Deep Reinforcement Learning: Playing CartPole through Asynchronous Advantage Actor-Critic (A3C) with tf.keras and eager execution.
From my understanding, we copy the weights ...
0
votes
0answers
21 views
In RL can you use a random sample from the state as observation for the agent?
Could reinforcement learning work in the following context :
Given that the initial state space is very large (10^6) and the actions would only effect a subspace of the state could we randomly select ...
0
votes
0answers
31 views
Why would you ignore episodes that loop back on the starting state in MCTS?
After reading about MCTS for policy learning and optimization, I don't understand why you would want to ignore episodes that loop back on the starting state. What advantage does this have and why ...
0
votes
0answers
22 views
DQN is unable to learn from image data
I am trying to write a DQN model that will be able to solve OpenAI gym CartPole environment. I successfully managed to do it using scalar observation data that ...
0
votes
0answers
35 views
How should I define the loss function when using DQN to estimate the probability density?
I'm doing a Deep Q-learning project. All of my rewards are positive and there are two terminal states. One of them has a zero reward and the other has a high positive reward.
The rewards are ...
0
votes
1answer
36 views
Non-Neural Network algorithms for large state space in zero sum games
I was reading online that Tic Tac Toe has a state space of 3^9 = 19,683. From my basic understanding, this sounds too large to use with Q Learning, as the Q table would be huge?
If that is the case, ...
0
votes
0answers
48 views
How to understand and visualize a trained RL agent's policy when the state space is high dimensional?
What are typical ways to understand and visualize a trained RL agent's policy when the state space is of high dimension (but not images)?
For example, suppose state and action are denoted by $s=(...
0
votes
0answers
23 views
Stable baseline learn from online game
i created a custom environment for an online game.
i want to know how can i train my model with stable baseline because learn function in stable baseline just take number of steps and in online game ...
0
votes
0answers
44 views
Understanding V- and Q-functions
Assume the existence of a Markov Decision Process consisting of:
State space $S$
Action space $A$
Transition model $T: S \times A \times S \to [0,1]$
Reward function $R: S \times A \times S \to \...
0
votes
0answers
25 views
Which hidden state should I use for a trajectory when incorporating LSTM into RL?
I'm trying to wrap my head around using LSTM in an RL algorithm like actor-critic or PPO. I've found this Github code which presents this in a very simple manner, however I have a very limited ...
0
votes
0answers
23 views
Curiosity Driven Learning affect optimal policy
I am trying to understand some of the different approaches used to overcome sparse rewards in a reinforcement learning setting for a research project. Particularly, I have looked at curiosity driven ...
0
votes
0answers
24 views
Concrete examples of models and policies in Tic Tac Toe environment
I'm having difficulty picturing how models and policies are represented. Could someone describe how they would look in the context/environment of a game of Tic Tac Toe? For example, "In Tic Tac Toe, ...
0
votes
0answers
20 views
How to pass observation from CartPole-v0 to neural network using tensorflow
I am trying to learn about RL by implementing DQN with tensorflow. However, I am really stuck with tensorflow.. I just don't understand it.
I think I have found the core of what I understand - I dont ...
0
votes
0answers
27 views
Can someone explain how and why actor-critic networks are created this way?
Deep Deterministic Policy Gradients (DDPG) and stable Baseline Code is presented here.
The actor-critic networks are created as follows:
...
0
votes
0answers
22 views
Ideas on how to train an AI to play Mario Kart with the DeSmuME Emulator
Gday guys,
i have this idea in my mind for quite a while. I want to teach an AI to play Mario Kart on the NDS, which can hopefully beat me and my friends one day. Iam familiar with the theoretical ...
0
votes
0answers
47 views
DQN unlearns certain OpenAI-Gym environments
I solved the OpenAI-Gym MountainCar-v0 environment using dqn(using low-state-dimensional input). When I used the same code for solving CartPole-v0 environment, the network got trained in the reverse ...
0
votes
0answers
31 views
Are there real-world problems where case-based reasoning is not suitable?
I know case-based reasoning has four stages: retrieve, retain, re-use and revise.
Used for solving new problems by adapting solutions that were used to solve old problems, like car issues.
The ...
0
votes
1answer
574 views
Reinforcement learning with PPO: rewards decreasing
I'm trying to train a PPO agent and my average rewards graph looks like this. Could this indicate that it's stuck at a local maximum? Do I need to promote exploring by increasing the entropy or does ...
