Questions tagged [reinforcement-learning]
For questions related to learning controlled by external positive reinforcement or negative feedback signal or both, where learning and use of what has been thus far learned occur concurrently.
0
votes
0answers
11 views
How to train a model by accounting for boundary constraints?
I've a robot traverse through a grid layout. Based on the wheel speed difference I classify actions as either straight, left or right. I computed the distances based on the time duration and the speed ...
1
vote
1answer
34 views
Reinforcement Learning (RL) how to obtain $p(s',r|s,a)$
I am trying to study the book Reinforcement Learning: An Introduction (Sutton & Barto, 2018). In chapter 3.1 the authors state the following exercise
Exercise 3.5 Give a table analogous to that ...
0
votes
0answers
8 views
Creating a Parkour agent using Deep Reinforcement Learning
How do I go about creating a Parkour agent which uses Deep RL. I have considered one approach wherein I can learn complex maneuvers using Imitation Learning (something like DeepMimic or GAIL paper). ...
1
vote
1answer
24 views
Some RL algorithms (especially policy gradients) initialize with random policies, which often manifests as random jitter on spot for a long time?
I am reviewing a statement on the website for ES regarding structured exploration.
https://blog.openai.com/evolution-strategies/
Structured exploration. Some RL algorithms (especially policy
...
1
vote
1answer
50 views
Reward-related formulation in reinforcement learning
I am referring to eq. 3.6 (p/g 49) based on Sutton's online book and can be found in an image below. I could not make sense of the final derivation of the equation $r(s, a, s')$. My question is ...
1
vote
1answer
24 views
When is Markov Decision Process (MDP) not adequate for goal-directed learning tasks
In the book Reinforcement Learning: An Introduction (Sutton & Barto, 2018). The authors ask
Exercise 3.2: Is the MDP framework adequate to usefully represent all goal-directed learning tasks? ...
2
votes
1answer
28 views
Reinforcement Learning (RL) expected reward (Sutton & Barto, 2018)
I am new to RL and I am trying to work through the book Reinforcement Learning: An Introduction I (Sutton & Barto, 2018). In chapter 3 on Finite Markov Decision Processes, the authors write the ...
6
votes
1answer
44 views
Q-Learning the generic maze solution
After doing some exercices on Q-learning for maze solving, I wondered : my q-learning algorithms solve only ONE maze. The AI doesn't learn how to solve mazes, so how can I achieve it ?
For instance ...
5
votes
3answers
202 views
What does the Agent in Reinforcement Learning exactly do?
I am not sure of what the Agent in Reinforcement Learning exactly is. I think it is not the Neural Net behind?
So what is the Agent?
What does the Agent in Reinforcement Learning exactly do?
0
votes
1answer
29 views
How do I know how changes in the weights are changing the reward in Reinforcement Learning
I already know the basics of the basic of Machine Learning. E.g.: Backpropagation, Convolution, etc.
First of let me explain Reinforcement learning to make sure I grasped the concept correctly.
In ...
1
vote
0answers
22 views
How can a fully automated vacuum cleaner use and update room information?
Assistive Subsystems
Consider an automated vacuum cleaner with the following subsystems under the command of an AI system to be designed. These subsystems limit the AI complexity to just intelligent ...
1
vote
2answers
41 views
Understanding LSTM/RNN structure
In keras when we apply LSTM/RNN model, we specify the node [i.e.,LSTM(128)]. I have a doubt how it actually works. From the LSTM/RNN unfolding image or description, I found that each RNN cell take one ...
2
votes
2answers
36 views
What models is Google's quick draw using?
Quick draw is a Google experiment using user generated online doodles and machine learning to play a game of "Guess what I'm drawing" similar to the board game Pictionary.
I'm interested if anyone ...
1
vote
1answer
31 views
Using reinforcement learning to find a preconditioner for linear systems of the form Ax = b
Sparse linear system are normally solved by using solvers like MINRES, Conjugate gradient, GMRES.
Efficient preconditioning, i.e., finding a matrix P such that PAx = Pb is easier to solve then the ...
2
votes
1answer
31 views
Should the reward or the Q value be clipped for reinforcement learning
When extending reinforcement learning to the continuous states, continuous action case, we must use function approximators (linear or non-linear) to approximate the Q-value. It is well known that non-...
2
votes
0answers
33 views
Dyna-Q algorithm, having trouble when adding the simulated experiences
I'm trying to create a simple Dyna-Q agent to solve small mazes, in python. For the Q function, Q(s,a), I'm just using a matrix, where each row is for a state value, and each column is for one of the ...
1
vote
0answers
37 views
Reinforcement learning for segmenting the robot path to reflect the true distances
I've a grid of rectangles acting as blocks. The robot traverses through the inter-spaces between these consecutive blocks. Now I have sensor data streaming in representing Right and left wheel speeds. ...
1
vote
1answer
27 views
In apprenticeship learning, is it possible to outperform the master?
As stated in the title, I'm wondering if it would be possible to "outperform" the master in the apprenticeship learning. I'm aware that the question might be not clear enough; but hopefully, someone ...
3
votes
1answer
57 views
Why do you not see dropout layers on reinforcement learning examples?
I've been looking at reinforcement learning, and specifically playing around with creating my own environments to use with the OpenAI Gym AI. I am using agents from the stable_baselines project to ...
2
votes
1answer
19 views
Dealing with Lags in Reinforcement Learning
Following up on my previous questions using a hypothetical AI system to manage air flow using dampers to achieve an optimal target of exactly equal airflow at a number of vents; (thank you for all ...
0
votes
1answer
22 views
Using a DQN with a variable amount of Valid Moves per turn for a Board Game
I have created a game on an 8x8 grid and there are 4 pieces which can move essentially like checkers pieces (Forward left or Forward right only). I have implemented a DQN in order to pull this off.
...
2
votes
1answer
55 views
PPO / TRPO Implementation
So, I recently watched this video on PPO and want to upgrade my actor-critic algorithm written in PyTorch with PPO, but I'am not sure how the new parameters / ...
1
vote
1answer
54 views
Is it possible to state an outliers detection problem as a reinforcement learning problem?
To me it seems to be ill defined. Partially because of absence of knowledge which points are to be considered outliers in the first place.
The problem which I have in mind is "bad market data" ...
0
votes
0answers
32 views
DQN cannot learn or converge [migrated]
I have implemented a DQN using keras.
The task is to collect the circles and avoid the red circle and crosses.
The associated rewards are +5, -5 and 0 otherwise.
if the agent go out of the board, the ...
2
votes
3answers
192 views
difficulty in understanding identifiability in Dueling Network paper
I have difficulty understanding the following paragraph in bracketed in red parentheses in the below excerpts from page 4 to page 5 from the paper Dueling Network Architectures for Deep Reinforcement ...
1
vote
1answer
32 views
Dependance of Value Function of an MDP on Policy
From what I understand, the value function estimates how 'good' it is for an agent to be in a state and a policy is a mapping of actions to state.
So if I have understood value function and policies ...
1
vote
0answers
12 views
Does inflation should occur in output layer when I do Artificial Neural Network to increase smartness of the model?
The idea that come to my mind is called Value Based Model for ANN. We use simple DCF formula to calculate kind of Q value: Rewards/Discount rate. Discount rate is a risk of getting the reward on the ...
1
vote
1answer
126 views
Does a solution for Wumpus World with neural networks exist?
The Wumpus World proposed in book of Stuart Russel and Peter Norvig, is a game which happens on a 4x4 board and the objective is to grab the gold and avoiding the threats that can kill you. The rules ...
3
votes
1answer
52 views
AlphaZero policy network
In AlphaZero, the policy (head of the) network maps game states to a distribution of the likelihood of taking each more. This distribution covers all possible moves from that state.
How is such a ...
4
votes
2answers
50 views
How can I minimize the number of answers that are relevant to a machine learning model?
Problem:
We have a fairly big database that is built up by our own users.
The way this data is entered is by asking the users 30ish questions that all have around 12 answers (x, a, A, B, C, ..., H). ...
3
votes
2answers
77 views
How can a reinforcement learning agent generalize if it is trained against only one opponent?
I started teaching myself about reinforcement learning a week ago and I have this confusion about the learning experience. Let's say we have the game Go. And we have an agent that we want to be able ...
0
votes
1answer
61 views
What is the physics engine used by DeepMimic?
I found a video for the paper DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
on YouTube.
I looked in the related paper, but could not find details of how to ...
4
votes
2answers
112 views
AlphaZero Value Network
The Alpha Zero (as well as AlphaGo Zero) papers say they trained the value head of the network by "minimizing the error between the predicted winner and the game winner" throughout its many self play ...
3
votes
1answer
64 views
Snake game: snake converges to going in the same direction every time
This is a q-learning snake using a neural network as a q function aproximator and I'm losing my mind here the current model it's worst than the initial one.
The current model uses a 32x32x32 ...
0
votes
0answers
37 views
Is my cart-pole agent behavior expected under these circumstances?
I'm trying to solve the cart-pole problem from Open-AI. For this task I'm using a one step actor-critic algorithm. The purpose of this implementation is mainly to study. For this reason I'm using a ...
0
votes
1answer
70 views
Should Q values be changing within an epoch/episode or should they change after one episode/epoch?
I am trying to use Deep-Q learning environment to learn Super Mario Bros. The implementation is on Github.
I have a neural network that Q values update within an episode for a very small learning ...
5
votes
1answer
60 views
Learning Rate Decay and Exploration Rate Decay
Should I be decaying the learning rate and the exploration rate in the same manner? What's too slow and too fast of an exploration and learning rate decay? Or is it specific from model to model?
8
votes
2answers
190 views
Why is baseline conditional on state at some timestep unbiased?
In robotics, the reinforcement learning technique is used for finding the control pattern for a robot. Unfortunately, most policy gradient method are statistically biased which could bring the robot ...
6
votes
1answer
72 views
Is Experience Replay like dreaming?
Drawing parallels between Machine Learning techniques and a human brain is a dangerous operation. When it is done successfully, it can be a powerful tool for vulgarisation, but when it is done with no ...
3
votes
2answers
371 views
Why is the derivative 0 if the policy is deterministic?
In the Berkeley RL class they mention the gradient would be 0 if the policy is deterministic. Why is that?
https://www.youtube.com/watch?v=XGmd3wcyDg8&feature=youtu.be&t=1071
1
vote
0answers
59 views
How many episodes does it take for a vanilla one-step actor-critic agent to master the OpenAI BipedalWalker-v2 problem?
I'm trying to solve the OpenAI BipedalWalker-v2 by using a one-step actor-critic agent. I'm implementing the solution using python and tensorflow.
I'm following this pseudo-code taken from the book ...
4
votes
1answer
47 views
Why does Clipped Surrogate Objective works in Proximal Policy Optimization
In ''Proximal Policy Optimization Algorithms'' , Schulman et al. (2017), page 3
I don't understand why the clipped surrogate objective works.
As written in the article : "With this scheme, we only ...
1
vote
1answer
32 views
Q Llearning for Shortest distance
Is it possible to form a table that will have simply the shortest distance from each source to destination using q learning.
If not suggest any other learning algorithm
4
votes
2answers
62 views
How to define states in reinforcement learning
I am studying Reinforcement Learning and the variants of it, and I am starting to get an understanding of how the algorithms work and how they apply to an MDP. What I don't understand is the process ...
8
votes
2answers
125 views
How to implement a constrained action space in reinforcement learning?
I'm coding a reinforcement learning model with a PPO agent thanks to the very good Tensorforce library, built on top of Tensorflow.
The first version was very simple and I'm now diving into a more ...
4
votes
1answer
37 views
In imitation learning, do you simply inject optimal (state, action, reward, s(t+1)) experiences into your experience replay buffer?
due to my RL having difficulties learning some control actions, I've decided to use Imitation learning / apprenticeship learning to guide my RL to perform the optimal actions. I've read a few ...
3
votes
1answer
123 views
Tuning of PPO metaparameters: a high level overview of what each parameter does
I am using the PPO algorithm implemented by tensorforce: https://github.com/reinforceio/tensorforce . It works great and I am very happy with the results.
However, I notice that there are many ...
3
votes
1answer
62 views
How does LSTM in deep reinforcement learning differ from experience replay?
In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the author processed the Atari game frames with an LSTM layer at the end. My questions are:
How does this method differ from the ...
3
votes
1answer
24 views
What is the pros and cons of increasing and decreasing the number of worker process in A3C?
In A3C, there are several child processes and one master process. The child precesses calculate the loss and backpropagation, and the master process sums them up and updates the parameters, if I ...
0
votes
0answers
27 views
Open Ai Gym states become corrupt?
I'm trying to implement a3c for flappy bird using this code https://github.com/awjuliani/DeepRL-Agents/blob/master/A3C-Doom.ipynb.
It works perfectly well but what I noticed is that when I save the ...
