# Questions tagged [reinforcement-learning]

For questions related to learning controlled by external positive reinforcement or negative feedback signal or both, where learning and use of what has been thus far learned occur concurrently.

824 questions
Filter by
Sorted by
Tagged with
252 views

901 views

### How do I apply reinforcement learning to a game with infinitely many actions?

I am trying to figure out how to use a reinforcement learning algorithm, if possible, as a "black box" to play a game. In this game, a player has to avoid flying birds. If he wants to move, he has to ...
526 views

### Ensure convergence of DDQN if true Q-values are very close

I am applying a Double DQN algorithm to a highly stochastic environment where some of the actions in the agent's action space have very similar "true" Q-values (i.e. the expected future reward from ...
210 views

### Reinforcement Learning (RL) how to obtain $p(s',r|s,a)$

I am trying to study the book Reinforcement Learning: An Introduction (Sutton & Barto, 2018). In chapter 3.1 the authors state the following exercise Exercise 3.5 Give a table analogous to that ...
137 views

### Some RL algorithms (especially policy gradients) initialize with random policies, which often manifests as random jitter on spot for a long time?

I am reviewing a statement on the website for ES regarding structured exploration. https://blog.openai.com/evolution-strategies/ Structured exploration. Some RL algorithms (especially policy ...
74 views

### Reward-related formulation in reinforcement learning

I am referring to eq. 3.6 (p/g 49) based on Sutton's online book and can be found in an image below. I could not make sense of the final derivation of the equation $r(s, a, s')$. My question is ...
519 views

### When is Markov Decision Process (MDP) not adequate for goal-directed learning tasks

In the book Reinforcement Learning: An Introduction (Sutton & Barto, 2018). The authors ask Exercise 3.2: Is the MDP framework adequate to usefully represent all goal-directed learning tasks? ...
110 views

### Understanding the notation in the definition of the expected reward

I am new to RL and I am trying to work through the book Reinforcement Learning: An Introduction I (Sutton & Barto, 2018). In chapter 3 on Finite Markov Decision Processes, the authors write the ...
133 views

### Q-Learning the generic maze solution

After doing some exercices on Q-learning for maze solving, I wondered : my q-learning algorithms solve only ONE maze. The AI doesn't learn how to solve mazes, so how can I achieve it ? For instance ...
2k views

### What does the agent in reinforcement learning exactly do?

What is an agent in reinforcement learning (RL)? I think it is not the neural network behind. What does the agent in RL exactly do?
77 views

### How do I know how changes in the weights are changing the reward in Reinforcement Learning

I already know the basics of the basic of Machine Learning. E.g.: Backpropagation, Convolution, etc. First of let me explain Reinforcement learning to make sure I grasped the concept correctly. In ...
177 views

### How can a fully automated vacuum cleaner use and update room information?

Assistive Subsystems Consider an automated vacuum cleaner with the following subsystems under the command of an AI system to be designed. These subsystems limit the AI complexity to just intelligent ...
1k views

### What models is Google's quick draw using?

Quick draw is a Google experiment using user generated online doodles and machine learning to play a game of "Guess what I'm drawing" similar to the board game Pictionary. I'm interested if anyone ...
109 views

### Using reinforcement learning to find a preconditioner for linear systems of the form Ax = b

Sparse linear system are normally solved by using solvers like MINRES, Conjugate gradient, GMRES. Efficient preconditioning, i.e., finding a matrix P such that PAx = Pb is easier to solve then the ...
1k views

### Should the reward or the Q value be clipped for reinforcement learning

When extending reinforcement learning to the continuous states, continuous action case, we must use function approximators (linear or non-linear) to approximate the Q-value. It is well known that non-...
103 views

### Dyna-Q algorithm, having trouble when adding the simulated experiences

I'm trying to create a simple Dyna-Q agent to solve small mazes, in python. For the Q function, Q(s,a), I'm just using a matrix, where each row is for a state value, and each column is for one of the ...
57 views

### Reinforcement learning for segmenting the robot path to reflect the true distances

I've a grid of rectangles acting as blocks. The robot traverses through the inter-spaces between these consecutive blocks. Now I have sensor data streaming in representing Right and left wheel speeds. ...
36 views

### In apprenticeship learning, is it possible to outperform the master?

As stated in the title, I'm wondering if it would be possible to "outperform" the master in the apprenticeship learning. I'm aware that the question might be not clear enough; but hopefully, someone ...
2k views

### Why do you not see dropout layers on reinforcement learning examples?

I've been looking at reinforcement learning, and specifically playing around with creating my own environments to use with the OpenAI Gym AI. I am using agents from the stable_baselines project to ...
78 views

### Dealing with Lags in Reinforcement Learning

Following up on my previous questions using a hypothetical AI system to manage air flow using dampers to achieve an optimal target of exactly equal airflow at a number of vents; (thank you for all ...
112 views

### Using a DQN with a variable amount of Valid Moves per turn for a Board Game

I have created a game on an 8x8 grid and there are 4 pieces which can move essentially like checkers pieces (Forward left or Forward right only). I have implemented a DQN in order to pull this off. ...
411 views

### How do I calculate the policy in the Proximal Policy Optimization algorithm?

I recently watched the video on Proximal Policy Optimization (PPO). Now, I want to upgrade my actor-critic algorithm written in PyTorch with PPO, but I'am not sure how the new parameters / thetas are ...
119 views

### Is it possible to state an outliers detection problem as a reinforcement learning problem?

To me it seems to be ill defined. Partially because of absence of knowledge which points are to be considered outliers in the first place. The problem which I have in mind is "bad market data" ...
478 views

### Difficulty in understanding identifiability in the “Dueling Network Architectures for Deep Reinforcement Learning” paper

I have difficulty understanding the following paragraph in the below excerpts from page 4 to page 5 from the paper Dueling Network Architectures for Deep Reinforcement Learning. The author said "we ...
38 views

### Dependance of Value Function of an MDP on Policy

From what I understand, the value function estimates how 'good' it is for an agent to be in a state and a policy is a mapping of actions to state. So if I have understood value function and policies ...
15 views

### Does inflation should occur in output layer when I do Artificial Neural Network to increase smartness of the model?

The idea that come to my mind is called Value Based Model for ANN. We use simple DCF formula to calculate kind of Q value: Rewards/Discount rate. Discount rate is a risk of getting the reward on the ...
457 views

### Does a solution for Wumpus World with neural networks exist?

The Wumpus World proposed in book of Stuart Russel and Peter Norvig, is a game which happens on a 4x4 board and the objective is to grab the gold and avoiding the threats that can kill you. The rules ...
936 views

### Why does the policy network in AlphaZero work?

In AlphaZero, the policy network (or head of the network) maps game states to a distribution of the likelihood of taking each action. This distribution covers all possible actions from that state. ...
191 views

### How can a reinforcement learning agent generalize if it is trained against only one opponent?

I started teaching myself about reinforcement learning a week ago and I have this confusion about the learning experience. Let's say we have the game Go. And we have an agent that we want to be able ...
380 views

### What is the physics engine used by DeepMimic?

I found a video for the paper DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills on YouTube. I looked in the related paper, but could not find details of how to ...
424 views

### AlphaZero Value Network

The Alpha Zero (as well as AlphaGo Zero) papers say they trained the value head of the network by "minimizing the error between the predicted winner and the game winner" throughout its many self play ...
318 views

### Snake game: snake converges to going in the same direction every time

This is a q-learning snake using a neural network as a q function aproximator and I'm losing my mind here the current model it's worst than the initial one. The current model uses a 32x32x32 ...
79 views

### Should Q values be changing within an epoch/episode or should they change after one episode/epoch?

I am trying to use Deep-Q learning environment to learn Super Mario Bros. The implementation is on Github. I have a neural network that Q values update within an episode for a very small learning ...
928 views

### Learning Rate Decay and Exploration Rate Decay

Should I be decaying the learning rate and the exploration rate in the same manner? What's too slow and too fast of an exploration and learning rate decay? Or is it specific from model to model?
604 views

### Why is baseline conditional on state at some timestep unbiased?

In robotics, the reinforcement learning technique is used for finding the control pattern for a robot. Unfortunately, most policy gradient method are statistically biased which could bring the robot ...
187 views

### Is Experience Replay like dreaming?

Drawing parallels between Machine Learning techniques and a human brain is a dangerous operation. When it is done successfully, it can be a powerful tool for vulgarisation, but when it is done with no ...
501 views

### Why is the derivative 0 if the policy is deterministic?

In the Berkeley RL class they mention the gradient would be 0 if the policy is deterministic. Why is that? https://www.youtube.com/watch?v=XGmd3wcyDg8&feature=youtu.be&t=1071
254 views

### How many episodes does it take for a vanilla one-step actor-critic agent to master the OpenAI BipedalWalker-v2 problem?

I'm trying to solve the OpenAI BipedalWalker-v2 by using a one-step actor-critic agent. I'm implementing the solution using python and tensorflow. I'm following this pseudo-code taken from the book ...
193 views

### Why does Clipped Surrogate Objective works in Proximal Policy Optimization

In ''Proximal Policy Optimization Algorithms'' , Schulman et al. (2017), page 3 I don't understand why the clipped surrogate objective works. As written in the article : "With this scheme, we only ...
42 views

### Can Q-learning be used to find the shortest distance from each source to destination?

Is it possible to form a table that will have simply the shortest distance from each source to destination using q learning? If not, suggest any other learning algorithm.