Questions tagged [reinforcement-learning]

For questions related to learning controlled by external positive reinforcement or negative feedback signal or both, where learning and use of what has been thus far learned occur concurrently.

Filter by
Sorted by
Tagged with
1
vote
0answers
252 views

Solving equations using reinforcement learning

I was lately curious about a reinforcement learning approach that would solve maths equations. For example, if I have the following equation: $$ f(g(h(w))) = 0 , with \ w = \begin{matrix} a_{11} &...
3
votes
1answer
86 views

How to overcome overfitting to single player styles in reinforcement learning?

I am implementing an actor-critic reinforcement learning algorithm for winning a two player tic-tac-toe like game. The agent is trained against a min-max player and after a number of episodes is able ...
2
votes
1answer
379 views

What is the mapping between actions and numbers in OpenAI's gym?

In a gym environment, the action space is often a discrete space, where each action is labeled by an integer. I cannot find a way to figure out the correspondence ...
2
votes
2answers
129 views

3D environment for RL research in Academia

I'm doing my thesis on Reinforcement Learning. My focus on Partially Observable Environments like 3D Games. I want to choose a 3D platform for testing and doing ...
1
vote
0answers
350 views

Continuous Advantage Actor Critic Implementation

I'm having trouble implementing AC for continuous action space. As far as I can tell, my code doesn't seem to have any bugs! The agent is learning "something" as its behaviour seems to vary ...
1
vote
0answers
396 views

Is my understanding of the differences between MDP, Semi MDP and POMDP correct?

I just wanted to confirm that my understanding of the different Markov Decision Processes are correct, because they are the fundamentals of reinforcement learning. Also, I read a few literature ...
2
votes
2answers
93 views

How to define reward function in POMDPs?

How do I define a reward function for my POMDP model? In the literature, it is common to use one simple number as a reward, but I am not sure if this is really how you define a function. Because this ...
1
vote
0answers
41 views

How to design the reward for an action which is the only legal action at some state

I am working on a RL project,but got stuck at one point: The task is continuous (Non-episodic). Following some suggestion from Sutton's RL book, I am using a value function approximation method with ...
3
votes
2answers
116 views

Reinforcement learning objective as conditional expectations

In one of his lectures Levine describes the objective of reinforcement learning as: $$J(\tau) = E_{\tau\sim p_\theta(\tau)}[r(\tau)]$$ where $\tau$ refers to a single trajectory and $p_\theta(\tau)$ ...
3
votes
1answer
901 views

How do I apply reinforcement learning to a game with infinitely many actions?

I am trying to figure out how to use a reinforcement learning algorithm, if possible, as a "black box" to play a game. In this game, a player has to avoid flying birds. If he wants to move, he has to ...
4
votes
1answer
526 views

Ensure convergence of DDQN if true Q-values are very close

I am applying a Double DQN algorithm to a highly stochastic environment where some of the actions in the agent's action space have very similar "true" Q-values (i.e. the expected future reward from ...
1
vote
2answers
210 views

Reinforcement Learning (RL) how to obtain $p(s',r|s,a)$

I am trying to study the book Reinforcement Learning: An Introduction (Sutton & Barto, 2018). In chapter 3.1 the authors state the following exercise Exercise 3.5 Give a table analogous to that ...
1
vote
1answer
137 views

Some RL algorithms (especially policy gradients) initialize with random policies, which often manifests as random jitter on spot for a long time?

I am reviewing a statement on the website for ES regarding structured exploration. https://blog.openai.com/evolution-strategies/ Structured exploration. Some RL algorithms (especially policy ...
2
votes
1answer
74 views

Reward-related formulation in reinforcement learning

I am referring to eq. 3.6 (p/g 49) based on Sutton's online book and can be found in an image below. I could not make sense of the final derivation of the equation $r(s, a, s')$. My question is ...
2
votes
1answer
519 views

When is Markov Decision Process (MDP) not adequate for goal-directed learning tasks

In the book Reinforcement Learning: An Introduction (Sutton & Barto, 2018). The authors ask Exercise 3.2: Is the MDP framework adequate to usefully represent all goal-directed learning tasks? ...
2
votes
1answer
110 views

Understanding the notation in the definition of the expected reward

I am new to RL and I am trying to work through the book Reinforcement Learning: An Introduction I (Sutton & Barto, 2018). In chapter 3 on Finite Markov Decision Processes, the authors write the ...
6
votes
1answer
133 views

Q-Learning the generic maze solution

After doing some exercices on Q-learning for maze solving, I wondered : my q-learning algorithms solve only ONE maze. The AI doesn't learn how to solve mazes, so how can I achieve it ? For instance ...
6
votes
1answer
2k views

What does the agent in reinforcement learning exactly do?

What is an agent in reinforcement learning (RL)? I think it is not the neural network behind. What does the agent in RL exactly do?
0
votes
1answer
77 views

How do I know how changes in the weights are changing the reward in Reinforcement Learning

I already know the basics of the basic of Machine Learning. E.g.: Backpropagation, Convolution, etc. First of let me explain Reinforcement learning to make sure I grasped the concept correctly. In ...
1
vote
0answers
177 views

How can a fully automated vacuum cleaner use and update room information?

Assistive Subsystems Consider an automated vacuum cleaner with the following subsystems under the command of an AI system to be designed. These subsystems limit the AI complexity to just intelligent ...
2
votes
2answers
1k views

What models is Google's quick draw using?

Quick draw is a Google experiment using user generated online doodles and machine learning to play a game of "Guess what I'm drawing" similar to the board game Pictionary. I'm interested if anyone ...
1
vote
1answer
109 views

Using reinforcement learning to find a preconditioner for linear systems of the form Ax = b

Sparse linear system are normally solved by using solvers like MINRES, Conjugate gradient, GMRES. Efficient preconditioning, i.e., finding a matrix P such that PAx = Pb is easier to solve then the ...
4
votes
1answer
1k views

Should the reward or the Q value be clipped for reinforcement learning

When extending reinforcement learning to the continuous states, continuous action case, we must use function approximators (linear or non-linear) to approximate the Q-value. It is well known that non-...
2
votes
0answers
103 views

Dyna-Q algorithm, having trouble when adding the simulated experiences

I'm trying to create a simple Dyna-Q agent to solve small mazes, in python. For the Q function, Q(s,a), I'm just using a matrix, where each row is for a state value, and each column is for one of the ...
1
vote
0answers
57 views

Reinforcement learning for segmenting the robot path to reflect the true distances

I've a grid of rectangles acting as blocks. The robot traverses through the inter-spaces between these consecutive blocks. Now I have sensor data streaming in representing Right and left wheel speeds. ...
1
vote
1answer
36 views

In apprenticeship learning, is it possible to outperform the master?

As stated in the title, I'm wondering if it would be possible to "outperform" the master in the apprenticeship learning. I'm aware that the question might be not clear enough; but hopefully, someone ...
13
votes
1answer
2k views

Why do you not see dropout layers on reinforcement learning examples?

I've been looking at reinforcement learning, and specifically playing around with creating my own environments to use with the OpenAI Gym AI. I am using agents from the stable_baselines project to ...
2
votes
1answer
78 views

Dealing with Lags in Reinforcement Learning

Following up on my previous questions using a hypothetical AI system to manage air flow using dampers to achieve an optimal target of exactly equal airflow at a number of vents; (thank you for all ...
1
vote
1answer
112 views

Using a DQN with a variable amount of Valid Moves per turn for a Board Game

I have created a game on an 8x8 grid and there are 4 pieces which can move essentially like checkers pieces (Forward left or Forward right only). I have implemented a DQN in order to pull this off. ...
2
votes
1answer
411 views

How do I calculate the policy in the Proximal Policy Optimization algorithm?

I recently watched the video on Proximal Policy Optimization (PPO). Now, I want to upgrade my actor-critic algorithm written in PyTorch with PPO, but I'am not sure how the new parameters / thetas are ...
1
vote
1answer
119 views

Is it possible to state an outliers detection problem as a reinforcement learning problem?

To me it seems to be ill defined. Partially because of absence of knowledge which points are to be considered outliers in the first place. The problem which I have in mind is "bad market data" ...
3
votes
3answers
478 views

Difficulty in understanding identifiability in the “Dueling Network Architectures for Deep Reinforcement Learning” paper

I have difficulty understanding the following paragraph in the below excerpts from page 4 to page 5 from the paper Dueling Network Architectures for Deep Reinforcement Learning. The author said "we ...
1
vote
1answer
38 views

Dependance of Value Function of an MDP on Policy

From what I understand, the value function estimates how 'good' it is for an agent to be in a state and a policy is a mapping of actions to state. So if I have understood value function and policies ...
1
vote
0answers
15 views

Does inflation should occur in output layer when I do Artificial Neural Network to increase smartness of the model?

The idea that come to my mind is called Value Based Model for ANN. We use simple DCF formula to calculate kind of Q value: Rewards/Discount rate. Discount rate is a risk of getting the reward on the ...
1
vote
1answer
457 views

Does a solution for Wumpus World with neural networks exist?

The Wumpus World proposed in book of Stuart Russel and Peter Norvig, is a game which happens on a 4x4 board and the objective is to grab the gold and avoiding the threats that can kill you. The rules ...
8
votes
1answer
936 views

Why does the policy network in AlphaZero work?

In AlphaZero, the policy network (or head of the network) maps game states to a distribution of the likelihood of taking each action. This distribution covers all possible actions from that state. ...
5
votes
2answers
191 views

How can a reinforcement learning agent generalize if it is trained against only one opponent?

I started teaching myself about reinforcement learning a week ago and I have this confusion about the learning experience. Let's say we have the game Go. And we have an agent that we want to be able ...
0
votes
1answer
380 views

What is the physics engine used by DeepMimic?

I found a video for the paper DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills on YouTube. I looked in the related paper, but could not find details of how to ...
4
votes
2answers
424 views

AlphaZero Value Network

The Alpha Zero (as well as AlphaGo Zero) papers say they trained the value head of the network by "minimizing the error between the predicted winner and the game winner" throughout its many self play ...
3
votes
2answers
318 views

Snake game: snake converges to going in the same direction every time

This is a q-learning snake using a neural network as a q function aproximator and I'm losing my mind here the current model it's worst than the initial one. The current model uses a 32x32x32 ...
0
votes
1answer
79 views

Should Q values be changing within an epoch/episode or should they change after one episode/epoch?

I am trying to use Deep-Q learning environment to learn Super Mario Bros. The implementation is on Github. I have a neural network that Q values update within an episode for a very small learning ...
6
votes
1answer
928 views

Learning Rate Decay and Exploration Rate Decay

Should I be decaying the learning rate and the exploration rate in the same manner? What's too slow and too fast of an exploration and learning rate decay? Or is it specific from model to model?
9
votes
2answers
604 views

Why is baseline conditional on state at some timestep unbiased?

In robotics, the reinforcement learning technique is used for finding the control pattern for a robot. Unfortunately, most policy gradient method are statistically biased which could bring the robot ...
7
votes
1answer
187 views

Is Experience Replay like dreaming?

Drawing parallels between Machine Learning techniques and a human brain is a dangerous operation. When it is done successfully, it can be a powerful tool for vulgarisation, but when it is done with no ...
4
votes
2answers
501 views

Why is the derivative 0 if the policy is deterministic?

In the Berkeley RL class they mention the gradient would be 0 if the policy is deterministic. Why is that? https://www.youtube.com/watch?v=XGmd3wcyDg8&feature=youtu.be&t=1071
2
votes
0answers
254 views

How many episodes does it take for a vanilla one-step actor-critic agent to master the OpenAI BipedalWalker-v2 problem?

I'm trying to solve the OpenAI BipedalWalker-v2 by using a one-step actor-critic agent. I'm implementing the solution using python and tensorflow. I'm following this pseudo-code taken from the book ...
4
votes
1answer
193 views

Why does Clipped Surrogate Objective works in Proximal Policy Optimization

In ''Proximal Policy Optimization Algorithms'' , Schulman et al. (2017), page 3 I don't understand why the clipped surrogate objective works. As written in the article : "With this scheme, we only ...
1
vote
1answer
42 views

Can Q-learning be used to find the shortest distance from each source to destination?

Is it possible to form a table that will have simply the shortest distance from each source to destination using q learning? If not, suggest any other learning algorithm.
14
votes
2answers
6k views

How to define states in reinforcement learning?

I am studying reinforcement learning and the variants of it. I am starting to get an understanding of how the algorithms work and how they apply to an MDP. What I don't understand is the process of ...
12
votes
3answers
3k views

How to implement a constrained action space in reinforcement learning?

I'm coding a reinforcement learning model with a PPO agent thanks to the very good Tensorforce library, built on top of Tensorflow. The first version was very simple and I'm now diving into a more ...

1 11 12 13 14 15 17