Questions tagged [reinforcement-learning]

For questions related to learning controlled by external positive reinforcement or negative feedback signal or both, where learning and use of what has been thus far learned occur concurrently.

Filter by
Sorted by
Tagged with
0
votes
1answer
237 views

Is it possible to use a feed-forward neural network to predict the actions in reinforcement learning?

I have done a lot of research on the internet about Reinforcement Learning and I found encountered methods of Reinforcement Learning: Q-Learning and Deep Q-Learning. And I have developed a vague idea ...
1
vote
2answers
1k views

Q Learning Algorithm not converging

I am trying to run Deep Q-learning algorithm on a game which i made in python using pygame library. The algorithm accepts the game screen (4 frames) as input to neural network which used as the ...
2
votes
1answer
522 views

Deep Q-Learning poor convergence on Stochastic Environment

I'm trying to implement a Deep Q-network in Keras/TF that learns to play Minesweeper (our stochastic environment). I have noticed that the agent learns to play the game pretty well with both small and ...
9
votes
1answer
4k views

What is the difference between an observation and a state in reinforcement learning?

I'm studying reinforcement learning. It seems that "state" and "observation" mean exactly the same thing. They both capture the current state of the game. Is there a difference between the two terms?...
2
votes
1answer
50 views

Reinforcement Learning to Grouped Scheduling Optimisation Problem

I am not sure the name of this kind of problem, but anyway, the situation is as below. Assign teachers into Groups and consider on each of their workload, availability etc. There are some other soft/...
1
vote
1answer
44 views

Are artificial intelligence learnings or trainings transferable from one agent to the other?

One disadvantage or weakness of Artificial Intelligence today the slow nature of learning or training success. For instance, an AI agent might require a 100,000 samples or more to reach an appreciable ...
2
votes
1answer
1k views

Understanding why the expectation is over the new policy $\pi'$ in the proof of the Policy Improvement Theorem

In reinforcement learning, policy improvement is a part of an algorithm called policy iteration, which attempts to find approximate solutions to the Bellman optimality equations. Pages 84 and 85 in ...
5
votes
6answers
382 views

Is reinforcement learning needed to create Strong AI?

By reinforcement learning, I don't mean the class of machine learning algorithms such as DeepQ, etc. I have in mind the general concept of learning based on rewards and punishment. Is it possible to ...
1
vote
1answer
457 views

What is the relation between back-propagation and reinforcement learning?

What is the relation between back-propagation and reinforcement learning?
2
votes
0answers
272 views

Solving equations using reinforcement learning

I was lately curious about a reinforcement learning approach that would solve maths equations. For example, if I have the following equation: $$ f(g(h(w))) = 0 , with \ w = \begin{matrix} a_{11} &...
1
vote
0answers
381 views

Continuous Advantage Actor Critic Implementation

I'm having trouble implementing AC for continuous action space. As far as I can tell, my code doesn't seem to have any bugs! The agent is learning "something" as its behaviour seems to vary ...
3
votes
1answer
538 views

Deep Q-Learning: why don't we use mini-batches during experience reply?

In examples and tutorial about DQN, I've often noticed that during the experience replay (training) phase people tend to use stochastic gradient descent / online learning. (e.g. link1, link2) ...
3
votes
1answer
122 views

How to overcome overfitting to single player styles in reinforcement learning?

I am implementing an actor-critic reinforcement learning algorithm for winning a two player tic-tac-toe like game. The agent is trained against a min-max player and after a number of episodes is able ...
2
votes
2answers
156 views

3D environment for RL research in Academia

I'm doing my thesis on Reinforcement Learning. My focus on Partially Observable Environments like 3D Games. I want to choose a 3D platform for testing and doing ...
1
vote
0answers
43 views

How to design the reward for an action which is the only legal action at some state

I am working on a RL project,but got stuck at one point: The task is continuous (Non-episodic). Following some suggestion from Sutton's RL book, I am using a value function approximation method with ...
4
votes
1answer
786 views

Ensure convergence of DDQN if true Q-values are very close

I am applying a Double DQN algorithm to a highly stochastic environment where some of the actions in the agent's action space have very similar "true" Q-values (i.e. the expected future reward from ...
3
votes
2answers
1k views

What layers to use in a Neural Network for card game

I am currently writing an engine to play a card game and I would like for an ANN to learn how to play the game. The game is currently playable, and I believe for this game a deep-recurrent-Q-network ...
1
vote
1answer
192 views

Some RL algorithms (especially policy gradients) initialize with random policies, which often manifests as random jitter on spot for a long time?

I am reviewing a statement on the website for ES regarding structured exploration. https://blog.openai.com/evolution-strategies/ Structured exploration. Some RL algorithms (especially policy ...
2
votes
1answer
81 views

Reward-related formulation in reinforcement learning

I am referring to eq. 3.6 (p/g 49) based on Sutton's online book and can be found in an image below. I could not make sense of the final derivation of the equation $r(s, a, s')$. My question is ...
6
votes
1answer
161 views

Q-Learning the generic maze solution

After doing some exercices on Q-learning for maze solving, I wondered : my q-learning algorithms solve only ONE maze. The AI doesn't learn how to solve mazes, so how can I achieve it ? For instance ...
4
votes
1answer
2k views

Should the reward or the Q value be clipped for reinforcement learning

When extending reinforcement learning to the continuous states, continuous action case, we must use function approximators (linear or non-linear) to approximate the Q-value. It is well known that non-...
0
votes
1answer
98 views

How do I know how changes in the weights are changing the reward in Reinforcement Learning

I already know the basics of the basic of Machine Learning. E.g.: Backpropagation, Convolution, etc. First of let me explain Reinforcement learning to make sure I grasped the concept correctly. In ...
6
votes
1answer
213 views

A solution for a famous problem in RL

I'm here to ask you for a solution on this problem which is: how to use Reinforcement Learning in Immersive Virtual Reality to make a person move to a specific location in a virtual environment. As ...
2
votes
1answer
2k views

What models is Google's quick draw using?

Quick draw is a Google experiment using user generated online doodles and machine learning to play a game of "Guess what I'm drawing" similar to the board game Pictionary. I'm interested if anyone ...
4
votes
1answer
342 views

How to deal with back-propagation when dealing with invalid moves in Reinforced Learning?

As discussed in this thread, you can handle invalid moves in Reinforced Learning by re-setting the probabilities of all illegal moves to zero and renormalising the output vector. In back-propagation, ...
3
votes
2answers
207 views

Should the actor or actor-target model be used to make predictions after training is complete (DDPG)?

The situation I am referring to the paper T. P. Lillicrap et al, "Continuous control with deep reinforcement learning" where they discuss deep learning in the context of continuous action spaces ("...
13
votes
1answer
2k views

Why do you not see dropout layers on reinforcement learning examples?

I've been looking at reinforcement learning, and specifically playing around with creating my own environments to use with the OpenAI Gym AI. I am using agents from the stable_baselines project to ...
1
vote
1answer
38 views

In apprenticeship learning, is it possible to outperform the master?

As stated in the title, I'm wondering if it would be possible to "outperform" the master in the apprenticeship learning. I'm aware that the question might be not clear enough; but hopefully, someone ...
1
vote
1answer
150 views

Using a DQN with a variable amount of Valid Moves per turn for a Board Game

I have created a game on an 8x8 grid and there are 4 pieces which can move essentially like checkers pieces (Forward left or Forward right only). I have implemented a DQN in order to pull this off. ...
1
vote
1answer
134 views

Is it possible to state an outliers detection problem as a reinforcement learning problem?

To me it seems to be ill defined. Partially because of absence of knowledge which points are to be considered outliers in the first place. The problem which I have in mind is "bad market data" ...
1
vote
1answer
40 views

Dependance of Value Function of an MDP on Policy

From what I understand, the value function estimates how 'good' it is for an agent to be in a state and a policy is a mapping of actions to state. So if I have understood value function and policies ...
1
vote
0answers
15 views

Does inflation should occur in output layer when I do Artificial Neural Network to increase smartness of the model?

The idea that come to my mind is called Value Based Model for ANN. We use simple DCF formula to calculate kind of Q value: Rewards/Discount rate. Discount rate is a risk of getting the reward on the ...
1
vote
1answer
516 views

Does a solution for Wumpus World with neural networks exist?

The Wumpus World proposed in book of Stuart Russel and Peter Norvig, is a game which happens on a 4x4 board and the objective is to grab the gold and avoiding the threats that can kill you. The rules ...
4
votes
1answer
420 views

Reinforcement Learning in asteroid game

Introduction An attractive asteroid game was described in the paper from 2007: quote: “In our first experiment, the virtual agent is a spaceship pilot, The pilot’s task is to maneuver the ...
0
votes
1answer
79 views

Should Q values be changing within an epoch/episode or should they change after one episode/epoch?

I am trying to use Deep-Q learning environment to learn Super Mario Bros. The implementation is on Github. I have a neural network that Q values update within an episode for a very small learning ...
5
votes
2answers
230 views

How can a reinforcement learning agent generalize if it is trained against only one opponent?

I started teaching myself about reinforcement learning a week ago and I have this confusion about the learning experience. Let's say we have the game Go. And we have an agent that we want to be able ...
1
vote
1answer
427 views

What is the physics engine used by DeepMimic?

I found a video for the paper DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills on YouTube. I looked in the related paper, but could not find details of how to ...
3
votes
1answer
235 views

What type of reinforcement learning can I do restricted to ~200MB on an average smartphone?

This concerns a set of finite, non-trivial, combinatorial games [M] in the form of an app. A sample game can be found here. Because this is a mass market product, we can't take up too much space, ...
6
votes
1answer
1k views

Learning Rate Decay and Exploration Rate Decay

Should I be decaying the learning rate and the exploration rate in the same manner? What's too slow and too fast of an exploration and learning rate decay? Or is it specific from model to model?
2
votes
0answers
310 views

How many episodes does it take for a vanilla one-step actor-critic agent to master the OpenAI BipedalWalker-v2 problem?

I'm trying to solve the OpenAI BipedalWalker-v2 by using a one-step actor-critic agent. I'm implementing the solution using python and tensorflow. I'm following this pseudo-code taken from the book ...
7
votes
1answer
203 views

Is Experience Replay like dreaming?

Drawing parallels between Machine Learning techniques and a human brain is a dangerous operation. When it is done successfully, it can be a powerful tool for vulgarisation, but when it is done with no ...
4
votes
2answers
513 views

Why is the derivative 0 if the policy is deterministic?

In the Berkeley RL class they mention the gradient would be 0 if the policy is deterministic. Why is that? https://www.youtube.com/watch?v=XGmd3wcyDg8&feature=youtu.be&t=1071
3
votes
1answer
82 views

Implementing AI/ML in customer service

I am working on a task where I am required to automate the customer service request channel. The process is quite typical. A customer queries about a product via email, the person on the front channel ...
4
votes
1answer
51 views

In imitation learning, do you simply inject optimal (state, action, reward, s(t+1)) experiences into your experience replay buffer?

due to my RL having difficulties learning some control actions, I've decided to use Imitation learning / apprenticeship learning to guide my RL to perform the optimal actions. I've read a few ...
3
votes
1answer
925 views

Tuning of PPO metaparameters: a high level overview of what each parameter does

I am using the PPO algorithm implemented by tensorforce: https://github.com/reinforceio/tensorforce . It works great and I am very happy with the results. However, I notice that there are many ...
3
votes
1answer
40 views

What is the pros and cons of increasing and decreasing the number of worker process in A3C?

In A3C, there are several child processes and one master process. The child precesses calculate the loss and backpropagation, and the master process sums them up and updates the parameters, if I ...
3
votes
1answer
81 views

In reinforcement learning, Is the optimal value (V*) corresponding to performing the best action in a given state?

it seems I am a little confused about the optimal value (V*) and optimal action-value (Q*) in reinforcement learning and just want some clarity because some blogs I read on Medium and GitHub are ...
6
votes
2answers
577 views

How should I handle action selection in the terminal state when implementing SARSA?

I recently started learning about reinforcement learning and currently I am trying to implement the SARSA algorithm, however I do not know how to deal with $Q(s', a')$, when $s'$ is the terminal state....
4
votes
2answers
4k views

How to combine backpropagation in neural nets and reinforcement learning?

As I am trying to make an AI with reinforcement learning, I have found out and implemented a lot of things such as both these topics (NNs and RL) separately. But when trying to combine them, I have ...
4
votes
0answers
79 views

Is there any theoretical capabilities of apply deep successor representations with A3C algorithm?

Deep Successor Representations(DSR) has given better performance in tasks like navigation when it compares to normal model-free RL tasks. Basically, DSR is a hybrid of model-free RL and Model-Based RL....