Questions tagged [reinforcement-learning]
For questions related to learning controlled by external positive reinforcement or negative feedback signal or both, where learning and use of what has been thus far learned occur concurrently.
1
vote
0answers
6 views
Reward discounting in reinforcement learning for a Pong game
I am trying to understand how to train a neural network to win a Pong game using reinforcement learning, by following the blog post
Spinning up a Pong AI with deep reinforcement learning.
The ...
0
votes
0answers
20 views
improving tic tac toe model
For this tic tac toe model , how do I achieve the following model improvement for the original kaggle kernel ?
-> Try using a deeper neural network, and trying different learning rates, to see if it ...
2
votes
1answer
33 views
Reinforcement Learning n-step return
when reading the newest v2 of Sutton + Barto Reinforcement Learning, in Ch7 section 1 about N-step bootsrapping, they write about something they call the "n-step return error reduction property":
But ...
0
votes
0answers
10 views
What does the output of openai baselines mean?
I get the following output when training a PPO model on my environment:
...
0
votes
1answer
33 views
DQN exploration strategy for large grid-world environment
My task involves a large grid-world type of environment (grid size may be $30\times30$, $50\times50$, $100\times100$, at the largest $200\times200$). Each element in this grid either contains a 0 or a ...
0
votes
0answers
23 views
Reinforcement learning: How to combine action and magnitude within a single model?
Say for example you're training an AI-controlled bot (using a Markov Decision Process and a DQN) to clear a basic obstacle course; some obstacles you have to run over, some jump, sit, some squat etc, ...
0
votes
0answers
3 views
What is the difference between set of transition samples D and pattern set P in algorithm implementation of NFQ?
Neural Fitted Q iteration Algorithm :
What is the difference between the set of transition samples D and pattern set P in algorithm implementation of NFQ in Paper "Neural Fitted Q Iteration - First ...
1
vote
1answer
26 views
How do we actually sample action from policy in policy gradient methods?
This might be a stupid question, but bear with me i'm only a beginner at this. Recently i started to look at policy gradient methods and policies are represented as functions with features for larger ...
0
votes
1answer
22 views
Neural Network Optimizers in Reinforcement Learning non-well behaved environments
https://stackoverflow.com/questions/36162180/gradient-descent-vs-adagrad-vs-momentum-in-tensorflow
Here, the nice gifs explain how different algorithms approach towards the root. Unfortunately, the ...
1
vote
2answers
59 views
For forecasting and trading control, given limited data, what AI approaches are well matched?
I'm working on stock price prediction and automatic or semi-automatic control of trading. The price trends of these stocks exhibit recurring patterns that may be exploited. My dataset is currently ...
1
vote
1answer
34 views
A Question on TRPO proofs
In the TRPO paper, in Lemma 2 of Appendix A, I did not quite understand deriving inequality (31) from (30), which is:
$$\bar{A}(s) = P(a \neq \tilde{a} | s) \mathbb{E}_{(a, \tilde{a}) \sim (\pi, \...
2
votes
2answers
44 views
How to deal with episode termination in Advantage Actor-Critic algorithm?
Advantage Actor-Critic algorithm may use the following expression to get 1-step estimate of the advantage:
$ A(s_t,a_t) = r(s_t, a_t) + \gamma V(s_{t+1}) (1 - done_{t+1}) - V(s_t) $
where $done_{t+...
-1
votes
1answer
43 views
DQN it's not working properly
I'm trying to build a DQN to replicate the DeepMind results. I'm doing with a simple DQN for the moment, but it isn't learning properly: after +5000 episodes, it couldn't get more than 9-10 points. ...
3
votes
0answers
50 views
Scrabble game using machine learning
I've been thinking if machine learning can be used to play the game Scrabble. My knowledge is limited in the ML field, thus I've seeking some pointers :)
I want to know how could I possibly build a ...
2
votes
1answer
78 views
How to build AI bots for board games like monopoly?
I am trying to build a Q learning-based bot for board games, specifically monopoly. I am fairly new to Q-learning and currently, I have only implemented some bots that can play simple games like Tic-...
1
vote
1answer
46 views
Reinforcement Learning Rewards with Internal State
I'm new to ML / Reinforcement Learning.
I'm looking at source from
https://github.com/openai/gym/blob/master/gym/envs/toy_text/blackjack.py
where reward is calculated with
...
1
vote
0answers
26 views
Why are all the actions converging to the same index?
I am using PPO with an LSTM agent. My agent is performing 10 actions for each episode, one action is corresponding to one LSTM timestep and the action space is discrete. I have only one reward per ...
0
votes
0answers
42 views
Q-Learning fails to converge even after 50K iterations for a simple board game - What could be the reason for this?
To get a feel of the model-free reinforcement learning, I tried to implement Q-learning for a simple 10 X 10 game board having 4 possible actions where the agent could move either North, East, South, ...
1
vote
0answers
23 views
Why does Q-learning converges to optimal policy even if I am acting suboptimally?
In Q-learning, during training, it doesn’t matter how I select actions. The algorithm always converges to optimal optimal policy. Why does this happen?
0
votes
1answer
38 views
Is it possible to use a feed-forward neural network to predict the actions in reinforcement learning?
I have done a lot of research on the internet about Reinforcement Learning and I found encountered methods of Reinforcement Learning: Q-Learning and Deep Q-Learning. And I have developed a vague idea ...
1
vote
0answers
20 views
What Actually is Off-Policy Q-Learning?
Recently I have come across an information stating:
Q-learning converges to optimal policy -- even if you’re acting suboptimally!
It also states:
When an optimal policy is still learned from ...
2
votes
1answer
51 views
What is the time complexity of the value iteration algorithm?
Recently, I have come across the information that the time complexity for each iteration of the value iteration algorithm is $\mathcal{O}(|S|^{2}|A|)$, where $|S|$ is the number of states and $|A|$ ...
1
vote
1answer
42 views
Deep Q-Learning poor convergence on Stochastic Environment
I'm trying to implement a Deep Q-network in Keras/TF that learns to play Minesweeper (our stochastic environment). I have noticed that the agent learns to play the game pretty well with both small and ...
3
votes
2answers
49 views
What is the best way to integrate unchangeable ethics into a chatbot
I am building a generative model chatbot as a research and learning project. One of the most important parts of my project is to research ways in which I can make this chatbot work in a consistently ...
1
vote
1answer
29 views
Importance Sampling for Monte Carlo Reinforcement Learning
Am reading the book titled "Reinforcement Learning - An Introduction by Sutton and Barto" https://drive.google.com/file/d/1opPSz5AZ_kVa1uWOdOiveNiBFiEOHjkG/view
and arriving at chapter 5 about Monte ...
2
votes
1answer
30 views
Reinforcement Learning to Grouped Scheduling Optimisation Problem
I am not sure the name of this kind of problem, but anyway, the situation is as below.
Assign teachers into Groups and consider on each of their workload, availability etc.
There are some other soft/...
0
votes
0answers
13 views
How to decide on including a variable in the agent's state?
I have built a simple model for energy consumption in a building. The inputs to the model are the outside temperature, relative humidity, solar irradiation value and a temperature control value that ...
1
vote
0answers
46 views
How does Hindsight Experience Replay learn from unsuccessful trajectories
I am confused by how HER learns from unsuccessful trajectories. I understand that from failed trajectories it creates 'fake' goals that it can learn from.
Ignoring HER for now, if in the case where ...
0
votes
1answer
38 views
Are artificial intelligence learnings or trainings transferable from one agent to the other?
One disadvantage or weakness of Artificial Intelligence today the slow nature of learning or training success. For instance, an AI agent might require a 100,000 samples or more to reach an appreciable ...
4
votes
1answer
43 views
Building AI from chess - data shape from simulation
Problem
My problem is the following: Given 1000 wins, losses, and ties from a chess simulation I am using, what shape should each game be (I.e., sequence of moves leading to win/loss/tie) in order to ...
1
vote
0answers
20 views
Imitation Learning
I am trying to understand the proof of theorem 2.1 form this paper:
Ross, Stéphane, and Drew Bagnell. "Efficient reductions for imitation learning." Proceedings of the thirteenth international ...
3
votes
1answer
39 views
Deep Q-Learning: why don't we use mini-batches during experience reply?
In examples and tutorial about DQN, I've often noticed that during the experience replay (training) phase people tend to use stochastic gradient descent / online learning. (e.g. link1, link2)
...
1
vote
0answers
132 views
Solving equations using reinforcement learning
I was lately curious about a reinforcement learning approach that would solve maths equations.
For example, if I have the following equation:
$$ f(g(h(w))) = 0 , with \ w = \begin{matrix}
a_{11} &...
2
votes
1answer
32 views
How to overcome overfitting to single player styles in reinforcement learning?
I am implementing an actor-critic reinforcement learning algorithm for winning a two player tic-tac-toe like game. The agent is trained against a min-max player and after a number of episodes is able ...
2
votes
1answer
25 views
What is the meaning of the actions, in `gym` from OpenAI?
In a gym environment the action space is often a discrete space where each action is labeled by an integer. I cannot find a way to figure out the correspondence ...
2
votes
2answers
41 views
3D environment for RL research in Academia
I'm doing my thesis on Reinforcement Learning. My focus on Partially Observable Environments like 3D Games. I want to choose a 3D platform for testing and doing ...
1
vote
1answer
112 views
Continuous Advantage Actor Critic Implementation
I'm having trouble implementing AC for continuous action space. As far as I can tell, my code doesn't seem to have any bugs! The agent is learning "something" as its behaviour seems to vary ...
0
votes
0answers
43 views
Differences between MDP, Semi MDP, and POMDP
I just wanted to confirm that my understanding of the different Markov Decision Processes are correct because they are the fundamentals of reinforcement learning. Also, I read a few literature sources,...
1
vote
2answers
42 views
How to define reward function in POMDP
How do I define a reward function for my POMDP model?
In literature it is common to use one simple number as a reward, but I am not sure if this is really how you define a function. Because this way ...
1
vote
0answers
29 views
How to design the reward for an action which is the only legal action at some state
I am working on a RL project,but got stuck at one point: The task is continuous (Non-episodic). Following some suggestion from Sutton's RL book, I am using a value function approximation method with ...
3
votes
2answers
68 views
Reinforcement learning objective as conditional expectations
In one of his lectures Levine describes the objective of reinforcement learning as: $$J(\tau) = E_{\tau\sim p_\theta(\tau)}[r(\tau)]$$
where $\tau$ refers to a single trajectory and $p_\theta(\tau)$ ...
0
votes
0answers
23 views
Bellman error in linear value function approximation is non-convex?
This is in reference to Sutton's Reinforcement Introduction book (second edition). Chapter "Off-policy methods with approximation" > "Linera value-function geometry".
Linear value function space is a ...
1
vote
1answer
48 views
Not sure where to start Reinforcement Learning for a game with many actions
I am trying to figure out how to use Reinforcement Learning as a black box for a game-ai.
These are some of the steps I have followed;
In the game, a player has to avoid flying birds. If he wants ...
2
votes
1answer
94 views
Ensure convergence of DDQN if true Q-values are very close
I am applying a Double DQN algorithm to a highly stochastic environment where some of the actions in the agent's action space have very similar "true" Q-values (i.e. the expected future reward from ...
0
votes
0answers
17 views
How to train a model by accounting for boundary constraints?
I've a robot traverse through a grid layout. Based on the wheel speed difference I classify actions as either straight, left or right. I computed the distances based on the time duration and the speed ...
1
vote
1answer
43 views
Reinforcement Learning (RL) how to obtain $p(s',r|s,a)$
I am trying to study the book Reinforcement Learning: An Introduction (Sutton & Barto, 2018). In chapter 3.1 the authors state the following exercise
Exercise 3.5 Give a table analogous to that ...
0
votes
0answers
11 views
Creating a Parkour agent using Deep Reinforcement Learning
How do I go about creating a Parkour agent which uses Deep RL. I have considered one approach wherein I can learn complex maneuvers using Imitation Learning (something like DeepMimic or GAIL paper). ...
1
vote
1answer
37 views
Some RL algorithms (especially policy gradients) initialize with random policies, which often manifests as random jitter on spot for a long time?
I am reviewing a statement on the website for ES regarding structured exploration.
https://blog.openai.com/evolution-strategies/
Structured exploration. Some RL algorithms (especially policy
...
1
vote
1answer
63 views
Reward-related formulation in reinforcement learning
I am referring to eq. 3.6 (p/g 49) based on Sutton's online book and can be found in an image below. I could not make sense of the final derivation of the equation $r(s, a, s')$. My question is ...
1
vote
1answer
41 views
When is Markov Decision Process (MDP) not adequate for goal-directed learning tasks
In the book Reinforcement Learning: An Introduction (Sutton & Barto, 2018). The authors ask
Exercise 3.2: Is the MDP framework adequate to usefully represent all goal-directed learning tasks? ...
