Questions tagged [reinforcement-learning]
For questions related to learning controlled by external positive reinforcement or negative feedback signal or both, where learning and use of what has been thus far learned occur concurrently.
817
questions
1
vote
1answer
45 views
Why isn't my implementation of DQN using TensorFlow on the FrozenWorld environment working?
I am trying to test DQN on FrozenWorld environment in gym using TensorFlow 2.x. The update rule is (off policy)
$$Q(s,a) \leftarrow Q(s,a)+\alpha (r+\gamma~ max_{a'}Q(s',a')-Q(s,a))$$
I am using an ...
2
votes
0answers
24 views
Can we use a neural network that is trained using Reinforcement Learning for dynamic game level difficulty designing in realtime?
I am a newbie to Machine Learning and AI. As per my understanding, with the use of reinforcement learning
(reward/punishment environment), we can train a neural network to play a game. I would like ...
1
vote
0answers
16 views
Does apprenticeship learning require prospective data?
I am thinking of applying apprenticeship learning on retrospective data. From looking at this paper by Ng https://ai.stanford.edu/~ang/papers/icml04-apprentice.pdf which talks about apprenticeship ...
2
votes
1answer
24 views
Reinforcement-learning: grey-scaling vs color of CNN input. Tradeoff?
I'm doing reinforcement learning and have a visual observation as state input for my agent. In the Deepmind Atari paper they greyscale the input image before they input it into the CNN to reduce the ...
1
vote
1answer
32 views
How to estimate the error during training in deep reinforcement learning
How do I calculate the error during the training phase for deep reinforcement learning models?
Deep reinforcement learning is not supervised learning as far as I know. So how can the model know ...
2
votes
0answers
11 views
How does policy evaluation work for continuous state space model-free approaches?
How does policy evaluation work for continuous state space model-free approaches?
Theoretical model-based approach for the discrete state and action space can be computed via dynamic programming and ...
3
votes
1answer
29 views
Is the minimax algorithm model-based?
Trying to get my head around model-free and model-based algorithms in RL. In my research, I've seen the search trees created via the minimax algorithm. I presume these trees can only be created with a ...
0
votes
0answers
33 views
Understanding V- and Q-functions
Assume the existence of a Markov Decision Process consisting of:
State space $S$
Action space $A$
Transition model $T: S \times A \times S \to [0,1]$
Reward function $R: S \times A \times S \to \...
3
votes
1answer
73 views
Is there a family tree for reinforcement learning algorithms?
Can anyone point me in the direction of a nice graph that depicts the "family tree", or hierarchy, of RL algorithms (or models)? For example, it splits the learning into TD and Monte Carlo methods, ...
3
votes
1answer
43 views
Why not more TD(𝜆) in actor-critic algorithms?
Is there either an empirical or theoretical reason that actor-critic algorithms with eligibility traces have not been more fully explored? I was hoping to find a paper or implementation or both for ...
2
votes
2answers
32 views
Is there a good ratio between the positive and negative rewards in reinforcement learning?
Is there an ideal ratio in reinforcement learning between the positive and negative rewards?
Suppose I have the scenario of moving a robot across the river. There are two options, walk across the ...
0
votes
1answer
31 views
What is the purpose of the arrow $\leftarrow$ in this formula?
What is the purpose of the arrow $\leftarrow$ in the formula below?
$$V(S_t) \leftarrow V(S_t) + \alpha \left[ G_t - V(S_t) \right]$$
I presume it's not the same as 'equals'.
1
vote
0answers
28 views
Implementing Actor-Critic with Experience Replay for Continuous Action Spaces
I have been trying to implement the ACER algorithm for continuous action spaces in reinforcement learning. The paper for the algorithm can be found here:
Sample Efficient Actor-Critic with Experience ...
1
vote
0answers
13 views
Can I apply experience on naive actor critic directly? Should it work?
Can I apply experience replay on naive actor-critic directly? Should it work?
I have tried that but unfortunately it didn't work.
2
votes
0answers
22 views
Evaluation a policy learned using Q - learning
I have been reading literature on reinforcement learning in healthcare. I am slightly confused between the policy evaluation for both SARSA and Q-learning.
To my knowledge, I believe that SARSA is ...
2
votes
1answer
48 views
Taxi-v3 help. What is meant exactly by convergence of the algo, the highest reward and optimal action for every state?
I started learning about Q table from this blog post Introduction to reinforcement learning and OpenAI Gym, by Justin Francis, which has a line as below -
After so many episodes, the algorithm will ...
3
votes
1answer
94 views
Why is the state-action value function used more than the state value function?
In reinforcement learning, the state-action value function seems to be used more than the state value function. Why is it so?
0
votes
0answers
9 views
How to encode board before input into the neural net?
Currently I'm working on an educational project (implementation of AlphaZero approach to different types of board games).
My biggest concern at the moment is how to encode board before input into the ...
1
vote
0answers
30 views
What kind of enemy to train a good RL-agents
So I want to create an RL-agent for two players-board game. I want to use a simple DQN for the first player (my RL-agent). Then, what kind of algorithm that should I use on the second player (my RL-...
1
vote
0answers
26 views
Policy $\pi$ obtained from inverse reinforcement learning vs reward shaping
I am working on a research project about the different reward functions being used in RL domain. I have read up on Inverse Reinforcement Learning (IRL) and Reward Shaping (RS). I would like to clarify ...
0
votes
0answers
11 views
Which hidden state should I use for a trajectory when incorporating LSTM into RL?
I'm trying to wrap my head around using LSTM in an RL algorithm like actor-critic or PPO. I've found this Github code which presents this in a very simple manner, however I have a very limited ...
0
votes
0answers
16 views
Curiosity Driven Learning affect optimal policy
I am trying to understand some of the different approaches used to overcome sparse rewards in a reinforcement learning setting for a research project. Particularly, I have looked at curiosity driven ...
1
vote
1answer
37 views
Immediate reward received in Atari game using DQN
I am trying to understand the different reward functions modelled in a reinforcement learning problem. I want to be able to know how the temporal credit assignment problem, (where the reward is ...
1
vote
0answers
38 views
Is it possible to use just one policy in a self-play setting? [closed]
I would like to ask is it possible to train an agent under self-playing setting but with just one policy to be trained? What are the foreseeable problems with such an implementation? My concern is as ...
1
vote
0answers
15 views
How to represent a state in a card game environment? (Wizard)
We are attempting to build an AI that manages to play the cardgame Wizard. So far er have a working network (based on the YOLO object-detection) that is abled to detect which cards are played. When ...
1
vote
0answers
16 views
In reinforcement learning what do we mean by a model? [duplicate]
I was going through the chapter "Monte Carlo Methods" from the book "Reinforcement Learning" by Sutton and Barto. The author says that when a model is not available then it is useful to estimate ...
3
votes
0answers
37 views
Are there reinforcement learning algorithms not based on Markov decision processes?
Are all RL algorithms based on the MDP? If not, could you give examples of some which aren't? I've looked elsewhere, but I haven't seen it explicitly said.
3
votes
0answers
30 views
How exactly does self-play work, and how does it relate to MCTS?
I am working towards using RL to create an AI for a two-player, hidden-information, a turn-based board game. I have just finished David Silver's RL course and Denny Britz's coding exercises, and so am ...
4
votes
1answer
178 views
What are the state-of-the-art meta-reinforcement learning methods?
This question can seem a little bit too broad, but I am wondering what are the current state-of-the-art works on meta reinforcement learning. Can you provide me with the current state-of-the-art in ...
2
votes
1answer
22 views
In the reinforcement learning is the value of terminal/goal state always zero?
Let's assume we are in the grid world with states in a 3X3 grid numbered as 0,1,...8. If 8 is the goal state and the reward of landing in the goal state is 10 and the reward of just wandering around ...
2
votes
0answers
44 views
Formulation of a Markov Decision Process Problem
Given a list of $N$ questions. If question $i$ is answered correctly (given probability $p_i$), we receive reward $R_i$; if not the quiz terminates. Find the optimal order of questions to maximize ...
3
votes
1answer
62 views
How does Monte Carlo have high variance?
I was going through David Silver's lecture on reinforcement learning (lecture 4). At 51:22 he says that Monte Carlo (MC) methods have high variance and zero bias. I understand the zero bias part. It ...
2
votes
1answer
57 views
What does the figure “Blackjack Value Function…” from Sutton represent?
I came across this graph in David Silver's youtube lecture and Sutton's book on reinforcement learning.
Can anyone help me understand the graph?
From the graph, for 10000 episodes what i see is ...
2
votes
1answer
37 views
How can I train a RL agent to play board games successfully without human play?
How would you go about training an RL Tic Tac Toe (well, any board game, really) application to learn how to play successfully (win the game), without a human having to play against the RL?
...
2
votes
1answer
28 views
How can I perform lane detection with reinforcement learning?
I'm quite new to reinforcement learning and my project will consist of detecting lanes with RL.
I'm using q-learning and I'm having a hard time thinking how my q table should look like, I mean - ...
0
votes
0answers
21 views
Concrete examples of models and policies in Tic Tac Toe environment
I'm having difficulty picturing how models and policies are represented. Could someone describe how they would look in the context/environment of a game of Tic Tac Toe? For example, "In Tic Tac Toe, ...
2
votes
1answer
144 views
Are model-free and off-policy algorithms the same?
In respect of RL, is model-free and off-policy the same thing, just different terminology? If not, what are the differences? I've read that the policy can be thought of as 'the brain', or decision ...
2
votes
0answers
16 views
What effect does increasing the actions in RL have?
Consider a 2D snake game, where the snake has to eat food to become longer. It must avoid hitting walls and biting into her tail.
Such a game could have a different amount of actions:
3 actions: go ...
1
vote
0answers
26 views
Reinforcement Learning Continuous Control (DDPG): How to avoid thrashing of issued actions? How to reward smooth output over flittering?
Currently I'm working on a continuous state / continuous action controller. It shall control a certain roll angle of an aircraft by issuing the correct aileron commands (between -1...1 continuous).
...
0
votes
0answers
11 views
How to pass observation from CartPole-v0 to neural network using tensorflow
I am trying to learn about RL by implementing DQN with tensorflow. However, I am really stuck with tensorflow.. I just don't understand it.
I think I have found the core of what I understand - I dont ...
3
votes
0answers
62 views
Why does reinforcement learning using a non-linear function approximator diverge when using strongly correlated data as input?
While reading the DQN paper, I found that randomly selecting and learning samples reduced divergence in RL using a non-linear function approximator (e.g a neural network).
So why does Reinforcement ...
2
votes
2answers
172 views
Simulating successful trajectories in Montezuma's Revenge turns out to be unsuccessful
I have written code in OpenAI's gym to simulate a random playing in Montezuma's Revenge where the agent randomly samples actions from the action space and tries to play the game. A success for me is ...
4
votes
1answer
62 views
How powerful is OpenAI's Gym and Universe in board games area?
I'm a big fan of computer board games and would like to make Python chess/go/shogi/mancala programs. Having heard of reinforcement learning, I decided to look at OpenAI Gym.
But first of all, I would ...
1
vote
1answer
24 views
Is the temperature equal to epsilon in Reinforcement Learning?
This is a piece of code from my homework.
...
1
vote
0answers
25 views
Can I solve this assignment problem with RL or AI planning, and if yes how?
I have a list of positive nonzero integers $T=[v_1,\dots,v_𝑛|v_𝑖\in Z^{\neq}]$ which sum up to $V=\sum_i v_i$. Typically, the length of T (number of integers) goes from 100 to 1000. The list is not ...
3
votes
1answer
28 views
What is the difference between the concepts “known environment” and “deterministic environment”?
According to the book "Artificial Intelligence: A Modern Approach", "In a known environment, the outcomes (or outcome probabilities if the environment is stochastic) for all actions are given.", and ...
2
votes
0answers
48 views
How does the update rule for the one-step actor-critic method work?
Can you please elucidate the math behind the update rule for the critic? I've seen in other places that just a squared distance of $R + \hat{v}(S', w) - \hat{v}(S,w)$ is used, but Sutton suggests an ...
1
vote
1answer
27 views
Q-learning problem wrong policy
I am coding out a simple 4x4 grid game whereby the agent starts at a particular state and his aim is to reach the terminal state. The agent is supposed to avoid traps along the way and reach the end ...
2
votes
1answer
35 views
Do we need an explicit policy to sample $A'$ in order to compute the target in SARSA or Q-learning?
I would much appreciate if you could point me in the right direction regarding this question about targets for SARSA and Q-learning (notation: $S$ is the current state, $A$ is the current action, $R$ ...
1
vote
1answer
52 views
Understanding the loss function in deep Q-learning
I am trying to understand how deep Q learning (DQN) works. To my current understanding, each $Q(s, a)$ functions is estimated to be a function of a feature vector of its state $\phi$(s) and the weight ...
