Questions tagged [reinforcement-learning]
For questions related to reinforcement learning, i.e. a machine learning technique where we imagine an agent that interacts with an environment (composed of states) in time steps by taking actions and receiving rewards (or reinforcements), then, based on these interactions, the agent tries to find a policy (i.e. a behavioural strategy) that maximizes the cumulative reward (in the long run), so the goal of the agent is to maximize the reward.
1,390
questions
0
votes
0answers
2 views
Relation between a value function of an MDP and a value function of the corresponding latent MDP
In paper "DeepMDP: Learning Continuous Latent Space Models for Representation Learning", Gelada et al. state in the beginning of section 2.4
The degree to which a value function of $\bar{\...
0
votes
1answer
19 views
How does one know that a problem is “model-free” in reinforcement learning?
Consider this slide from a Stanford lecture on reinforcement learning. It states that a model is
the agent's representation of how the world changes in response to the agent's action.
I've been ...
1
vote
1answer
23 views
Is this ML task possible?
I'm not very experienced in the world of machine learning and I'm wondering if this task is possible. (I apologize in advance if I'm not making total sense, but I think I can get the point across.)
...
1
vote
0answers
11 views
Mapping given probabilities to empirical probabilities
Consider following problem statement:
You have given $n$ actions. You can perform any of them. Each action gives you success with some probability. The challenge is to perform given finite number of ...
0
votes
0answers
24 views
What are the strategies for computationally heavy environments or long-time waiting environments?
I have an environment that is computationally heavy (takes several seconds to get a reward and next state). This limits reinforcement capability, due to poor sampling of the problem. There is any ...
-1
votes
0answers
33 views
what is meant by this line “OOM when allocating tensor with shape[1,160,160,4] ” [closed]
I was training the double DQN agent to play atari breakout along with experience replay(deque) of size 5000. It started randomly but after reaching episode 64 kernel died automatically. Jupyter ...
3
votes
2answers
191 views
What is the meaning of “exploration” in reinforcement and supervised learning?
While exploration is an integral part of reinforcement learning (RL), it does not pertain to supervised learning (SL) since the latter is already provided with the data set from the start.
That said, ...
1
vote
0answers
35 views
How many types of variational auto-encoders are there?
I have been studying about auto-encoders and variational auto-encoders. I would like to know how many variants of VAEs are there today.
If there are many variants, can they be used for feature ...
2
votes
0answers
22 views
What are proxy reward functions?
The understanding I have is that they somehow adjust the objective to make it easier to meet, without changing the reward function.
... the observed proxy reward function is the approximate solution ...
1
vote
0answers
24 views
Pytorch Deep q network not learning and step not stepping towards target
I am trying to create a simple deep q network for rl with conv2d layers. I canāt figure out what I am doing wrong, and the only thing I can see that doesnāt seem right is when I get the model ...
1
vote
0answers
21 views
Is there a UCB type algorithm for linear stochastic bandit with lasso regression?
Why is there no upper confidence bound algorithm for linear stochastic bandits that uses lasso regression in the case that the regression parameters are sparse in the features?
In particular, I don't ...
0
votes
1answer
40 views
What is the search depth of AlphaGo and AlphaGo Zero?
I cannot find reliable sources but someone says it is 40 moves and someone else says it is 50+ moves. I read their papers and they use value function (NN) and policy function to trim the tree, so more ...
0
votes
0answers
23 views
What framework for a project with a custom environment?
I'm planning an RL project and I have to decide which RL framework do I use if any at all. The project has a highly custom environment, and testing different algorithms will be required to obtain ...
4
votes
1answer
98 views
What, exactly, does the REINFORCE update equation mean?
I understand that this is the update for the parameters of a policy in REINFORCE:
$$
\Delta \theta_{t}=\alpha \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) v_{t}
$$
Where š£š” is ...
2
votes
1answer
71 views
What is wrong with equation 7.3 in Sutton & Barto's book?
Equation 7.3 of Sutton Barto book:
$$\text{Equation: } max_s|\mathbb{E}_\pi[G_{t:t+n}|S_t = s] - v_\pi| \le \gamma^nmax_s|V_{t+n-1}(s) - v_\pi(s)| $$
$$\text{where }G_{t:t+n} = R_{t+1} + \gamma R_{t+2}...
0
votes
0answers
20 views
Reinforcement comparison optimality
The following is definition of reinforcement comparison, which updates an average reward and a preference for each action
http://incompleteideas.net/book/first/ebook/node22.html
I want to know if this ...
2
votes
1answer
41 views
Are linear approximators better suited to some tasks compared to complex neural net functions?
Model based RL attempts to learn a function $f(s_{t+1}|s_t, a_t)$ representing the environment transitions, otherwise known as a model of the system. I see linear functions are still being used in ...
0
votes
1answer
52 views
What does $r : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$ mean in the article Hindsight Experience Replay, section 2.1?
Taken from section 2.1 in the article:
We consider the standard reinforcement learning formalism consisting of an agent interacting with an environment. To simplify the exposition we assume that the ...
14
votes
2answers
3k views
What is the “Hello World” problem of Reinforcement Learning?
As we all know, "Hello World" is usually the first program that any programmer learns/implements in any language/framework.
As AurƩlien GƩron mentioned in his book that MNIST is often called ...
0
votes
1answer
18 views
Why would the reward of A3C with LSTM suddenly drop off after many episodes?
I am training an A3C with stacked LSTM.
During initial training, my model was giving descent +ve reward. However, after many episodes, its reward just goes to zero and is continuing for a long time. ...
1
vote
1answer
31 views
What are some suitable positive functions as activations of neural networks?
I am working on a deep Q-learning project. My project is different than normal deep Q-learning. The rewards of my neural network must be positive because I need their values to importance sample ...
3
votes
1answer
38 views
How to implement RL policies learned on a finite horizon?
I am modelling a ride-hailing system where passenger requests continuously arrive into the system. An RL model is developed to learn how to match those requests with drivers efficiently.
Basically, ...
0
votes
0answers
35 views
How to prove variance infinite of monte carlo ordinary importance sampling estimator
In example 5.5 of Sutton and Barto's book for proving infinite variance of first visit monte carlo ordinary importance sampling estimator,
$\mathbb{E}[(\Pi_t\frac{\pi(A_t|S_t)}{b(A_t|S_t)}G_0)^2]$ is ...
1
vote
0answers
43 views
Use of virtual worlds (e.g. Second Life) for training Artificial General Intelligence agents?
There is emerging effort for Third Wave Artificial Intelligence (Artificial General Intelligence) (http://hlc.doc.ic.ac.uk/3AI_HLC_2019.html and https://www.darpa.mil/work-with-us/ai-next-campaign) ...
0
votes
0answers
19 views
Reward and loss follow the same shape in DQN
If the accumulated reward increases, the loss increases and vice versa. This is a strange behaviour. See the figure below for an example. What is the possibility of having this behaviour in DQN?
I ...
0
votes
1answer
52 views
Can entire neural networks be composed of only activation functions?
Inverse Reinforcement Learning based on GAIL and GAN-Guided Cost Learning(GAN-GCL), uses a discriminator to classify between expert demos and policy generated samples.
Adversarial iRL, build upon GAN-...
1
vote
1answer
51 views
In Alpha(Go)Zero, why is the policy extracted from MCTS better than the network one?
I've read through the Alpha(Go)Zero paper and there is only one thing I don't understand.
The paper on page 1 states:
The MCTS search outputs probabilities Ļ of playing each move. These search ...
2
votes
1answer
25 views
Why do we need to go back to policy evaluation after policy improvement if the policy is not stable?
Above is the algorithm for Policy Iteration from Sutton's RL book. So, step 2 actually looks like value iteration, and then, at step 3 (policy improvement), if the policy isn't stable it goes back to ...
1
vote
2answers
85 views
What are some programming-oriented resources for reinforcement learning?
I have been reading :
Reinforcement Learning: An Introduction by Sutton and Barto.
I admit it's a good read for learning RL whereas it's more theoretical with detailed algorithms.
Now, I want ...
1
vote
1answer
83 views
Value Iteration failing to converge to optimal value function in Sutton-Barto's Gambler problem
In Example 4.3:Gambler's Problem of Sutton and Barto's book whose code is given here.
In this code the value function array is initialized as np.zeros(states) where ...
0
votes
0answers
33 views
Does Multi-Agent Deep Deterministic Policy Gradient also work with discrete action spaces?
I would like to ask if Multi-Agent Deep Deterministic Policy Gradient (MADDPG) works fine with discrete action space.
DDPG works only with continuous action space, but I have read that MADDPG can also ...
2
votes
0answers
27 views
Handling a Large Discrete Action Space in Deep Q Learning
I am attempting to solve a timetabling problem using deep Q learning. It could be thought of as a resource allocation problem to obtain some certificate of 'optimality'. However, how to define and ...
1
vote
1answer
46 views
What's a good neural network for this problem?
I am very new to the field of AI so please bear with me.
Say there is a dice with three sides, -1,0 and 1, and I want to predict which side it lands on (so only one output is needed I guess). The ...
0
votes
0answers
20 views
How does the second phase of the evaluation function described in this article work?
I am trying to create an evaluation function for a general game player based on the research from this article An Automatically-Generated Evaluation Function in General Game Playing. I can't ...
1
vote
1answer
76 views
Bellman optimality equation in semi Markov decision process
I wrote a Python program for a simple inventory control problem where decision epochs are equally divided (every morning) and there is no lead time for orders (the time between submitting an order ...
0
votes
0answers
10 views
Does normal A3C works well for continuous state space?
I am trying to create an A3C but it is giving same action for all the states during the training.
The same action is also not an obvious way to maximise the reward.
However I donot know if normal a3c ...
1
vote
0answers
28 views
How to understand this NN architecture?
I was reading a paper Multi-Agent Reinforcement Learning for Adaptive
User Association in Dynamic mmWave Networks and I was stuck understanding the deep neural network architecture that was used. The ...
1
vote
0answers
14 views
is it ok to take random actions while training a3c as in below code
i am trying to train an A3C algorithm but I am getting same output in the multinomial function.
can I train the A3C with random actions as in below code.
can someone expert comment.
...
3
votes
1answer
57 views
How can I fix jerky movement in a continuous action space
I am training an agent to do object avoidance. The agent has control over its steering angle and its speed. The steering angle and speed are normalized in a $[ā1,1]$ range, where the sign encodes ...
0
votes
0answers
22 views
Understanding GLIE conditions for epsilon greedy approach
I was going through this course on reignforcement learning (the course has two lecture videos and corresponding slides) and I had a doubt. On slide 18 of this pdf, it states following condition for an ...
0
votes
0answers
33 views
Generalized advantage estimation DRL PPO
I have a question about generalized advantage estimation in multi agent environment. According to the formula I've computed GAE in this way:
...
2
votes
1answer
38 views
How do I design the rewards and penalties for an agent whose goal it is to explore a map
I am trying to train an agent to explore an unknown two-dimensional map while avoiding circular obstacles (with varying radii). The agent has control over its steering angle and its speed. The ...
3
votes
2answers
170 views
Why is the target called “target” in Monte Carlo and TD learning if it is not the true target?
I was going through Sutton's book and, using sample-based learning for estimating the expectations, we have this formula
$$
\text{new estimate} = \text{old estimate} + \alpha(\text{target} - \text{old ...
2
votes
1answer
44 views
How we are calculating average reward ($r(\pi)$) if the policy changes over time?
In the average reward setting the quality of a policy is defined as:
$$ r(\pi) = \lim_{h\to\infty}\frac{1}{h} \sum_{j=1}^{h}E[R_j] $$
When we reach the steady state distribution we can write the above ...
1
vote
1answer
39 views
What is the optimal exploration-exploitation trade-off in Q*bert?
I am training an RL agent with Deep Q-learning + Experience Replay on the Q*bert Atari environment. After 400,000 frames, my agent appears to have learned strategic information about the game, but ...
2
votes
1answer
65 views
Why is sampling non-uniformly from the replay memory an issue? (Prioritized experience replay)
I can't seem to understand why we need importance sampling in prioritized experience replay (PER). The authors of the paper write on page 5:
The estimation of the expected value with stochastic ...
3
votes
2answers
69 views
Why does (not) the distribution of states depend on the policy parameters that induce it?
I came across the following proof of what's commonly referred to as the log-derivative trick in policy-gradient algorithms, and I have a question -
While transitioning from the first line to the ...
2
votes
1answer
228 views
What's the optimal policy in the rock-paper-scissors game?
A deterministic policy in the rock-paper-scissors game can be easily exploited by the opponent - by doing just the right sequence of moves to defeat the agent. More often than not, I've heard that a ...
2
votes
0answers
30 views
What's an example of a simple policy but a complex value function?
Hado van Hasselt, a researcher at DeepMind, mentioned in one of his videos (from 7:20 to 8:20) on Youtube (about policy gradient methods) that there are cases when the policy is very simple compared ...
0
votes
1answer
27 views
How to calculate v min and v max for C51 DQN
Background: In C51 DQNs you must specify a v-min/max to be used during training. The way this is generally done is you take the max score possible for the game and set that to v-max, then v-min is ...
