harwiltz
  • Member for 1 year, 7 months
  • Last seen more than a month ago
What is the difference between reinforcement learning and optimal control?
5 votes

As a supplement to nbro's nice answer, I think a major difference between RL and optimal control lies in the motivation behind the problem you're solving. As has been pointed out by comments and ...

View answer
Is the dropout technique specific only to neural networks?
Accepted answer
4 votes

I'm sure you can use dropout in any parameterized model, but I suspect it'll only really be helpful if you have enough parameters/nodes. Also dropout in neural nets has a Bayesian meaning, Yarin Gal ...

View answer
Into which subcategories can reinforcement learning be divided?
Accepted answer
4 votes

Your two suggestions are not mutually exclusive. If you go by this process, you'll have to do a "Cartesian product" of a bunch of different RL categorizations which would get out of hand. I ...

View answer
What is the bias-variance trade-off in reinforcement learning?
Accepted answer
3 votes

The bias-variance trade-off that you're referring to has to do with the return estimator. Any RL algorithm you choose needs some estimate of the cumulative return, which is a random variable with many ...

View answer
Which policy has to be followed by a player while construction of its own Q-table?
Accepted answer
3 votes

I'll assume Q-player is being trained with Q learning (note, Q tables can be useful in other algorithms too, like SARSA). Q learning is an off policy algorithm, meaning that the Q values can be ...

View answer
Why is the optimal policy for an infinite horizon MDP deterministic?
Accepted answer
3 votes

Suppose you learned your action-value function perfectly. Recall that the action-value function measures the expected return after taking a given action in a given state. Now, the goal when solving an ...

View answer
How does an episode end in OpenAI Gym's "MountainCar-v0" environment?
Accepted answer
3 votes

The episode ends when either the car reaches the goal, or a maximum number of timesteps has passed. By default the episode will terminate after 200 steps. You can customize this with the ...

View answer
What is the purpose of a Neural Network in Reinforcement Learning when we have a Q-learning update rule?
3 votes

I don't think people generally do use neural nets for grid world. As long as the state and action spaces are small enough, you should be able to store Q values in a table like you suggested. Neural ...

View answer
What is convergence analysis, and why is it needed in reinforcement learning?
3 votes

Convergence analysis is about proving that your policy and/or value function converge to some desired value, which is usually the fixed-point of an operator or an extremum. So it essentially proves ...

View answer
Why AlphaGo didn't use Deep Q-Learning?
3 votes

Deep Q Learning is a model-free algorithm. In the case of Go (and chess for that matter) the model of the game is very simple and deterministic. It's a perfect information game, so it's trivial to ...

View answer
What do the terms 'Bellman backup' and 'Bellman error' mean?
Accepted answer
2 votes

A Bellman backup is an application of a Bellman operator. For example, the step $$ V(x)\leftarrow \alpha(R + \mathbf{E}[V(x')]) + (1-\alpha)V(x) $$ Is a Bellman backup for some learning rate $\alpha$. ...

View answer
Are policy and value iteration used only in grid world like scenarios?
Accepted answer
2 votes

Policy and value iteration both require you to, for each possible transition and each corresponding possible reward at each state, compute a statistic of $r + \gamma V(s')$. In order for this to be ...

View answer
When should one prefer using Total Variational Divergence over KL divergence in RL
2 votes

To add to nbro's answer, I'd say also that much of the time the distance measure isn't simply a design decision, rather it comes up naturally from the model of the problem. For instance, minimizing ...

View answer
How is weighted average computed in Deep Q networks
2 votes

Let's say $Q$ is the old estimate, $Q'$ the new estimate, and $R$ is the return. We have $$ Q' = Q + \alpha(R-Q) $$ This can be rewritten as $$ Q' = (1-\alpha)Q + \alpha R $$ When $\alpha$ is a ...

View answer
How to optimize neural network parameters with REINFORCE
2 votes

The term REINFORCE actually corresponds to a method of estimating gradients, it is not particular to reinforcement learning. The paper you linked doesn't appear to deal with RL at all, so the issue ...

View answer
How can I classify policy gradient methods in RL?
Accepted answer
2 votes

DP, Monte Carlo, and TD are methods of estimating returns. Policy gradient describes methods of learning a policy. So policy gradients serve a different purpose than the other things you mentioned. ...

View answer
How can I read any AI paper?
2 votes

I think the best way to make reading papers easier is to practice (as in, read lots of papers, try implementing them, etc), and to discuss them with other students/researchers. Sometimes it's tough to ...

View answer
How do I calculate the return given the discount factor and a sequence of rewards?
2 votes

You know all the rewards. They're 5, 7, 7, 7, and 7s forever. The problem now boils down to essentially a geometric series computation. $$ G_0 = R_0 + \gamma G_1 $$ $$ G_0 = 5 + \gamma\sum_{k=0}^\...

View answer
Why can't we fully exploit the environment after the first episode in Q-learning?
2 votes

BlueTurtle's answer is good, but I'd like to add something. Your question realistically has nothing to do with Q Learning, in fact, you can ask the same thing about just about any RL algorithm. In ...

View answer
Is the Bandit Problem an MDP?
1 votes

The bandit problem is an MDP. You can make the same argument about needing data to learn in the stateful MDP setting. The thing is, the data you need (the past rewards in this case) was drawn iid (...

View answer
Reinforcement learning simple problem: agent not learning, wrong action
1 votes

I see some issues in your code of the environment. Firstly, and probably most importantly, you should not be incrementing the reward. In your code, every time the agent hits $t=475$ for example, the ...

View answer
What are the available exploration strategies for continuous action space scenarios in RL?
1 votes

Firstly, note that the Gaussian policies you describe are not equivalent to $\epsilon$-greedy, mainly for this reason: for a fixed policy, the policy's variance in the Gaussian case depends on the ...

View answer
Why one unit in the layers of neural network is not enough?
1 votes

So if I understand correctly, you're proposing to use a neutral net with $N$ input units (let's say data is in $\mathbf{R}^N$), 1 hidden unit, and whatever the necessary output needs to be. Let's say ...

View answer
How to calculate v min and v max for C51 DQN
Accepted answer
1 votes

If you're using a discount factor less than 1, you should be able to compute a maximum return (likewise, a minimum return) based on the max (min) reward you can earn at each timestep. However, this ...

View answer
What is the expectation of an empirical model in model based RL?
Accepted answer
1 votes

If I understand your question correctly, the significance of this is due to the fact that $s'$ is random. In the RHS of the equation it is assumed that $V(\cdot)$ is known for each state, but the ...

View answer
If the transition model is available, why would we use sample-based algorithms?
1 votes

A full Bellman update can be intractable. For instance, if your state space or action space are continuous, the full Bellman update is intractable. You can try to solve this by discretizing, but if ...

View answer
How should we interpret all the different metrics in reinforcement learning?
1 votes

As you said, generally the most important one is reward per episode. If this isn't increasing overall, there's a problem (of course this metric can fluctuate, I mean to say that macroscopically it ...

View answer
How does training for DQN work if messing up in the environment in costly?
Accepted answer
1 votes

In general, you need to actively explore the environment to gather data to train your Q network. However, especially in your self-driving car example, you might be looking for Batch RL. In Batch RL ...

View answer
How do I test an LSTM-based reinforcement learning model using any Atari games in OpenAI gym?
1 votes

If I understand your problem correctly, you can test on just about any environment, and just omit parts of the observations to ensure your RNN is learning. For example, you can test on cartpole, ...

View answer
Should I use exploration strategy in Policy Gradient algorithms?
1 votes

Neil Slater's answer is very nice, but I have a couple more suggestions: You can use entropy regularization. Basically, you modify your loss function to penalize low policy entropy (so less loss for ...

View answer