New answers tagged q-learning
2
Why is this a convergence criterion?
It is because $R$ and $S'$ are stochastic. A large learning rate applied when these values have variance would not converge to mean, but would wander around typically within some value proportional to $\alpha\sigma$ of the true value, where $\sigma$ is the standard deviation of the term $R + \gamma\text{max}_aQ(S',a)$. ...
1
I will try to explain this problem with the very tangible example of chess. In chess, the number of possible states is any configuration that you can make with the pieces on the board. So, the starting position is a state, and after you did one move you are in a different state. The total number of chess states is more than $10^{100}$. It is therefore very ...
2
Having too many states to actually visit is a common problem in RL. This is exactly why we often use function approximation. If you replace your q table with a good function approximator such as a neural network, it should be able to generelize well to states it has not yet encountered.
If you do not use a function approximator but stick with a table, the ...
2
Q-learning uses the maximizing value at each step,
Mostly true. The target policy that Q-learning learns the action values of is the one with the maximum value. While training, Q-learning will take one action randomly (typically with a high probability of taking the action with maximum value), whilst it makes updates to value estimates assuming it will ...
0
Q-learning is an off-policy learning algorithm. We are following the behaviour policy, $b$, which is $\epsilon-$greedy. This behaviour policy need not be an optimal policy rather it is a more explorable policy. But we are learning the target policy, $\pi$, which is argmax of state action value $(Q(s,a))$. This target policy is by definition optimal policy.
...
4
I know this might be specific to different problems but does anyone know if there is any rule of thumb or references on what constitutes a large state space?
Not really, it is all relative. There are two main ways in which the scale of a value table might be too much:
Memory required to represent the table. This is relatively simple to calculate for any ...
4
This post contains many answers that describe the difference between on-policy vs. off-policy.
Your book may be referring to how the current (DQN-based) state-of-the-art (SOTA) algorithms, such as Ape-X, R2D2, Agent57 are technically "off-policy", since they use a (very large!) replay buffer, often filled in a distributed manner. This has a number ...
1
What does it mean when ϵ=0 and ϵ=1? If ϵ=1, does it mean that the agent explores randomly? If this intuition is right, then it will not learn anything - right? On the other hand, if I set ϵ=0, does this imply that the agent doesn't explore?
You are correct, when ϵ=1 the agent acts randomly. When ϵ=0, the agent always takes the current greedy actions. Both ...
1
For off-policy learning you must have two policies - a behaviour policy and a target policy. If the two policies are the same, then you end up with SARSA, not Q learning.
You cannot use Q learning directly for evaluating a fixed target policy, because it directly learns optimal value function as the target policy, regardless of the behaviour policy. Instead ...
0
I don't know what is your dataset exactly look like. But based on assumption, I would like to refer something --
You can think your MDP environment this way
action = {stay, go}
reward = {something like based on visitor's satisfaction maybe rating}
state = {current money in hand, city, other some variable those key feature to make next iteration action}
I ...
Top 50 recent answers are included
