# Tag Info

5

If your algorithm is executed multiple (or enough) times using an outer loop, it would converge to similar results as Q-learning would with $\gamma = 0$ (as you don't look what is the expected future reward). In this case, the difference is that you would pass as much time to explore each possible couple of (state, action) while Q-learning would pass more ...

5

Q-learning is guaranteed to converge (in the tabular case) under some mild conditions, one of which is that in the limit we visit each state-action tuple infinitely many times. If your random random policy (i.e. 100% exploration) is guaranteeing this and the other conditions are met (which they probably are) then Q-learning will converge. The reason that ...

4

I am going to stick with Q learning here to keep things simple. Most value-based reinforcement learning used for optimal control will have some statement similar to: Choose $a$ from $s$ using policy derived from $Q$ First, yes this is always the current Q function or Q table, evaluated for the state of interest. When you are choosing the agent's best guess ...

4

for example, the "greedy policy" always chooses the action with the highest expected return, no matter which state we are in The "no matter which state we are in" there is generally not true; in general, the expected return depends on the state we are in and the action we choose, not just the action. In general, I wouldn't say that a policy is a mapping ...

3

I'll assume Q-player is being trained with Q learning (note, Q tables can be useful in other algorithms too, like SARSA). Q learning is an off policy algorithm, meaning that the Q values can be learned regardless of the policy used to collect data. So the Q player can be following a random policy, or even a fixed pre defined policy if you want. Usually, ...

3

Many techniques for the exploration/exploitation dilemma that are inspired by multi-armed bandit problems, such as UCB1, assume that you can explicitly enumerate all state-action pairs; in fact, multi-armed bandit problems usually only have just one "state", and then this requirement turns into only requiring the ability to enumerate actions. In RL ...

3

You can indeed use UCB in the RL setting. See e.g. section 38.5 Upper Confidence Bounds for Reinforcement Learning (page 521) of the book Bandit Algorithms by Csaba Szepesvari and Tor Lattimore for the details. However, compared to $\epsilon$-greedy (widely used in RL), UCB1 is more computationally expensive, given that, for each action, you need to ...

3

For single-step Q learning, the behaviour policy can be any stochastic policy without any further adjustment to the update rules. You don't have to use $\epsilon$-greedy based on current Q function approximation, although that is a common choice because it works well in general cases. However, you should always allow some chance of taking all actions if you ...

2

In addition to the RF [*], you also need to define an exploratory policy (an example is the $\epsilon$-greedy), which allows you to explore the environment and learn the state-action value function $\hat{q}$. Moreover, although you don't need to know the details (i.e. the specific probabilities of transitioning from one state to the other) of the transition ...

2

In short, yes, provided that you have a small number of states. In pretty much any real system, the number of states is much higher than you could ever hope to explore exhaustively in any reasonable time. This is why you need to set some sort of exploration/exploitation policy to make sure that you mostly visit promising states while also checking states ...

1

In part it depends on the on-policy method you are using. In general you are not free to change the policy arbitrarily for on-policy policy gradient methods such as PPO or A3C. However, if you are willing to consider the added exploration strategy as part of the current target policy, and can express it mathematically, you should be able to add an ...

1

I'm not familiar with your game so I can't tell you what a good heuristic woul be in your specific case, but I can give you some advice on how to look for a good heuristic function. As a rule of thumb, the heuristic function for a MiniMax algorithm is best kept simple and efficient, so you can get deeper into the tree. But it depends on how costly it is to ...

Only top voted, non community-wiki answers of a minimum length are eligible