Do I need to store the policy for RL?

Question

I am creating a zero-sum game with RL and wondered if I need to store the policy, or if there are other RL methods that produce similar results (consistently beating the human player) without the need to store the policy and comes the correct decision 'on the fly' - would this be this off-policy?

Answer 1

If your game agent performs any kind of advance learning from self play or database of moves, that will generate parameters for some kind of model (e.g. a table of expected values, or neural network weights to select a preferred action). This is unavoidable, and if you want to re-use the results of that machine learning, you absolutely have to store the parameters somewhere.

if there are other RL methods that produce similar results (consistently beating the human player) without the need to store the policy and comes the correct decision 'on the fly' - would this be this off-policy?

If your agent can access a model that accurately predicts (or accurately samples) outcomes from actions, it can look ahead and plan from the current state. Typically board games and the like do allow you to have such a model, based on the game rules. Some look-ahead techniques are essentially RL methods applied with a focus on solving a decision "just in time", others are more related to search. Common techniques used in game playing are A* search, minimax search (with alpha-beta pruning for performance improvement), Monte Carlo Tree Search. Used purely with just some pre-coded heuristics and a game model, these search techniques do not require you to store model parameters. The downside is that you must spend more computing resources per game move in order to run the search/planning and drive the policy. However if your game is simple, or your heuristics good, or you don't mind a significant wait per computer move, then this approach can be very effective.

This difference between learning and planning is not directly about on-policy vs off-policy, although planning may be considered off-policy as it assesses many actions, most of which the agent does not take.

What a planning method typically looks like in terms of data is a memory-based representation of a game tree starting from the current position, with internal "scores" very similar to RL value functions used to track consequences of action choices. Search and planning methods each have different ways to prune the tree down to a reasonable size and select which game states and actions to look ahead further in. Using "pure" search allows you to work exclusively with this temporary in-memory representation, discarding it after it selects a next action.

You have probably heard of AlphaGo, AlphaGo Zero etc, which are state-of-the-art game playing agents that learn through self play. These, and similar agents, use a combination of a learned policy (from self-play), plus a look-ahead search which refines it for the current game position, getting the best of both learning and planning. The parameters for any learned policy do have to be stored of course. But the benefit of the combined approach is that you can balance resources placed into general learning of the game (stored as parameters to a value function or policy) and specific choice of action for a current game position, which can be much more focussed.