# Importance of initialisation of State-Action/State values in RL

I was wondering is there any empirical/theoretical evidence on the effect of initial values of State-Action/State values on the training of an RL agent (the values an RL agent assigns to visited states) via MC methods Policy Evaluation and GLIE Policy Improvement.

For example, consider 2 initialisation scenarios of Windy Gridworld problem:

Implementation: I have modified the problem along with step penalty to include a non-desired terminal state and a desired terminal state which will be conveyed to the agent as a negative and positive reward state respectively, and the implementation takes care that the MC sampling ends at the terminal state and gives out penalty/reward as a State-Action value and not State value since this is a control problem. Also I have 5 moves, North, South, East, West, Stay.

NOTE: I am not sure whether this changes the objective of the problem. In the original problem it was to reduce the number of steps required to reach final stage.

• We initialise the reward/penalty as a value which is higher to the values randomly initialised to the State-Action pairs, example, initialise reward as $$20$$ and the random initialisations as values from $$1$$ to $$7$$.

• We initialise the reward/penalty as a value which is comparable to the values randomly initialised to the State-Action pairs, example, initialise reward as $$5$$ and the random initialisations as values from $$1$$ to $$10$$

As far I see in the first case the algorithm will easily quickly converge as the reward is very high for the terminal reward state which will skew the agent to try to reach the reward stage.

In the second case, this might not be true if the reward state is surrounded by other high reward states, the agent will try to go to those states. The step penalty ensures that the agent finally reaches the terminal state, but will this skew the path of the agent and severly affect its convergence time? This might be problematic in large state spaces since we will not be able to explore the entire state space, but the presence of exploratory constant $$\epsilon$$ might derail the training by going to a large false reward state. Is my understanding correct?

• Feel free to clarify the question as it might not be very clear what I am asking especially the terminologies. – DuttaA Jul 6 '19 at 20:47
• Can you clarify what you mean with initialising "rewards/penalties" and "values"? I think talking about initialisation of "values" makes sense if you're referring to a table of $Q(s, a)$ value estimates for state-action pairs, or a table of $V(s)$ value estimates for states, and if those same table entries are the ones you are subsequently adjusting with your RL algorithm. I have no idea what you mean with "initialising rewards" though... normally, the rewards are inherent to the problem domain (though there are things like reward shaping and auxiliary objectives I suppose). – Dennis Soemers Jul 7 '19 at 12:07
• @DennisSoemers yes the rewards are known. Now I want to reach a desired state and avoid a non-desired state, so $Q(s,a)$ for those States will be +reward and -penalty irrespective of the action. But here I am talking about the initialization of $Q(s,a)$ of the rest of the positions and their respective actions. – DuttaA Jul 7 '19 at 14:08
• @DennisSoemers since the reward and penalty is merely a subjective concept. For example in normal gridworld example if we change the step penalty to +5 instead of +1 all the state values just get multiplied by a 5 factor. So here I have chosen the term initialize for reward and penalties as well as other state values, feel free to edit in a suitable term. – DuttaA Jul 7 '19 at 14:12
• Right. The confusion with the term "initialise" there is that the term implies you're just selecting an "initial" value for those variables, that it's something that will change over time. Initialising $Q(s, a)$ variables to certain values makes sense because we expect them to change afterwards (during training). The rewards assign to certain states or state-action pairs typically do not change over time, so it's not so much "initialising" them as it is just "selecting" or "assigning" values to them – Dennis Soemers Jul 7 '19 at 14:39

## 1 Answer

There seem to be two different ideas in this question here:

1. What's the impact / importance of our choice for reward values?
2. What's the impact / importance of our choice for initial value estimates (how do we initialise our table of $$Q(s, a)$$ values in the case of a simple, tabular RL algorithm like Sarsa or $$Q$$-learning)?

The reward values are typically assumed to be a part of the problem definition - something we shouldn't modify if we're using an existing problem definition as a benchmark. But if we're in charge of defining the problem ourselves, we can of course also pick the reward values. Modifying them may indeed have a huge impact on the speed with which RL algorithms are able to learn a task - but it may also intrinsically changes the task, it changes the objective of the problem, it may change which policies are optimal.

As for initialisation of our table of value approximations: by default, we normally assume an all-$$0$$ initialisation. However, it is a fairly common trick (in tabular RL algorithms, without function approximation) to initialise value estimates optimistically; pick higher initial $$Q(s, a)$$ value estimates than are likely (or even pick values higher than a known upper bound on what the true value possibly could be). This is often beneficial - also in large gridworlds with sparse rewards (e.g. a single distant goal somewhere) and negative rewards (i.e. costs) incurred for every step taken - because it incentivises exploration of state-action pairs that have not yet been tried.

Suppose you have your gridworld with negative rewards associated with every time step, and the optimal policy being one that takes you to a distant goal as soon as possible. If all $$Q(s, a)$$ are initialised to $$0$$ (or worse, to negative values), your agent may quickly learn that everything it does is equally bad anyway, and get stuck near the starting position. If all $$Q(s, a)$$ values are initialised optimistically (to large, positive values), your agent during the learning process will still have optimistic expectations of what it can achieve if it just tries to navigate to unexplored parts of the state-action space.