I was wondering is there any empirical/theoretical evidence on the effect of initial values of State-Action/State values on the training of an RL agent (the values an RL agent assigns to visited states) via MC methods Policy Evaluation and GLIE Policy Improvement.
For example, consider 2 initialisation scenarios of Windy Gridworld problem:
Implementation: I have modified the problem along with step penalty to include a non-desired terminal state and a desired terminal state which will be conveyed to the agent as a negative and positive reward state respectively, and the implementation takes care that the MC sampling ends at the terminal state and gives out penalty/reward as a State-Action value and not State value since this is a control problem. Also I have 5 moves, North, South, East, West, Stay.
NOTE: I am not sure whether this changes the objective of the problem. In the original problem it was to reduce the number of steps required to reach final stage.
We initialise the reward/penalty as a value which is higher to the values randomly initialised to the State-Action pairs, example, initialise reward as $20$ and the random initialisations as values from $1$ to $7$.
We initialise the reward/penalty as a value which is comparable to the values randomly initialised to the State-Action pairs, example, initialise reward as $5$ and the random initialisations as values from $1$ to $10$
As far I see in the first case the algorithm will easily quickly converge as the reward is very high for the terminal reward state which will skew the agent to try to reach the reward stage.
In the second case, this might not be true if the reward state is surrounded by other high reward states, the agent will try to go to those states. The step penalty ensures that the agent finally reaches the terminal state, but will this skew the path of the agent and severly affect its convergence time? This might be problematic in large state spaces since we will not be able to explore the entire state space, but the presence of exploratory constant $\epsilon$ might derail the training by going to a large false reward state. Is my understanding correct?