In deep Q-learning we typically use epsilon-greedy policy during training. We choose a random action for a certain probability $\epsilon$, and choose the action that maximize the current Q-value estimate for probability $1-\epsilon$, and the value of $\epsilon$ decays slowly over the training process
I am wondering if directly swapping the random action with an expert policy can speed-up the Q-learning process?
For convergence to optimal value functions and optimal policies to be guaranteed in general, you need action coverage in a stochastic exploration policy. Typically that means implementing some kind of $\epsilon$-soft policy but not necessarily $\epsilon$-greedy. Without this, the agent will never look at the off-policy options that might be better. The agent also needs data about bad actions that it should not take, otherwise it will not learn to avoid them by itself.
In general, swapping the $\epsilon$-greedy policy out for a deterministic expert policy will not work. However, there are a few ways that it could be made to work:
The expert policy is stochastic and has coverage, but prefers "better" actions. You still need full coverage for the Q-learning agent to estimate returns from bad actions, unless the expert policy is also going to be applied in production to avoid those bad actions.
Instead of starting with an $\epsilon$-greedy policy and a high value e.g. $\epsilon = 1$, start with the expert policy for some fixed number of iterations, enough for the agent to approximately learn the value function for the expert policy. Then switch back to $\epsilon$-greedy with a relatively low value for exploration e.g. $\epsilon = 0.1$. This will initialise your agent with knowledge from the expert and help speed up leaning if that expert policy is any good.
If you have good knowledge of when the expert policy is truly optimal, and can detect that from the state, then use it instead of Q values in those states. You may still want an $\epsilon$ probability of taking random action instead applied to that choice, depending on whether you want the agent to properly learn about the worse alternatives. I have successfully used this approach for an agent learning a board game, where minimax searches that find forced end game results can easily be tracked and used in preference to Q tables when they apply.
If the environment is highly stochastic - e.g. in a board game using dice - it may not matter that you are using a deterministic fallback policy instead of $\epsilon$-greedy, because the agent will see enough variation in states despite not actively exploring itself.
If the expert policy is not necessarily optimal, but is perfect at avoiding terrible decisions, use it to filter available actions. This would need to be done in production as well as during learning, because the agent will never learn about those terrible decisions unless it experiences them. Although you could also fake that experience and feed it to the update step if the expert is also very good as emulating what would happen - this might be possible if the expert can reliably detect that the episode would end with a certain reward immediately for example.
Any hybrid variation of the above ideas, such as switching between three policies: greedy Q-table, expert and random choice based on evolving probabilities and knowledge of when the expert policy is best applied.
