Should the actor or actor-target model be used to make predictions after training is complete (DDPG)?

Question

The situation

I am referring to the paper T. P. Lillicrap et al, "Continuous control with deep reinforcement learning" where they discuss deep learning in the context of continuous action spaces ("Deep Deterministic Policy Gradient").

Based on the DPG approach ("Deterministic Policy Gradient", see D. Silver et al, "Deterministic Policy Gradient Algorithms"), which employs two neural networks to approximate the actor function mu(s) and the critic function Q(s,a), they use a similar structure.
However one characteristic they found is that in order to make the learning converge it is necessary to have two additional "target" networks mu'(s) and Q'(s,a) which are used to calculate the target ("true") value of the reward:

y_t = r(s_t, a) + gamma * Q'(s_t1, mu'(s_t1))

Then after each training step a "soft" update of the target weights w_mu', w_Q' with the actual weights w_mu, w_Q is performed:

w' = (1 - tau)*w' + tau*w

where tau << 1. According to the paper

This means that the target values are constrained to change slowly, greatly improving the stability of learning.

So the target networks mu' and Q' are used to predict the "true" (target) value of the expected reward which the other two networks try to approximate during the learning phase.

They sketch the training procedure as follows:

The question

So my question now is, after the training is complete, which of the two networks mu or mu' should be used for making predictions?

Equivalently to the training phase I suppose that mu should be used without the exploration noise but since it is mu' that is used during the training for predicting the "true" (unnoisy) action for the reward computation, I'm apt to use mu'.

Or does this even matter? If the training was to last long enough shouldn't both versions of the actor have converged to the same state?

Answer 1

It does not matter, target should converge to the main network after long enough training.

Answer 2

So my question now is, after the training is complete, which of the two networks mu or mu' should be used for making predictions?

After training is "complete", use mu, i.e., the online network because this is the most updated actor network you have. In the most ideal case, mu' will be equal to mu. But if your training is truly complete, mu = mu'.

Equivalently to the training phase I suppose that mu should be used without the exploration noise but since it is mu' that is used during the training for predicting the "true" (unnoisy) action for the reward computation, I'm apt to use mu'.

In the training phase, you must use exploration noise when mu maps states to actions. The actor is a policy gradient, i.e. based on policy iteration. So it will directly map to an action everytime. If no exploratory noise is added, it will map to the same value every time, unless you update your network weights. What this means is that if you're given some states, s. Without exploration noise, the actor will always output the same actions, and never explore.

mu' is ONLY used for the supervised learning portion of DDPG, where it stabilizes training of 2 neural networks that essentially train off each other.

Imagine this problem: a worker and his supervisor in a factory. Both the worker and the supervisor knows nothing at the start (initialized neural networks). The worker's first action is to pick up a 200 lb box. The supervisor then positively rewards him. So from this, the worker learned that picking up heavy objects = good. The worker then picks up another heavy box, and this time the supervisor yells at him because the supervisor learned that its dangerous. So now the worker himself is confused, because he does not know if picking up heavy objects is good or bad because the supervisor himself changes his mind.

In DDPG, the actor and critic behaves the same way. So we introduce target networks to make it so both the actor and critic don't keep changing their minds, so they can actually learn things.

Or does this even matter? If the training was to last long enough shouldn't both versions of the actor have converged to the same state?

That is correct, but in reality, that is very rare.