Suppose that in version 1 of a reinforcement-learning system an optimal policy $A$ got generated for executing a task. But, in a newer version 2 of that application (with new code changes), there might be some policy $B$ that would do slightly (1-2%) better than policy $A$.
How do you allow the system to learn that "better" policy $B$? I think the answer is retraining.
But during the training process, the old policy $A$ might still keep accumulating rewards delaying policy $B$ to be recognised as the "better" policy than $A$. This could get worse if each newer version of the system would contain a better policy which is only slightly better than the previous release's best policy. It would take a very long time to find the best policy.
Is this accepted in real-world RL systems? Or should I be figuring out a way to tell the system that "Hey, there might be a better policy somewhere, try to find that instead of rewarding existing policies."?