Background
I am working on a robotic arm controlled by a DQN + a python script I wrote. The DQN receives the 5 joint states, the coordinates of a target, the coordinates of the obstacle and outputs the best action to take (in terms of joint rotations). The python script checks if the action suggested by the DQN is safe. If it is, it performs it. Otherwise, it performs the second highest-ranking action from the DQN; and so on. If no action is possible, collision: we fail.
During training, this python functionality wasn't present: the arm learned how to behave without anything else to correct his behaviour. With this addition on the already-trained network, the performance raised from 78 to 95%. Now my advisor (bachelor's thesis) asked me to leave the external controller on during training to check whether this improves learning.
Question
Here's what happens during training; at each step:
- the ANN selects an action
- if it is legal, the python script executes it, otherwise it chooses another one.
Now... On which action should I perform backprop? The one proposed by the arm or the one which was really executed? (So, which action should I perform backprop on?)
I am really confused. On the one hand, the arm did choose an action so my idea was that we should, in fact, learn on THAT action. On the other hand, during the exploration phase ($\epsilon$ greedy), we backprop on the action which was randomly selected and executed, with no interest on what was the output of the arm. So, it would be rational too, in this case, to perform backprop on the action really executed; so the one chosen by the script.
What is the right thing to do here?. (Bonus question: is it reasonable to train with this functionality on? Wouldn't it be better to train the Network by itself, and then later, enhance its performance with this functionality?)