As part of a bigger scope project, I'm training a RL agent that attempts to reconstruct, pixel by pixel, the trajectory of a neuron on a segmented image. To give a better insight on the task that I'm trying to achieve, here is an example of an attempt on simple vessel ground truth image (red : vessel/ground truth, green : agent trajectory, yellow : overlap between the two) :
Details on the environment and the model :
At each step, the agent is picking an action based on the output of a Deep Q-Network and chooses the next pixel to move the cursor to. It can move in any of the 8 directions (in diagonal too). The goal is to match the red curve with the green one by updating each step the position of the cursor. At the moment, I'm training the agent with a few different such ground truth images and for a fair amount of episodes and it's really struggling at sticking close to the red line. I set a maximum number of pixels to stop the agent after a while in case it doesn't approximately match the red line (which it does not). I'm used to applying RL to games like Atari etc, but here it's a bit more abstract to determine what would be a smart way to design the reward system.
At the moment, I'm using the completeness score that compares what fraction of the pixels of the dilated red line are covered by the agent's path as part of my reward system. If the completeness increases as compared to the previous step, I give a reward of +1, if it doesn't, I give a reward of -0.1 (to penalize the agent adding useless pixels). I've tried a few other things like directly using the completeness score as the reward, but this technique is the overall best I can achieve, but I believe that my system is not very optimal which might cause the poor performance, as I think that the design of the reward system is essential to the good performing of a RL agent. I'm therefore open to any suggestions on how to improve or re-think the reward system by anyone that's got a bit more experience than me in designing RL agents when it's not as obvious as for some simple games.
Thanks in advance for your help !