If you know it is symmetric, then you could do a couple things.
Zero out a half.
Don't bother learning both halves of the image. Just put a zero mask over the upper or lower half of the output matrix and just have the network regress the other half. Just don't make the network do more work than it needs to do.
Learn both, but add symmetric loss
In your ...