You can try doing image segmentation the traditional way, just using the image data. If you want to use the non-image data, then, you can introduce classification as another task for your network. It will provide some regularization to your model. But, this is one way you can still use non-image data whilst still working with image outputs.
It would be better to run experiments on this. But, here's answer analytically.
The models will be different.
With augmentation, the network starts to learn to combat the noise too.
With late onset augmentation, the network will start to deviate from its original solution to combat noise.
Comparing this to the ball rolling down the hill, in case of ...