I am working with a dataset where each input sample is a matrix, and the output corresponding to each input is also a matrix (of shape
(400, 10)). The input samples do not have translation invariance. Each output image has shape
(16, 16) . The output matrices have translation invariance.
I want to build a neural network which can learn how to predict the output images from the aforementioned data. It seems to me that one needs to think of a neural network here which does regression on the output images to learn. Presently, I am using 1000 data samples for learning in the neural network (input samples and corresponding output images). What is the best way to build a neural network for this task?
Presently, I am using multi-layer perceptron (MLP) with mean square error (MSE) loss for this task. I flatten the input matrices before I feed them into the MLP, and use a standard MLP with multiple hidden layers (4-5) with many hidden units for this task.
While a visual inspection shows that the true and predicted output images are in relatively good agreement for training data, I find a mismatch between true and predicted (reconstructed) output images for validation data. The bottom plot shows pictures of true and predicted (reconstructed) output images for training and validation data for chosen samples.
Presently, I am using training and validation loss curves (with respect to iteration) to measure performance. I want to have a robust metric for comparison which can tell me whether the prediction is a random image or not.
How can I get the model to generalize to the validation set better?