2
$\begingroup$

I am working on an image to image regression task which requires me to develop a deep learning model that takes in a sequence of 5 images and return another image. The sequence of 5 images and the output images are conceptually and temporally related. In fact, the 5 images in the sequence each correspond to timestep in a simulation and the output, the one I am trying to predict, corresponds eventually to the 6th timestep of that sequence.

For now, I have been training a simple regression-type CNN model which takes in the sequence of 5 images stored in a list and outputs an image corresponding to the next timestep in the simulation. This does work with a small and rather simple dataset (13000 images) but works a bit worse on a more diverse and larger dataset (102000 images).

For this reason, I have been researching a bit now in order to find a better way to carry out this task and I found the idea of ConvLSTMs. However, I have seen these applied to the prediction of feature and the output of a sentence describing that image. What I wanted to know is whether ConvLSTMs can also output images, but more importantly if they can be applied to my case. If not, what other types of deep learning network can be suitable for this task?

Thanks in advance!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

If you only need 5 frames to predict the next frame, then I'd recommend a U-Net architecture, wich is basically a CNN encoder/decoder network in which the decoder uses the intermediate features produced in the encoder as well as its own features to produce an output image. Also, in additional to using a conventional L2 loss for the output image, you can always add an additional GAN loss to make the image look more realistic.

If using a longer history of frames can help, then I recommend taking a look at "Recurrent Environment Simulators" and combining it with the ideas above.

Hope it helps!

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks a lot for your answer @Mehran Shakerinava, I can't bump up your answer yet becuase my reputation is under 15, but still thanks. Can I adapt U-net architectures for tasks other than image segmentation? $\endgroup$
    – lr99
    Sep 1, 2019 at 14:13
  • 1
    $\begingroup$ @lr99 You're welcome! The U-Net architecture can be used to produce any image. In image segmentation, this output image is suggesting how the image can be segmented, but in general it could be anything. $\endgroup$
    – Mehran Shakerinava
    Sep 1, 2019 at 14:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .