Which techniques can achieve neural doodle in real-time?

Question

Ideally I'd like to watch movie which is deep dreamed in real-time. Most algorithms which I know are too slow or not designed for real-time processing.

For example I'm bored with some movie which I've watched thousands of time and I'd like to add some "dreaming" to it which is real-time filter which takes input frames, then it's processing and enhances the images through artificial neural network to achieve doodled output.

Doesn't have to be exactly DeepDream or hallucinogenic technique (which could be too much to watch for 2h), but with any similar ANN algorithm. I'm more interested into achieving desired real-time use.

What kind of techniques can achieve such efficiency?

Answer 1

Most of the algorithms (based on image synthesis and style transfer, e.g. neural-doodle) haven't been proven to be highly effective in terms of real-time image processing.

However the following studies discusses such algorithms for real-time texture synthesis:

• Feed-forward Synthesis of Textures and Stylized Images

  The approach is to move the computational burden to a learning stage, making trained network (CNN) light-weight and compact in order to generate multiple samples of the same texture. This can generate textures as good as comparable to Gatys~et~al, but significantly faster.

• Perceptual Losses for Real-Time Style Transfer and Super-Resolution

  This method uses parallel work which can generate high-quality images by defining and optimizing loss functions based on high-level features extracted from pretrained networks.

• Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks

  This uses precomputed feed-forward networks that captures the feature statistics of Markovian patches in order to generate outputs of arbitrary dimensions. This can be applied to texture synthesis, style transfer and video stylization.

^{Source: Above list suggested on neural-doodle project.}