1
$\begingroup$

I've been playing around with neural style transfer for a about a year now, and I've been doing it with two general approaches. The first has been using a script that is available on the Keras GitHub, which produced this result when I applied it to a picture I took of an oregon grape as content and the The Great Wave off Kanagawa painting as the style. If memory serves, this image had content, style, and total variation weights equal to 1.0 and ran it for 100 iterations. https://scontent.fyvr1-1.fna.fbcdn.net/v/t1.0-9/81825142_220261185641035_2692609611696963584_o.jpg?_nc_cat=102&_nc_sid=8024bb&_nc_ohc=jcGx4Nv74cYAX9CsRTz&_nc_ht=scontent.fyvr1-1.fna&oh=c13abcd965911e0c858874d7b9173a52&oe=5EBA0371

The second approach I've been using is a free service called Deep Dream Generator (not to be confused with Google's DeepDream), like the output below where the content was an L-system fractal I made (a story for another time) and the style was one of the many styles shown in the 'popular style' section of the service. Note that this service offers 'Deep Style', 'Thin Style', and 'Deep Dream'. I am interesting in and currently referring to 'Deep Style'. https://q4j2g5j9.stackpathcdn.com/ddg-dream/c17830af0d8eead6fc6a727a2d5f8ba39aa4074f.jpg

At first it was just my isolated opinion that the quality of output from the Deep Dream Generator was better, but many people have confirmed that they share this opinion. The claim that one service is better than the other when it comes to making art is certainly subjective, but the claim that observational data supports a general preference is independently testable and therefore somewhat objective.

The script mentioned above allows you to control the hyperparaters that weight the content loss, style loss, and total variation loss. I sweeped values from 0 to 2 stepping by 1/10th of the range for each of the parameters giving 1000 combinations, which I turned into multiple panels of images where I saw that the output mostly looked the same! Below is an example panel.

https://scontent.fyvr1-1.fna.fbcdn.net/v/t1.0-9/82413905_219310189069468_4944756059307769856_n.jpg?_nc_cat=108&_nc_sid=8024bb&_nc_ohc=XcVjqhZUgycAX-yAnDg&_nc_ht=scontent.fyvr1-1.fna&oh=653dae375ffbd8642e9939d0be9cd8a4&oe=5EBAFEA2

Looking for what could be different, I noted that the Deep Style tool has some additional hyperparameters that the Keras setup does not. Here's my consideration of what seems similar or different.

Firstly, they have an 'Enhance' setting which is described as How much strength to apply when enhancing the image. Higher value will produce higher quality output but will diminish some of the original texture and structure. I'm doubtful this is equivalent to the Keras model's hyperparameters, and I'm not sure what they're doing here.

Secondly is 'Resolution'which is described as Max resolution in MP (Mega Pixels). I can easily change the size of images I'm training with, at least within what my computer system resources can handle.

Thirdly is 'Depth' described as The more depth you set the better quality you will get but it will take more time to generate the image. I suspect they have different style transfer models where different selections of layers can be used, with 'deeper' being the inclusion of more layers.

Fourthly, is 'Style Weight' described as Default is 50%. Less means more like the original image and more means more like the style image. I strongly suspect that this is the same style weight hyperparater in the aforementioned Keras model, and therefore I can readily control this.

Fifthly is 'Style Scale' described as Scale style image. Makes extracted patterns from the style image bigger/smaller. I'm not really sure if this represents the total varation hyperparater in the loss function or not. The Keras script describes the 'total variation' as follows: The total variation loss imposes local spatial continuity between the pixels of the combination image, giving it visual coherence. For all I know, the total variation could be the 'Enhance' mentioned above.

The sixth is 'Preserve Colors', which doesn't have a description. As in the example above, I usually have this turned off when I use the online service, but I definitely don't have an analog for the Keras setup I'm using. I've wondered if this is training on a black and white version first to change the structures slightly, and then do a second colorizing step... But I really don't know with any confidence.

What are the actual math or computer science concepts behind these unfamiliar hyperparameters?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.