# Is this ML task possible?

What I want to do is from an Internet challenge to transform any given image into the Polish flag using the available filters and crop tool on the iPhone camera app. Here's an example.

There aren't nearly enough of these videos to train a neural network using a labeled dataset, and (while I haven't ruled it out) I don't think automatically inserting a polish flag into an image then adding random filters to it to create my own dataset would work out.

My thinking is that I would feed a neural network the image and it would output a value for each filter & cropping coordinates. Then, I could easily calculate the loss by comparing the resulting picture to a picture of the polish flag. The obvious problem here is that you don't know how each of the neurons in the last layer affects the loss so you can't perform back propagation.

Is my best bet to mathematically calculate the loss (by this I mean as opposed to using high level libraries, which would be difficult but I'm sure it's possible) so I can find the partial derivative of each last layer neuron with respect to the loss function and then backpropagate? Would this even work? Are there any alternatives that you recommend?

I think the best thing to use here is a form of "structured prediction". Our "target" is a sequence of operations. The framework of structured prediction allows us to chain together as many filters as we want.

With a neural network of fixed architecture, you would have to make sure you have enough space for all the filters you might need.

• Good idea! I was thinking of something like this but I wasn't sure if it would be practical. Thanks for letting me know what it's called and that it's a viable option. I realized that the order you apply the filters matters: for example, if you use a gradient filter then rotate the image 90 degrees the gradient will be perpendicular to the gradient when you rotate the image first. For this reason, would it be a wise option to train a neural network to make one change at a time (whichever change will get it closer to the end result) and then propagate through it multiple times? Sep 24 '20 at 22:28
• My only concern is that it wouldn't be able to do any long-term planning and I don't know how important that is here. Sep 24 '20 at 22:29