What I want to do is from an Internet challenge to transform any given image into the Polish flag using the available filters and crop tool on the iPhone camera app. Here's an example.
There aren't nearly enough of these videos to train a neural network using a labeled dataset, and (while I haven't ruled it out) I don't think automatically inserting a polish flag into an image then adding random filters to it to create my own dataset would work out.
My thinking is that I would feed a neural network the image and it would output a value for each filter & cropping coordinates. Then, I could easily calculate the loss by comparing the resulting picture to a picture of the polish flag. The obvious problem here is that you don't know how each of the neurons in the last layer affects the loss so you can't perform back propagation.
Is my best bet to mathematically calculate the loss (by this I mean as opposed to using high level libraries, which would be difficult but I'm sure it's possible) so I can find the partial derivative of each last layer neuron with respect to the loss function and then backpropagate? Would this even work? Are there any alternatives that you recommend?