Finding the optimal combination of inputs which return maximal output

Question

I am currently working on a problem and now got stuck to implement one of it's steps. This is a simple attempt to explain what I am currently facing, which is something that I am aiming to implement in my regression simulation in python.

Let's say that I fit a non-linear model to my data. Now, I want to find the combination of inputs within a specified range that returns the the highest outcome. When I am using a quadratic function or only a few inputs, this task is quite simple. However, the problem comes when trying to apply the same logic for more complex models. Supposing that I have 9 variables as inputs, I will have to test all possible combinations and that would be computationally unfeasible by doing it with meshgrid if you want to cover a range with a several intervals in between.

So, here it comes my question, is there such a way to avoid having to go through this computationally costly process in order to achieve the combinations of inputs defined in a given range that return the optimal output?

Answer 1

If your model is gradient-based, such as a neural network, then may also be able to use gradient methods to drive virtual inputs:

• Freeze all network weights to the trained version

• Define a loss function that decribes how you want the output - or any internal measure - to behave. E.g. to maximise the output, the loss function can simply be the negative of the output, assuming you will perform gradient descent later. Some libraries will also support gradient ascent to maximise a function.

• Define your input as a variable that can be optimised and instantiate an optimiser (details will vary depending on your library)

• Start with a random or best-guess input, and iterate normal training routine (feed forward then backpropagate) to get better and better inputs

This is basically how Deep Dream and Style Transfer algorithms work - the detail that is different is definition of the loss functions. It is also a way to make adversarial attacks against known models, for example taking a picture of a car, modifying it such that a classifier returns that it is an ostrich (whilst it still looks like a car to a human).

It is not guaranteed to find the absolute best inputs for a given range, but should find good approximations of local minima or maxima far faster than a meshgrid when there are many dimensions. You could combine the idea with a coarse meshgrid or random search for start points for a better chance of finding the best results within input range constraints.

You should bear in mind that the ideal inputs you discover will only be as accurate as your trained model will let them be. If you discover maximising inputs in the model, that use inputs that are far away from any training examples, then in reality those inputs might not get anything like the result that the model predicts. Statistical models with many degrees of freedom are typically ok at interpolating between data points, but very bad at extrapolating beyond data that has been observed.