If I have a lot of input output pairs as training data

<float Xi, float Yi>

and I have a parametrized approximation function (I know the function algorithm, but not the values of the many many parameters it contains) which shall approximate the process by which the original data pairs were generated. The function takes two input values:

// c is a precomputed classifier for x and can have values from 0 to 255, so there can be up to 256 different classes
y = f(float x, int c)

the hidden parameters of the function are some big lookup tables (a lot of free parameters, but still much fewer than the amount of data points in the training data)

Now I want to fit all the hidden parameters that f contains AND compute for each Xi a ci, such that for the fitted function the error over all i of Yi - f(Xi, ci) is minimized

So, using some algorithm I want to fit the parameters of f and also classify the inputs Xi so that f(Xi, ci) aproximates Yi

How is this kind of problem called and what kind of algorithm is used to solve it?

I assume it's possible to initialize all hidden parameters as well as all ci with random values and then somehow use back propagation of the error to iteratively find parameters and ci such that the function works well.

What I don't know is whether this is a well known class of problem and I just don't know the name of it, so I'm asking for pointers.

Or maybe in other words: I have a function that has a certain layout (for performance reasons) which I want to use to approximate and interpolate my training data, I want to tune the parameters of this function such that it approximates the original data well. since the data points fall into some 'categories', I want to pre-classify the data-points to make it easier for the function to do its job. What kind of algorithm do I use to find the function's parameters and to pre-classify the input?


This problem is typically called parameter estimation or inverse modelling, and there are a variety of techniques to solve it.

If your free parameters are all continuous (i.e. none are discrete, such as integers), and your model function is differentiable, then you can turn the model into a computation graph in e.g. TensorFlow and use gradient descent methods with MSE loss, learning parameters on the data in much the same way as a neural networks. Most of the toolkits built for neural networks can do this, you just have to ignore the pre-packaged layer models and the utilities to manage them. You will need to take care with parameter initialisation, ideally you already have some rough starting values.

You can use other approaches, such as search methods too. Even when you have some discrete-valued parameters then you can use genetic algorithms for instance.

The best method to use will depend on details of your model, it is not possible to point out a generic "best". If you can use a gradient-based method directly, as you suggest in the question, then that might be the most efficient in terms of computation, provided you have some way to set reasonable initial parameters.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.