How can I determine the mathematical relation between the input and output variables?

Question

I would like to take in some input values for $n$ variables, say $R$, $B$, and $G$. Let $Y$ denote the response variable of these $n$ inputs (in this example, we have $3$ inputs). Other than these, I would like to use a reference/target value to compare the results.

Now, suppose the relation between the inputs ($R$, $B$ and $G$) with the output $Y$ is (let's say):

$$Y = R + B + G$$

But the system/machine has no knowledge of this relation. It can only read its inputs, $R$, $B$ and $G$, and the output, $Y$. Also, the system is provided with the reference value, say, $\text{REF} = 30$ (suppose).

The aim of the machine is to find this relation between its inputs and output(s). For this, I have come across some quite useful material online like this forum query and Approximation by Superpositions of a Sigmoidal Function by G. Cybenko and felt that it were possible. Also, I doubt that Polynomial Regression may be helpful as suggested Here.

One vague approach that comes to my mind is to use a truth table like approach to somehow deduce the effect of the inputs on the output and hence, get a function for it. But neither am I sure how to proceed with it, nor do I trust its credibility.

Is there any alternative/already existing method to accomplish this?

Answer 1

There are always a large number of possible functions that can produce a given set of input-output values. The challenge is to find a simplest function (according to whatever criteria you choose) to produce those values.

One approach is to write a general function of the input variables, comprising terms of all order in R, G, and B, with a coefficient for each term, then search for values of the coefficients that A) reproduce the known input-output values accurately and B) leave the largest number of coefficients equal to zero.

Several different algorithms can be used to do the search efficiently. My choice would be a genetic algorithm to seek the minimum of the RMS difference between the produced and known I-O values, summed with the product of a gradually increasing parameter and the number of nonzero coefficients.

Answer 2

As I see your explanation, what you're looking for can be stated via another way saying

Y = aX + b

where Y is output vector, X input vector and a & b are the coefficients you want to find.

Why so? And what is happening?

First thing first: I recommend to look the video [4] about how matrices and vectors work together and form after multiplication very familiar equations:

Y = a1 * x1 + a2 * x2 + a3 * x3

Now you see, you do not only get R + G + B but also some constants supplied for each of the variables.

About polynomial solutions I found [2] and [3] but reading through [3] you'll soon notice it is about completely different approach:

Y = a1 * x + a2 * x^2 + a3 * x^3

which you don't want.

So, you find something called linear regression and a so called Deep Neural Network for example to solve it [1].

I would summarize the source [1] in these steps:

• You have to find some training data. That is examples of correct Y values, when X values are known.

• Then you build in Python note book some code that has: a Neural Network with hidden layer(s), Activation Function, Back Propagation and Objective Function.

• Many many iterations with the data, called Training.

There are plenty of samples and courses online about these in detail, but with correct tools and tutorials some dozen lines and no more are needed.

Process is ended with validation phase with some more known data items and results. It can tell you how well the estimated model works.

---

Final Notes:

As you may observe, the solution includes quite a many funny terms you have to learn somehow before mastering the task. For example Udemy has great online courses on this topic, also free tutorials are available on another sites. Your plans sound quite ambitious compared to the knowledge you have so far, so I really do recommend you learn little bit more to be able to fine tune the already given examples online. For example tutorial [5] includes one. It is at first quite complicated code, you'd need quite a lot practice to master it line by line.

---

In short:

Find your favorite tutorial, study neural networks a little bit (basics) and pick code sample to start experimenting. It is a long way but it is worth it.

---

Source:

[1] https://lightsapplications.wordpress.com/linear-regression-and-deep-learning/

[2] https://www.ritchieng.com/machine-learning-polynomial-regression/

[3] https://arachnoid.com/polysolve/

[4] https://www.youtube.com/watch?v=F2lJ7oSwcyY

[5] https://missinglink.ai/guides/neural-network-concepts/backpropagation-neural-networks-process-examples-code-minus-math/

Answer 3

I found a solution to this some time back. I have been studying function approximation (within linear regression) for some time. Here's how I did it:

Neural Networks have been proved to be universal function approximators. So, even a single hidden layer would be sufficient to approximate a function simple as addition (Even somewhat complex functions like the Sine and any random CONTINUOUS wiggly function have been approximated)

First, I used a high level API like TensorFlow and Keras and implemented it here

The model was trained on the data (input-output pairs)

R    = np.array([-4, -10,  -2,  8, 5, 22,  3],  dtype=float)
B    = np.array([4, -10,  0,  0, 15, 5,  1],  dtype=float)
G    = np.array([0, 10,  5,  8, 1, 2,  38],  dtype=float)

Y    = np.array([0, -10, 3, 16, 21, 29, 42],  dtype=float)

And trained as follows:

#Create a hidden layer with 2 neurons
hidden = tf.keras.layers.Dense(units=2, input_shape=[3])

#Create the output (final) layer which symbolises value of **Y**
output = tf.keras.layers.Dense(units=1)

#Combine layers to form the neural network and compile it
model = tf.keras.Sequential([hidden, output])
model.compile(loss='mean_squared_error', optimizer=tf.keras.optimizers.Adam(0.1))
history = model.fit(RBG,Y, epochs=500, verbose=False)

The model converges in about 50 epochs

Also, I have implemented the same using only C/C++ and used GNU plot to visualize the results.