# Using parameter estimation for training a neural network

Assume that we have 4 layers in a neural network.

$$z_1 = L_1(x, W_1)$$ $$z_2 = L_2(z_1, W_2)$$ $$z_3 = L_3(z_2, W_3)$$ $$y = L_1(z_3, W_4)$$

Where $$x$$ is the vector input, $$y$$ is the vector output and $$W_i, i = 1..4$$ is the weight matrix.

Assume that I could estimate parameters in a function.

$$b = f(a, w)$$

Where the $$b$$ is a real value and $$a$$ is the input vector and $$w$$ is the weight vector parameter. The function $$f$$ could be like this.

$$b = \text{activation}(a_1*w_1 + a_2*w_2 + a_3*w_3 + \dots + a_n*w_n)$$

Here we can interpret $$b$$ as the neuron output. Estimate $$w_n$$ is very easy if we know $$b$$ and $$a_n$$. This can be done used by recursive least squares or a Kalman filter.

Question:

If every neuron in a neural network is a function that has inputs and weights, can I use parameter estimation for estimating all weights in a neural network if I did parameter estimation for every neuron inside a neural network?

I found a paper where they are using a Unscented Kalman Filter for parameter estimation.

Function $$D_{k|k-1} = G[x_k, W_{k|k-1}]$$ can be interpreted as a neuron function where $$W_{k|k-1}$$ is a matrix with different types of weights and $$D_{k|k-1}$$ is different types of outputs from that neuron. No, it's not a "multivariable output"-neuron. It's just the way how to estimate the best weights by using different weights.

The error of the neuron output is: $$d_k - \hat d_k$$ in equation (41). So when the error is small, that means the output of the neuron is OK and that means the real weights $$\hat w_k$$ has been found.

• I don't understand what you mean exactly by "parameter estimation" when you ask the question "can I use parameter estimation for estimating all weights in a neural network...?". Can you clarify that? It's not very clear the relationship between your idea of "estimating parameters" and the paper that you found.
– nbro
Sep 19 at 18:48
• @nbro I assume that parameter estimation is a tool for estimating unknown parameters inside an equation, by using data. Sep 20 at 6:08
• So, would back-propagation (which is usually used to train neural networks) be a "parameter estimation" tool/technique or you have something else in mind?
– nbro
Sep 20 at 12:53
• @nbro No. Back-propagation is a optimization method. This method above is estimation. Back propagation you computing the weights by setting the outputs first. In this method above, we don't know the outputs. We only set the inputs and compute the weights just by looking how large the error is between the real data and estimated data. Sep 20 at 13:28
• But back-propagation computes the gradient based on the error, which seems that it's what you're suggesting.
– nbro
Sep 20 at 13:30