I have two functions $f(x)$ and $g(x)$, and each of them can be computed with a neural network $\phi_f$ and $\phi_g$.

My question is, how can I write a neural net for $f(x)g(x)$?

So, for example, if $g(x)$ is constant and equal to $c$ and $\phi_f = ((A_1,b_1),...(A_L,b_L))$, then $\phi_{fg} = ((A_1,b_1),...,(cA_L,cb_L))$.

Actually, I need to show it for $f(x)=x$ and $g(x)=x^2$ if this make something easier.

  • $\begingroup$ Did you mean $\phi_{fg} = ((cA_1,cb_1),...,(cA_L,cb_L))$ rather than $\phi_{fg} = ((A_1,b_1),...,(cA_L,cb_L))$? $\endgroup$
    – nbro
    Aug 17 '20 at 1:01
  • $\begingroup$ no because only the output should be multiplied with c $\endgroup$ Aug 17 '20 at 8:12
  • $\begingroup$ In the title you write $f(x) * g(x)$ and in the body $f(x)g(x)$. Is that a convolution or multiplication? If $g(x)$ is constant then you would have $(cA_L, cb_L)$ only if the activation function of the final layer is linear so it's not true in general. $\endgroup$
    – Brale
    Aug 17 '20 at 18:37
  • $\begingroup$ @Brale Sorry, it was me who rewrote the title. It's multiplication in all cases. $\endgroup$
    – nbro
    Aug 17 '20 at 23:46

Use another network h which takes f(x) and g(x) as input i.e. h(f(x), g(x)).

Training psuedo code(pytorch):

for epoch in epochs:
    for batch, x in dataset:

for epoch in epochs:
    for batch, x in dataset:
        # freeze f and g (using torch.no_grad in pytorch)
        with torch.no_grad:
            fx = f(x)
            gx = g(x)
        pred = h(fx, gx)
        loss = loss_fn(pred, (fx . gx))
  • $\begingroup$ thank you for your answer but i dont want to know how to implement it but how to write it theoretically as a neural net $\endgroup$ Aug 16 '20 at 16:33

You need to express the product of the activation functions of your neurons via some combination of individual activation functions.

For example, if the product of your activation functions is equal to the linear combination of such functions then the network for $f(x)g(x)$ can be derived from the networks for $f(x)$ and $g(x)$ exactly.

Let's assume for simplicity that $f(x)$ and $g(x)$ are approximated with trained networks that consist of just one neuron, $$f(x) = A(w_f x + b_f)$$ $$g(x) = A(w_g x + b_g)$$ Then, if the activation function $A$ is such that $$A(w_f x + b_f)A(w_g x + b_g) = \sum\limits_{i=1}^N \alpha_iA(w_i x + b_i)$$ then $f(x)g(x)$ is approximated by a neural network consisting of $N$ neurons: $$f(x)g(x) = \sum\limits_{i=1}^N \alpha_iA(w_i x + b_i)$$ This is easily generalized to a network that consists of multiple neurons.

An example is when some of your neurons have activation functions $A(z)=\cos(z)$ and others are $A(z)=\sin(z)$. Then you can build the network computing $f(x)g(x)$ from the networks computing $f(x)$ and $g(x)$ using the product-to-sum identities, like $2\sin(\theta)\sin(\phi) = cos(\theta-\phi)-cos(\theta+\phi)$, etc.

If the product of your activation functions cannot be expressed as a linear combination of such functions (as in the case of sigmoids) then the network for $f(x)g(x)$ can be derived from the networks for $f(x)$ and $g(x)$ approximately. You need to figure out the parameters of the approximation $$A(z_f)A(z_g) \approx \sum\limits_{i=1}^N \alpha_iA(z_i)$$ Such parameters are: the number of terms, $N$, the coefficients $\alpha_i$, and the range $[z_{min}, z_{max}]$ where this approximation works with a desired accuracy.

Linear combination is not the only opportunity. Say, your activation is $A(z)=a^z$. Then, $$A(w_f x + b_f)A(w_g x + b_g)=a^{(w_f+w_g) x + (b_f+b_g)}=a^{w_{fg} x + b_{fg}}$$ so the weights and the biases of the product are exactly expressed via the weights and the biases of the multiplicands.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.