Given two neural networks that compute two functions $f(x)$ and $g(x)$, how can I create a neural network that computes $f(x)g(x)$?

Question

I have two functions $f(x)$ and $g(x)$, and each of them can be computed with a neural network $\phi_f$ and $\phi_g$.

My question is, how can I write a neural net for $f(x)g(x)$?

So, for example, if $g(x)$ is constant and equal to $c$ and $\phi_f = ((A_1,b_1),...(A_L,b_L))$, then $\phi_{fg} = ((A_1,b_1),...,(cA_L,cb_L))$.

Actually, I need to show it for $f(x)=x$ and $g(x)=x^2$ if this make something easier.

Answer 1

Use another network h which takes f(x) and g(x) as input i.e. h(f(x), g(x)).

Training psuedo code(pytorch):

for epoch in epochs:
    for batch, x in dataset:
        train(f)
        train(g)


for epoch in epochs:
    for batch, x in dataset:
        # freeze f and g (using torch.no_grad in pytorch)
        with torch.no_grad:
            fx = f(x)
            gx = g(x)
        pred = h(fx, gx)
        loss = loss_fn(pred, (fx . gx))
        backpropagate()
        optimise()
        

Answer 2

You need to express the product of the activation functions of your neurons via some combination of individual activation functions.

For example, if the product of your activation functions is equal to the linear combination of such functions then the network for $f(x)g(x)$ can be derived from the networks for $f(x)$ and $g(x)$ exactly.

Let's assume for simplicity that $f(x)$ and $g(x)$ are approximated with trained networks that consist of just one neuron, $$f(x) = A(w_f x + b_f)$$ $$g(x) = A(w_g x + b_g)$$ Then, if the activation function $A$ is such that $$A(w_f x + b_f)A(w_g x + b_g) = \sum\limits_{i=1}^N \alpha_iA(w_i x + b_i)$$ then $f(x)g(x)$ is approximated by a neural network consisting of $N$ neurons: $$f(x)g(x) = \sum\limits_{i=1}^N \alpha_iA(w_i x + b_i)$$ This is easily generalized to a network that consists of multiple neurons.

An example is when some of your neurons have activation functions $A(z)=\cos(z)$ and others are $A(z)=\sin(z)$. Then you can build the network computing $f(x)g(x)$ from the networks computing $f(x)$ and $g(x)$ using the product-to-sum identities, like $2\sin(\theta)\sin(\phi) = cos(\theta-\phi)-cos(\theta+\phi)$, etc.

If the product of your activation functions cannot be expressed as a linear combination of such functions (as in the case of sigmoids) then the network for $f(x)g(x)$ can be derived from the networks for $f(x)$ and $g(x)$ approximately. You need to figure out the parameters of the approximation $$A(z_f)A(z_g) \approx \sum\limits_{i=1}^N \alpha_iA(z_i)$$ Such parameters are: the number of terms, $N$, the coefficients $\alpha_i$, and the range $[z_{min}, z_{max}]$ where this approximation works with a desired accuracy.

Linear combination is not the only opportunity. Say, your activation is $A(z)=a^z$. Then, $$A(w_f x + b_f)A(w_g x + b_g)=a^{(w_f+w_g) x + (b_f+b_g)}=a^{w_{fg} x + b_{fg}}$$ so the weights and the biases of the product are exactly expressed via the weights and the biases of the multiplicands.