Let's say I have two databases, $(\mathbf{x_i}, \mathbf{\hat{p_i}})$ and $(\mathbf{x_j}, \mathbf{\hat{q_j}})$. A neural network with weights $\theta$ can receive an input $\mathbf{x}$ and produce an output $\mathbf{y}$. Mathematically, $\mathbf{y} = f_{NN}(\mathbf{x},\theta)$. To compare the output of my neural network and the database, I need two wrappers, $\mathbf{p}=g(\mathbf{y})$ and $\mathbf{q}=h(\mathbf{y})$.

The problem is: only $g(\cdot)$ is differentiable while writing $h(\cdot)$ in a differentiable manner would take a huge effort.

Is there any efficient way to train my neural network to minimize the following loss function? $$ \mathcal{L}(\theta) = \sum_i \left\{g\left[f_{NN}(\mathbf{x_i}, \theta)\right] - \mathbf{\hat{p_i}}\right\}^2 + \sum_j \left\{h\left[f_{NN}(\mathbf{x_j}, \theta)\right] - \mathbf{\hat{q_j}}\right\}^2 $$

My thinking

If I am using gradient descent-type algorithm, I can only optimize the first part of the loss function while ignoring the second part. If I am using evolutionary-type algorithm, I can optimize both parts, but it will take a long time and I don't make a full use of the differentiable property of $g(\cdot)$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.