2
$\begingroup$

My question is regarding standard dense-connected feed forward neural networks with sigmoidal activation.

I am studying Bayesian Optimization for hyper-parameter selection for neural networks. There is no doubt that this is an effective method, but I just wan't to delve a little deeper into the maths.

Question: Are neural networks Lipschitz functions?

$\endgroup$
2
$\begingroup$

I'm not an expert in this area, but it would appear to depend on the choice of activation function:

That said, this paper appears to give some conditions (specifically for dynamic ANNs) for which networks with activation function involving $e^x$ can be Lipschitz continuous, so possibly the above is not the whole story.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.