For Keras on TensorFlow, a layer class constructor comes with these:

  • kernel_regularizer=...
  • bias_regularizer=...
  • activity_regularizer=...

For example, Dense layer:

The first one, kernel_regularizer is easy to understand, it regularises weights, makes weights smaller to avoid overfitting on training data only.

Is kernel_regularizer enough? When should I use bias_regularizer and activity_regularizer too?

  • 1
    $\begingroup$ See this answer. $\endgroup$
    – nbro
    Jul 18 '20 at 14:22

Regularizer's are used as a means to combat over fitting.They essentially create a cost function penalty which tries to prevent quantities from becoming to large. I have primarily used kernel regularizers. First I try to control over fitting using dropout layers. If that does not do the job or leads to poor training accuracy I try the Kernel regularizer. I usually stop at that point. I think activity regularization would be my next option to prevent outputs from becoming to large. I suspect weight regularization effectively can pretty much achieve the same result.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.