2
$\begingroup$

If a neural network has a limited number of neuron parameters to find, -let's say only 1000 parameters-, it is generally better to spend the parameters on weights or neuron bias?

For example, if each neuron has 2 weights and one bias, it uses 3 parameters per neuron, so only 333 neurons would be available.

But if each neuron uses no bias parameter, then 500 neurons are available with 1000 parameters.

I'm concerned with overfiting by using too many parameters, so I want to minimize the number of parameters meanwhile maximizing the quality of the result.

Improve this question
$\endgroup$
3
  • $\begingroup$ Are you just interested in 1. having or not biases or 2. in general, how many connections a neuron has? From your title, it seems like you're trying to understand whether you should or not have biases. However, from the body of your post, it seems like you're also interested in how you should define all the connections (including biases) between neurons (e.g. if a neural network should be sparse or not). $\endgroup$
    – nbro ♦
    Jun 3 '20 at 16:43
  • $\begingroup$ @nbro I'm asking about the choice between (more neurons) or (less neurons with bias parameters). I guess that more neurons require more computing, and more cycles for convergence, but more neurons can achieve higher accuracy. If the constrain is the number of parameters, then I guess that more neurons should be preferable. $\endgroup$
    – tutizeri
    Jun 3 '20 at 17:46
  • $\begingroup$ The capacity of a network is proportional to the number of parameters and not necessarily the number of neurons, so I think you're making a possible wrong assumption here. I think your question is more related to the sparsity of the parameters (i.e. how many connections you have and how they affect learning). $\endgroup$
    – nbro ♦
    Jun 3 '20 at 18:00
1
$\begingroup$

First of all, your estimates are a bit off. If you have 300 neurons, you won't have just 2 weights per neuron, but much more, assuming full connectivity

Bias isn't just an extra parameter to fit, it is an important adjustable parameter that sets the offset of the separating hyperplane represented by each neuron. Think of the simple equation $ax+b$, there's no way to shift the line unless you use the $b$ (bias) part.

This would be especially important for small number of nodes and classification tasks (think perceptrons etc)

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for your answer. I said 300 neurons with 2 weights just as an illustrative example. Do you mean that the bias parameter cannot be replaced by using more neurons without bias? Or just that it is more efficient to use bias than adding neurons? $\endgroup$
    – tutizeri
    Jun 4 '20 at 18:02
  • $\begingroup$ @tutizeri the bias parameter cannot be replaced by using more neurons without bias, think of it as fitting a line to separate some points, you can rotate the line with the slope (weights) but it still passes through the origin, if your data points are not distributed such that they can be separated by this line, you will need to move the line. This is for one neuron. So, the bias is an important parameter without which your weights can be useless. $\endgroup$
    – SajanGohil
    Jun 5 '20 at 11:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.