1
$\begingroup$

Can someone please point me to where I can read up on why non linearities that can produce values larger than 1 or smaller than 0 work. My understanding is that neurons can only produce values between 0 and 1 and that this assumption can be used in things like cross entropy. Are my assumptions just completely wrong?

$\endgroup$
2
$\begingroup$

Why wouldn't they work?

Each neuron's output is equal to a function over the sum of all its weights multiplied by their corresponding neurons. If that function is the Sigmoid function, then the output is squashed from [0,1]. If the entire layer uses a SoftMax function, then the output of all neurons is squashed from [0,1] and their sum equals 1. In other others, they represent a set of probabilities, where you can then use cross-entropy to optimize their values (cross-entropy measures the difference between two probability distributions).

RelU and ElU are simply other types of functions, whose output is not bounded from [0,1]. They are differentiable, as all other functions, and can be used in any network.

$\endgroup$
2
$\begingroup$

Christopher Olah's blog post describes it better that I ever could. Basically, most data we come across can't be separated with a single line, but with some kind of curve. Non-linearities allow us to distort the input space in ways that make the data linearly separable, making classification more accuarate.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.