Can someone please point me to where I can read up on why non linearities that can produce values larger than 1 or smaller than 0 work. My understanding is that neurons can only produce values between 0 and 1 and that this assumption can be used in things like cross entropy. Are my assumptions just completely wrong?
Why wouldn't they work?
Each neuron's output is equal to a function over the sum of all its weights multiplied by their corresponding neurons. If that function is the Sigmoid function, then the output is squashed from [0,1]. If the entire layer uses a SoftMax function, then the output of all neurons is squashed from [0,1] and their sum equals 1. In other others, they represent a set of probabilities, where you can then use cross-entropy to optimize their values (cross-entropy measures the difference between two probability distributions).
Christopher Olah's blog post describes it better that I ever could. Basically, most data we come across can't be separated with a single line, but with some kind of curve. Non-linearities allow us to distort the input space in ways that make the data linearly separable, making classification more accuarate.