The universal approximation theorem states, that a feed-forward network with a single hidden layer containing a finite number of neurons can approximate continuous functions on compact subsets of $R^n$.
No matter what the function, there is guaranteed to be a neural network so that for every possible input, $x$, the value $f(x)$ (or some close approximation) is output from the network.
So, for continuous functions, this seems plausible. Interestingly, in the same article, Nielsen mentioned "any function".
Later, he writes
However, even if the function we'd really like to compute is discontinuous, it's often the case that a continuous approximation is good enough.
The last statement leaves open a gap, to ask how well an approximation practically can be.
Let's ignore contradictory input/output training pairs like $f(0)=0$ and $f(0)=1$, which actually don't event represent a function anyway.
Furthermore, assume that the training data is generated randomly, which would practically result in a discontinuous function.
How does a neural network learn such data? Will a learning algorithm always be able to find a neural network that approximates the function represented by the input-output pairs?