To answer the question, I want to go a small sidestep to the first neural network ever invented, the McCulloch-Pitts Neurons from 1943. From today's perspective it may be surprising, but these primitive neurons are Turing complete, that means they can compute any algorithm like a “Neural Turing Machine (NTM)”. The reason is, because the McCulloch-Pitts neurons are implementations of logicgates (and, or, not) which can be connected to any function, including a counter, a for loop and a prime-number-checker.
Neural networks have no limitations from a computing perspective, because today's neural networks are more powerful than the former McCulloch Pitts neurons. The limitation is only the gradient descent algorithm for finding the weights. The learning algorithm is not very efficient and there is no better alternative available. In the error map, the problem must be reduced to zero-error, that means the neural network outputs the right value. Gradient descent is a heuristic to follow a certain path in the maze in the hope, that the door out is at the end. In many cases, gradient descent finds only an impasse. Perhaps an example:
A neural network should print out, if the input number is a prime number or not. With gradient descent the error rate can reduced to the value “5.4”. In theory, the neural network is able to find the prime-number algorithm, that means to reduce the error rate down to 0.0. But the gradient descent algorithm doesn't know how to reduce the error-rate from “5.4” to “5.3” and further. He is iin a local minimum. The reason is, that the error map is under the fog-of-war, to uncover the information CPU-intensive trial&error search has to be done.
Bypassing the problem is easy: The programmer has to implement the problem-solving algorithm by hand in a normal programming language, and use a neural network only for finding a parameter in his algorithm.