I have an idea to find the optimal number of hidden neurons required in a neural network, but I'm not sure how accurate it is.
Assuming that it has only 1 hidden layer, it is a classification problem with 1 output node (so it's a binary classification task), has N input nodes for N features in the data set, and every node is connected to every node in the next layer.
I'm thinking that to ensure that the network is able to extract all of the useful relations between the data, then every piece of data must be linked to every other piece of data, like in a complete graph. So, if you have 6 inputs, there must therefore be 15 edges to make it complete. Any more and it will be recomputing previously computed information and any less will be not computing every possible relation.
So, if a network has 6 input nodes, 1 hidden node, 1 output node. There will be 6 + 1 connections. With 6 input nodes, 2 hidden nodes, and 1 output node, there will be 12 + 2 connections. With 3 hidden nodes there will be 21 connections. Therefore, the hidden layer should have 3 hidden nodes to ensure all possibilities are covered.
This answer discusses another method. For the sake of argument, I've tried to keep both examples using the same data. If this idea is computed with 6 input features, 1 output node, $\alpha = 2$, and 60 samples in the training set, this would result in a maximum of 4 hidden neurons. As 60 samples is very small, increasing this to 600 would result in a maximum of 42 hidden neurons.
Based on my idea, I think there should be at most 3 hidden nodes and I can't imagine anymore being useful, but would there be any reason to go beyond 3 and up to 42, like in the second example?