You are right!
1- the number of hidden layers shouldn't be too high! Because of the gradient descent when the number of layers is too large, the gradient effect on the first layers become too small! This is why the Resnet model was introduced.
2- the number of hidden layers shouldn't be too small to extracts good features. It's proved that in CNN networks the first layers extract very simple elements like lines and curves but last layers extracts more complex features.
3- number of hidden units is a hyper-parameters and usually you should find it by testing or based on your background knowledge.
But what can you do at all?
As you can tests different parameters and compare their results there is some other options! One option is grid search you can check this tutorial https://towardsdatascience.com/grid-search-for-model-tuning-3319b259367e