TL;DR:One does not know ahead of time what hyper-parameters will achieve optimal performance. So what you need is an iterative implementation strategy:
When working with neural networks it is key to make sure that you spend your time wisely. It is possible to spend lots of time on a dead end simply because you made an assumption about your model at the very beginning.
So when selecting activation functions and other hyper parameters don't over think things. That is, get a quick and dirty model up and running and tune from there. From this model you can iterate. For example, you could start with ReLU activations in the hidden layers and as you tune your model you could experiment with other other activations.
That is, the data and the task at hand along with your tuning shape your model. A highly recommended video on this is A. Ng's lecture here and this video from the A. Ng deep learning specialization.
Some content not in the video is how to use learning curves to help define your iterations. These help you decide what you should do next when your model is not achieving desired performance.