What is the best activation function for the embedding layer in a deep auto-encoder?

Question

I am designing a deep autoencoder for graph embedding (exactly node embedding) following this paper SDNE. In the original paper, they used the sigmoid activation for all hidden layers in the autoencoder model, even for the embedding layer.

However, I think the embedding layer should use the tanh activation and the reconstruction layer should be used ReLU activation. Because, embedding is in the range $[-1, 1]$ and reconstruction layer is in the range $[0, x]$, which generates better results due to a larger range for representation and directed graph. Instead of in the range $[0,1]$ from sigmoid will lead to a lack of embedding information.

So, what is the best activation function for deep autoencoders to capture good information about the structure of graph?

Answer 1

It is hard to tell what exactly is better because these are hyperparameters. However, the sigmoid activation function is closer to biological neurons.

In the paper below, Bengio demonstrates why ReLU activation functions are better for hidden layers. In summary, they increase the sparsity of calculations (matrix in each layer shod multiply to its relative weights), and because of that, it made it possible for data to classify faster and easier.

http://proceedings.mlr.press/v15/glorot11a/glorot11a.pdf