Autoencoders: Where does the encoder end and the decoder begin?

Question

Consider a simple Autoencoder neural net:

from torch import nn

class AE(nn.Module):
    def __init__(self, x_dim, z_dim, h_dim=42):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(dim_in, dim_h),
            nn.ReLU(),
            nn.Linear(dim_h, z_dim)
        )
        self.decoder = nn.Sequential(
            nn.Linear(z_dim, h_dim),
            nn.ReLU(),
            nn.Linear(h_dim, x_dim),
        )

    def forward(self, x):
        z = self.encoder(x)
        x = self.decoder(z)
        return x

In popular literature, it is generally implied that the output of AE.encoder is solely responsible for encoding whereas AE.decoder is solely responsible for the decoding. Why is that?

If we consider that encoding is a more complex task than decoding, there is no actual guarantee that the network won't use the first three layers for encoding and only the last for decoding (or vice versa). This might especially be the case if we consider asymmetrical autoencoder architectures.

Answer 1

I think that, in the case of a (deterministic) auto-encoder that only compresses the data, which is trained with MSE, it's an assumption or convention that the encoder is just that part of the neural network that compresses the data into a more compact representation/vector.

However, I understand your concern, because, if you train the encoder and decoder jointly, one may think that the decoder is also responsible for encoding, which I assume you define as the transformation of the input into something else, before reconstructing it. I don't know to what extent one may entertain this idea, but, honestly, I would just stick to the convention of considering the encoder the part that compresses the data because, in my mind at least, encoding usually means to compress, and that's what the encoder does. If you define encoding as any transformation, then any neural network, or any layer, or, even more generally, any function, can be considered an encoder, but it's not what people (in ML) usually refer to when they mention an encoder.

In the case of the VAE, the situation is even clearer, I would say, because the VAE was formulated to model a specific graphical model, where $x$, the (original/reconstructed) input depends on a latent variable $z$, although, even in this case, the encoder and decoder are trained jointly with the ELBO, which contains 2 terms, one for the encoder and the other for the decoder. It's true that the optimization of this objective takes into account both terms, but, again, the convention is to consider the encoder the part that produces the latent vector.