I know this is a subjective question, but I was thinking how does one decide on their encoder architecture in the case of Bi-directional GANs.
The first idea coming to my mind is basically mirroring the generator's architecture, which can end up being something very similar to the discriminator architecture.
Nonetheless, mirroring the generator's architecture means that we will add yet another complex network in our GAN framework.
GANs are known for unstable training, so is adding another complex network something we should avoid in Bi-GANs? What is the common practice in the literature? Should we stick to a network with less parameters when it comes to the encoder?