Clarification on GANs for text generation

Question

A GAN-like architecture for text generation is proposed in 'Generative Adversarial Networks for Text Generation'.

The setup is the following:

• The generator of the GAN is proposed to be a recurrent neural network that its by itself a text generator.
• The internal latent vectors of the GAN ( denoted by $G(\vec{z})$ ) are the outputs of the generator RNN.
• The discriminator is not made explicit; but let's imagine that is "any architecture that can classify text into two categories" ("real" and "fake" in our case).
• The generator RNN is trained to minimize $(1- D(G(\vec{z})))$ (where $D(G(\vec{z})))$ is the discriminator output) as usual.

Now it comes a somewhat obscure statement:

Remember that while decoding using an RNN, at every time step we make the choice of the next word by picking the word corresponding to the maximum probability from the output of the softmax function. This “picking” operation is non-differentiable.

Question:

What is the precise statement of that vague paragraph?

My expectation is to explicitly write the function that is just continuous but not differentiable.

Thanks in advance!

Answer 1

When you sample a sentence using an RNN, at each step $t$, if $N$ is the size of your vocabulary, $W$ the array with the $N$ words in your vocabulary, and $f$ your RNN without its final linear layer, you do:

$$p_{t+1} = \text{softmax}(f(w_t))$$ $$i_{t+1} = \text{argmax}(p_{t+1})$$ $$w_{t+1} = W[i_{t+1}]$$ where $p_{t+1}$ is a vector of probabilities of size $N$, and $w_{t+1}$ a word from the vocabulary.

The $\text{argmax}$ operation to get $w_{t+1}$ is not differentiable.

I'm not sure what you mean by continuous but not differentiable. To train the GAN, you're going to have to write a loss function that you'll have to differentiate to do gradient descent on the weights of the GAN. If that loss calculation involves non diffentiable operations, you have a problem.

In practice, in neural network libraries such as JAX or PyTorch, it doesn't matter that the loss is not differentiable as long as it's piece-wise differentiable, which is the case of the $\text{argmax}$ function.