Could someone clear my doubt on the loss function used in SeqGAN paper . The paper uses policy gradient method to train the generator which is a recurrent neural network here.
- Have I interpreted the terms correctly?
- What are we summing over? The entire vocabulary of words?
Suppose, total time steps = 4, vocabulary size = 3, rollout size = 2. so now the RNN output softmax layer will have three slots at each of the four time steps telling how likely each word is. Now we sample a word from each time step with some probability, these will be our G on the RHS of the equation. Then to calculate Q for intermediate time steps, at each time step we fix the words so far from beginning to that time step t and sample again in the future time steps. Now we have in total two complete sentences for each step, except for the last time step because we already have a complete sentence. Now we give these (2+2+2+1) sentences to the discriminator to get the rewards. Then at each time step we average the corresponding two sentences to get Q values.. We have the Q's and G's at each step we simply take a sum of products and we are done.
- The term yt (token selected at t) belongs to Y (vocabulary) confuses me. Shouldn't it be sum over t?
- or are we considering the G's for all words at each time step and do monte carlo rollouts for all of the words in the vocabulary?