Could someone clear my doubt on the loss function used in SeqGAN paper . The paper uses policy gradient method to train the generator which is a recurrent neural network here.

  1. Have I interpreted the terms correctly?
  2. What are we summing over? The entire vocabulary of words?

Loss function - my interpretation: enter image description here

enter image description here Suppose, total time steps = 4, vocabulary size = 3, rollout size = 2. so now the RNN output softmax layer will have three slots at each of the four time steps telling how likely each word is. Now we sample a word from each time step with some probability, these will be our G on the RHS of the equation. Then to calculate Q for intermediate time steps, at each time step we fix the words so far from beginning to that time step t and sample again in the future time steps. Now we have in total two complete sentences for each step, except for the last time step because we already have a complete sentence. Now we give these (2+2+2+1) sentences to the discriminator to get the rewards. Then at each time step we average the corresponding two sentences to get Q values.. We have the Q's and G's at each step we simply take a sum of products and we are done.

  1. The term yt (token selected at t) belongs to Y (vocabulary) confuses me. Shouldn't it be sum over t?
  2. or are we considering the G's for all words at each time step and do monte carlo rollouts for all of the words in the vocabulary?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.