I've a pretrained model for sequence generation that I'd like to improve with RL but there are several shady points.

So, I have the following model and loss function:

def policy_gradient_loss(y_true, y_pred):
    return -tf.reduce_mean(tf.math.log(y_pred) * y_true)

embedding_layer = Embedding(input_dim=vocab_size, output_dim=128)(input_layer)
lstm_layer = LSTM(128)(embedding_layer)
output_layers = Dense(vocab_size, activation='softmax')(lstm_layer)
model = Model(inputs=input_layer, outputs=output_layers)
model.compile(loss=policy_gradient_loss, optimizer='adam')

The states or items of the sequences are dependent on all the previous states (I think it's called higher order Markov chains). For the RL training, I generate a batch of sentences. After the generation of a sentence has finished, I calculate a reward for it. The reward is a single float between -100 (worst) and 100 (best) and it values the whole sentence. After the generation of a batch of sentences has finished, every sentences are padded (with 0) to the length of the longest sentence. model.fit(...) is then called to update the network:

def _get_reward(sentence):
    # This much more complicate ofc, just to show the interval.
    return random.randint(-100, 100)

def _input_generator(batch_size):
    while True:
        x = [], y = []
        for _ in range(batch_size):
            sentence = generate_sentence()
            reward = _get_reward(sentence)
        max_len = max(map(len, x))
        yield np.array([pad_sequences([sentence], maxlen=max_len, value=0)[0] for sentence in x]), np.array(y)


My questions:

  1. Is the used loss function and network setup suitable for this scenario?
  2. Does the interval of the rewards direct the learning into the expected direction (-100 to avoid, +100 to prefer)? I've doubts since the log() of a number in the range (0, 1) is negative, multiplying it with -1 makes it positive, but if the reward is negative, then it will be negative again. And if this negative value is minimised, then the model will learn to generate the wrong results. I've read a similar question here: but the user provided the answer had also some doubts.
  3. Should I use some kind of discounting on my rewards? If yes and IIUC then it means to assign a discounted reward value to every item of a generated sequence. But then, what to assign to the padding values or how to handle padding?
  4. Is there any other issue with my approach that an experienced RL professional can spot immediately?
  • $\begingroup$ Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. $\endgroup$
    – Community Bot
    Commented Aug 30, 2023 at 4:24


You must log in to answer this question.