Suppose you have a pre-trained autoregressive language model, and some cost function C mapping strings to numbers. Lower costs mean a given generated string is "better". Pass the model some input sequence and autoregressively generate an output sequence, say by just directly sampling from the model's output distribution at each timestep, for say n timesteps. Now let (l1, ... ln) denote the logits of the tokens selected at timesteps 1 through n. A very simply reinforcement learning objective is

C(y)(l1 + ... + ln)

where C(y) is the cost of the output sequence. In other words, if C(y) is positive (bad output), make those logits smaller, and if C(y) is negative (good output), make the logits bigger.

This seems like a sensible and simple reinforcement learning objective, but when I run it, all I find is that the system just minimizes its logits. These are histograms of the model's output logits at various points during training - as training progresses the histograms move to the left.

enter image description here

Basically the model is tending towards a uniform output distribution, minimizing the loss by minimizing the l1 + ... + ln factor rather than the C(y) factor.

How do SOTA reinforcement learning methods get around this issue?



You must log in to answer this question.

Browse other questions tagged .