LLM-like architecture capable of dynamically learning from its own output

Question

Language Learning Models (LLMs) have demonstrated remarkable capabilities in quick learning during inference. They can effectively grasp a concept from a single example and generate relevant outputs. However, a noticeable limitation of LLMs is their inability to work on large-scale projects, such as generating a cohesive book, due to context size constraints. One potential solution is enabling LLMs to learn from their own outputs, but the learning rate during the training phase is significantly slower than the rate at which they absorb and process concepts during inference.

I am interested in exploring the possibility of an architecture that allows for fast and efficient learning, enabling the AI to dynamically and incrementally train from its own output. This would facilitate the production of large-scale cohesive outputs that surpass context limitations. Although RNNs like RWKV theoretically offer "infinite context size," it is not practically useful as the model tends to "forget" concepts that are distant in the prompt.

Human learning involves continuous adjustments of synapses while working on a problem, which seems like an ideal approach to emulate. Are there any existing or proposed architectures that incorporate this mechanism, allowing for dynamic learning from generated outputs and the creation of large-scale cohesive content in LLMs?

Answer 1

in my (limited) experience, simply re-training an existing tensor model with every exchange ends up by deteriorating previous runs. In other words, a traditional LLM can be trained on the original dataset, then fine-tuned once. What causes this deterioration?

Answer 2

RLHF (Reinforcement Learning through Human Feedback) is quite like your idea. When a model learns from its own outputs, it needs a way to tell if those outputs are good or bad. RLHF uses a method called reinforcement learning to train the model. It checks whether the model's outputs get good feedback (a positive reward) or bad feedback (a negative reward).

This is similar to how humans learn. We tend to do things that make us feel good or get a positive result and avoid things that make us feel bad or get a negative result.