It seems the keywords you're looking for are knowledge distillation, transfer learning, domain adaptation, and fine tuning of LLMs which are all based on the fundamental semantic vector embedding and similarity learning as discussed in today's another related post which you may refer. Knowledge base acquisition and learning are inherently semantic in nature, thus similarity learning of the knowledge manifold in the embedding space is foundational in this respect in addition to the traditional syntactic concept learning and logic proof derivation.

Knowledge distillation transfers knowledge from a large model to a smaller model without loss of validity. As smaller models are less expensive to evaluate, they can be deployed on less powerful hardware (such as a mobile device)... The idea of using the output of one neural network to train another neural network was studied as the teacher-student network configuration. Hope below quoted sections are relevant and help your research.

Domain adaptation is a field associated with machine learning and transfer learning. This scenario arises when we aim at learning a model from a source data distribution and applying that model on a different (but related) target data distribution. For instance, one of the tasks of the common spam filtering problem consists in adapting a model from one user (the source distribution) to a new user who receives significantly different emails (the target distribution).

In deep learning, fine-tuning is an approach to transfer learning in which the parameters of a pre-trained model are trained on new data.[1] Fine-tuning can be done on the entire neural network, or on only a subset of its layers, in which case the layers that are not being fine-tuned are "frozen" (or, not changed during the backpropagation step)... Fine-tuning can be combined with a reinforcement learning from human feedback-based objective to produce language models like ChatGPT (a fine-tuned version of GPT-3) and Sparrow.

It seems the keywords you're looking for are knowledge distillation, transfer learning, domain adaptation, and fine tuning of LLMs which are all based on the fundamental semantic vector embedding and similarity learning as discussed in today's another related post which you may refer. Knowledge base acquisition and learning are inherently semantic in nature, thus similarity learning of the knowledge manifold in the embedding space is foundational in this respect in addition to the traditional syntactic concept learning and proof-tree natural deduction.

Knowledge distillation transfers knowledge from a large model to a smaller model without loss of validity. As smaller models are less expensive to evaluate, they can be deployed on less powerful hardware (such as a mobile device)... The idea of using the output of one neural network to train another neural network was studied as the teacher-student network configuration. Hope below quoted sections are relevant and help your research.

Domain adaptation is a field associated with machine learning and transfer learning. This scenario arises when we aim at learning a model from a source data distribution and applying that model on a different (but related) target data distribution. For instance, one of the tasks of the common spam filtering problem consists in adapting a model from one user (the source distribution) to a new user who receives significantly different emails (the target distribution).

In deep learning, fine-tuning is an approach to transfer learning in which the parameters of a pre-trained model are trained on new data.[1] Fine-tuning can be done on the entire neural network, or on only a subset of its layers, in which case the layers that are not being fine-tuned are "frozen" (or, not changed during the backpropagation step)... Fine-tuning can be combined with a reinforcement learning from human feedback-based objective to produce language models like ChatGPT (a fine-tuned version of GPT-3) and Sparrow.