1
$\begingroup$

I am currently working with large language models like llama and mistral, interested in techniques for making these models more explainable. I am looking for some tools or techniques which can help me with insights of how their models comes up with the predictions.

Which are the tools and techniques for interpreting the decision-making process?

$\endgroup$

2 Answers 2

2
$\begingroup$

This is an active research area, and we can be confident that we don't know the best techniques yet. There are overviews of the field on Wikipedia, in this July 2023 special issue, and in this tag on the AI alignment forum. This blog post talks specifically about LLM interpretability.

The most promising technique I know of for LLMs comes from Anthropic in their Towards Monosemanticity paper. Scott Alexander wrote an explainer with pointers to important background papers. Unfortunately, it doesn't scale to Llama/Mistral levels yet.

For more near-term applications, attention heatmaps should be scalable with some work. Here is a 2019 blog post constructing one for BERT and a GitHub repo applying it to Llama.

$\endgroup$
2
$\begingroup$

Simple explanations of decision-making in LLMs are usually false. Getting a correct explanation of how e.g. LLaMA-2-7b solves some task can be difficult. Here's my mental model of how tough it is in early 2024:

  • could likely be done by reusing existing code over a few weeks of work: indirect object identification, modular addition
  • research project: factual Q&A on a given topic, e.g. capitals of states, or birthdates of famous figures
  • tough research project: anything involving composition of multiple facts stored in model's weights
  • impossible with today's technology: e.g. solving coding problems, refusing to answer harmful requests

My understanding is that, in early 2024, there does not exist a task where looking up how a 7B LLM does it is trivial, in the sense that you could get a confidently correct explanation in a few hours, using openly available tools.

I recommend taking a look at TransformerLens; this repo is used by many recent mechanistic interpretability. I guess (but have no inside info) that Google DeepMind and Anthropic have better tools internally.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .