In the paper A Simple Neural Attentive Meta-Learner, the authors mentioned right before Section 3.1:
we preserve the internal state of a SNAIL across episode boundaries, which allows it to have memory that spans multiple episodes. The observations also contain a binary input that indicates episode termination.
As far as I can understand, SNAIL uses temporal convolutions to aggregate contextual information, from which causal attention learns to distill specific pieces of information. Temporal convolutions does not seems to maintain any internal state, and neither does the attention mechanism they use after this paper. This makes me wonder: "What is the internal state of a SNAIL?"