Insight into architecture details of Vision Llama 3.2

Question

I would like to understand the internals of Meta's new multimodal Vision-Text 3.2 models. Not much I could find in online sources or blogs. The code is available and I'm trying to read it, but my understanding is still limited.

It seems to me that the main ingredient is that we have an image and a text

• the image is encoded using a pretrained image encoder first, but I'm not sure if it was trained separately or as part of the overall Llama3.2 training
• Both "image sequence" and "text sequence" interact in some cross-attention layers, during the image forward pass

I'm not sure how they handle multiple images interleaved with texts, however.

Are there any helpful online sources with details, or would someone educated ellaborate on the details of the architecture and training here?

I would also like to understand what is the advantage of this cross-attention compared to Llava which only uses a projection of the encoded image sequence to the Llama's input embedding space and seems to be conceptually simpler.

I will also appreciate partial answers or relevant remarks.

Answer 1

In image projection models like Llava image features are first passed through an image encoder (e.g., a CNN or ViT) and then projected into the shared text embedding space. Once projected, the image embeddings are treated as if they are a separate sequence of tokens alongside the text tokens. Each sequence (image and text) is processed by the transformer independently, and their interaction is restricted to how the sequences are interleaved and combined at the input stage either during training or inference.

In contrast, cross-attention such as in Llama 3.2 means how each granular token (whether from the text or image) can attend to tokens from the other modality during each layer’s computation, which creates iterative interactions where both modalities can continuously inform and refine the model’s understanding. At lower layers this might be simple matching (e.g., "dog" in text and a "dog" in the image), while at higher layers this might involve more abstract reasoning (e.g., "The dog in the park is barking at the bird"). Also such iterative interactions can learn certain visual features which are only relevant in specific textual contexts, for instance, recognizing a dog in the image may not be helpful unless the context in the text also mentions a dog.

Finally projection model's fixed transformation of an image into a text-based space via feature extraction could lose important spatial or semantic details of the image. In cross-attention models, the model preserves the image's richness using separate image tokens and maintaining their detailed information through the interaction with the text. This is especially beneficial in tasks that require fine-grained understanding, like image captioning, where knowing the exact position of objects and their relationships to text is critical.