Transformer: Attention is all you need ViT: 16 * 16 image DeiT, Swin Transformer, MAE, DINO, DINOv2, SAM, CLIP, BLIP-2, Flamingo, LLaVA

ViT becomes the foundation of Multimodal. image -> tokens and then let transformer process it. Everything is tokens.

---

Linear: nn.Linear(1024, 4096) 1024d vision feature -> 4096d LLM embedding vision feature, LLM embedding

essence: transform the coordinate

---

MLP: Multi-Layer Perception multi-layer + non-linear activate

$MLP(x)=W_2\sigma (W_{1}x+b_1)+b_2$ Linear, GELU, Linear

Combination of Linear is still Linear, doesn't add the ability of expression for example: $W{2}_{}(W_{1}x)=Wx$

Why MLP > Linear: MLP introduces activation (ReLU, GELU, SiLU) adds the non-linear so MLP can fit complex function

In multimodal, MLP learns vision semantics to language semantics

many projector: ViT -> MLP -> LLM

---

Projector = the module that project vision feature to language space Linear, MLP, Transformer, Q-Former

---

Q-Former comes from BLIP-2 means Querying Transformer

Q-Former = actively extract information and cross-modal interaction

模块	能力	非线性	Attention	信息选择	压缩能力
Linear	弱	无	无	无	弱
MLP	中	有	无	无	中
Q-Former	强	有	有	强	强

数学：

$$Attention(Q_{query},K_{vision},V_{vision})$$

即：

• Query：
  learnable tokens
• K/V：
  visual features

和 normal self-attention:

$$Q=K=V$$

different

Learnable queries

trending:

Perceiver
Cross attention
Q-Former
Resampler

---