Multimodal models (Vision-Language and Audio-Language models) introduce significantly wider attack surfaces. Safety guardrails that are robust against pure text can be effortlessly bypassed when harmful requests are rendered as stylized typography inside an image, or when visual embeddings are perturbed with adversarial gradient noise.
Cross-Modal Alignment Gap
Most safety post-training is performed predominantly on text data. The vision encoder projects visual features into the text embedding space, creating pathways where visual semantics bypass text-only refusal classifiers.
Typographic Attacks
Rendering forbidden text on a natural background or in unusual fonts causes the vision encoder (CLIP/SigLIP) to read the text and inject it into the LLM backbone without triggering text input regex filters.
Projected Gradient Descent (PGD) on Pixels
Continuous pixel perturbations (epsilon <= 8/255) can be optimized to force the visual projection layer to produce embeddings that mimic affirmative compliance tokens.
MATHEMATICAL FORMULATION · VISUAL ADVERSARIAL LOSS FORMULATION
Where V(x) is the visual encoder embedding, y_target is the compliance state, and epsilon is the bounded L_infinity perturbation radius that keeps the adversarial image visually indistinguishable from the original image.
REAL-WORLD PRODUCTION ENGINEERING
Pass all user-uploaded images through an independent OCR pre-processor and evaluate the extracted text with standard text moderation guardrails.
Train vision encoders with joint multimodal adversarial contrastive loss to align safety boundaries across text and vision embeddings.
Apply random image transformations (JPEG compression, slight rotation, blur) before feeding into the vision encoder to break brittle pixel perturbations.