Securing Multimodal Agents in the Wild
I formulated the action gap, the distance between model-level and agent-level safety. I built a multi-step document-analysis agent that records action-level execution traces in MLflow, then ran attacks across six categories (direct, indirect, prompt, context, tool, hybrid) on an AgentHarm subset scored by an LLM judge. Qwen2.5-32B-Instruct and GPT-4.1-mini ran under identical agent logic. Stronger model-level refusal did not imply agent-level safety. GPT-4.1-mini refused more objectives yet had higher attack success under adaptive attacks, 45.7% against 37.5%, and per-action attack success reached 0.84.