TLDR Tech / InfoSec
1Password open sources a benchmark to stop AI agents from leaking credentials
SCAM (Security Comprehension and Awareness Measure) is an open-source benchmark from 1Password that tests whether AI agents behave safely in real workflows, such as opening emails, retrieving credentials, and filling out login forms. It addresses the gap where models that can identify phishing when prompted still fall for attacks when operating autonomously. Testing across eight frontier models revealed safety scores ranging from 35% to 92%, with every model exhibiting critical failures, such as entering credentials on phishing pages or forwarding passwords to external parties. However, applying a short security "skill file" dramatically reduced failures across all models. Released under the MIT License, the benchmark includes 30 workplace scenarios, a scoring framework, and video replay tooling to support enterprise AI agent safety evaluation.
Read the full story