RESEARCH
September 2024: hijackable web agents, automated vishing, post-quantum under attack
September 2024 marks AI security's shift from models to agents: web pages that steal personal data from agents and a defense based on information flow control. Social engineering gets automated with LLM-driven phone calls, while a field study scales back expectations for phishing training. In post-quantum, fault injection attacks hit ML-KEM and ML-DSA on microcontrollers.
Archive: reconstruction written in November 2026
1. EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage
arXiv · Zeyi Liao, Lingbo Mo, Chejian Xu, Mintong Kang et al. · AI & LLM Security
In short: Hidden content on a website tricks web agents into handing over users' personal data.
The paper studies the privacy risks of generalist web agents on compromised websites. The EIA attack injects content adapted to the environment the agent operates in. Across 177 real action steps involving personal data, it steals specific data up to 70% of the time and the full user request in 16%. It is hard to detect, and a defensive system prompt is not enough.
Why it matters. An agent that books or fills in forms for a user becomes a data leakage channel: it only needs to visit a manipulated page.
2. System-Level Defense against Indirect Prompt Injection Attacks: An Information Flow Control Perspective
arXiv · Fangzhou Wu, Ethan Cecchetti, Chaowei Xiao · Formal Methods & Verification
In short: A system-level defense against indirect prompt injection, based on information flow control.
In LLM-based systems each step's output feeds the next, so malicious input from any source can hijack execution. The paper proposes an "f-secure" system: planning produces structured executable plans and a security monitor keeps untrusted input out of planning. Formal models and benchmarks show robust protection while preserving functionality and efficiency. The code is public.
Why it matters. It moves the defense from prompt filtering to architecture: the kind of control to look for in agents that touch data or payments.
Read the paper · PDF · Code
3. The Early Bird Catches the Leak: Unveiling Timing Side Channels in LLM Serving Systems
IEEE Transactions on Information Forensics and Security · Linke Song, Zixuan Pang, Wenhao Wang, Zihao Wang et al. · Systems & Software Security
In short: Caches in shared LLM services leak system prompts and other users' prompts through response times.
The authors find new timing side channels in systems serving LLMs to many users, caused by shared caches and GPU memory allocation. By timing responses and classifying cache hits on the KV cache and semantic cache, an attacker infers private prompts with high accuracy. A token-by-token search recovers shared prompt prefixes. Black-box tests on popular online services confirm the risk.
Why it matters. LLM providers' performance optimizations become a confidentiality risk for customers sharing the infrastructure.
4. Models Are Codes: Towards Measuring Malicious Code Poisoning Attacks on Pre-trained Model Hubs
arXiv · Jian Zhao, Shenao Wang, Yanjie Zhao, Xinyi Hou et al. · Software Supply Chain
In short: First systematic study of malicious code hidden in Hugging Face models and datasets.
Pre-trained models are also code: some formats execute instructions when loaded. The paper classifies vulnerable formats and proposes MalHug, combining deserialization, taint analysis and heuristic patterns. Over three months on an internal mirror at a large fintech, it scanned more than 705,000 models and 176,000 datasets and found 91 malicious models and 9 malicious loading scripts.
Why it matters. Downloading a model from a public hub means running third-party code: a new link in the software supply chain.
5. DarkGram: A Large-Scale Analysis of Cybercriminal Activity Channels on Telegram
USENIX Security Symposium · Sayak Saha Roy, Elham Pourabbas Vafa, Kobra Khanmohammadi, Shirin Nilizadeh · Systems & Software Security
In short: 339 cybercrime Telegram channels, followed by 23.8 million users, analysed at scale.
DarkGram, a BERT-based classifier with 96% accuracy, analyses 53,605 posts published between February and May 2024. Compromised credentials, malware, exploit kits and scams circulate, often free to attract paying buyers. The channels also target their own subscribers: 28.1% of links are phishing and 38% of executables carry malware. After takedowns they migrate quickly.
Why it matters. It shows how easy and profitable the market for credentials and attack tools is, and how little channel takedowns achieve.
6. On the Feasibility of Fully AI-automated Vishing Attacks
arXiv · João Figueiredo, Afonso Carvalho, Daniel Castro, Daniel Gonçalves et al. · Systems & Software Security
In short: A fully automated, LLM-driven vishing system convinces many people to reveal sensitive data.
ViKing combines an LLM with speech synthesis and recognition, all built from public technology, to run fraudulent calls. In a controlled experiment with 240 participants it persuaded many to disclose sensitive information, including people explicitly warned about the risk. Conversations were generally rated as realistic.
Why it matters. The cost of a phone fraud campaign collapses: more credential theft and payment fraud attempts with the same number of attackers.
7. Content, Nudges and Incentives: A Study on the Effectiveness and Perception of Embedded Phishing Training
ACM Conference on Computer and Communications Security · Daniele Lain, Tarek Jost, Siniša Matetić, Kari Kostiainen et al.
In short: Embedded phishing training works as a reminder, not through its content, which almost nobody reads.
A large-scale mixed-methods study of a partner company's employees breaks down the training delivered to those who fail a phishing simulation. The effect comes from the periodic reminder of the threat, not from the training material, rarely consumed for lack of time and perceived usefulness. Delaying training is as effective as current practice, and rewards do not improve results.
Why it matters. Crediting the mere existence of a training programme says little: simulation frequency and measured results are what count.
8. Mind the Faulty Keccak: A Practical Fault Injection Attack Scheme Apply to All Phases of ML-KEM and ML-DSA
IACR Cryptology ePrint Archive · Yuxuan Wang, Jintong Yu, Shipei Qu, Xiaolin Zhang et al. · Post-Quantum Cryptography
In short: Faults induced in Keccak allow key recovery and signature forgery in ML-KEM and ML-DSA.
Keccak is the hash ML-KEM and ML-DSA use to derive sensitive values. By aborting its loops through fault injection, the authors build six attacks on ML-KEM and five on ML-DSA: key recovery, signature forgery and verification bypass, across every phase of the algorithms. On Cortex-M0+, M3, M4 and M33 with low-cost electromagnetic equipment, the fault succeeds 89.5% of the time.
Why it matters. Post-quantum migration on embedded devices needs physical countermeasures, not just adoption of the NIST standards.
9. The Impact of SBOM Generators on Vulnerability Assessment in Python: A Comparison and a Novel Approach
arXiv · Giacomo Benedetti, Serena Cofano, Alessandro Brighente, Mauro Conti · Software Supply Chain
In short: SBOMs from current tools are often incomplete and let known vulnerabilities slip through in Python projects.
The SBOM, the list of a software product's components, is promoted to make the supply chain transparent. The paper is the first analysis of how SBOM generator errors reduce downstream tools' ability to find vulnerabilities. It identifies the causes, mainly in component identification and dependency resolution, and proposes PIP-sbom, more accurate than the best existing tools.
Why it matters. An SBOM is not a guarantee in itself: its quality decides whether component vulnerabilities are actually seen.
10. You Might Have Known It Earlier: Analyzing the Role of Underground Forums in Threat Intelligence
Research in Attacks, Intrusions and Defenses · Tommaso Paladini, Lara Ferro, Mario Polino, Stefano Zanero et al. · Malware & Threat Intelligence
In short: Over twenty years, underground forums anticipated security reports for more than 60% of threats, but the lead is shrinking.
The authors analyse 88 million hacker forum posts and 75,000 articles over twenty years, extracting entities with BERT models and comparing dates. Forum discussions precede official reports for over 60% of entities. Since 2012, however, the lead has steadily declined, possibly due to more scrutiny or a move to other channels, while new malware still circulates in the forums.
Why it matters. Threat intelligence must follow actors where they move, for instance to messaging channels, to give useful early warning.
Trends of the month
AI & LLM Security
134 papers this month on this topic
AI security is already one of the month's most frequent topics. The focus moves from isolated models to the systems around them: web agents acting on websites, pipelines chaining steps, infrastructure shared between customers. Attacks exploit the environment, meaning manipulated pages and shared caches, more than the model itself. For insurers, the useful question is how a company uses agents and with what permissions, not just which model it adopts.
Papers cited:
- EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage
- System-Level Defense against Indirect Prompt Injection Attacks: An Information Flow Control Perspective
- The Early Bird Catches the Leak: Unveiling Timing Side Channels in LLM Serving Systems
Usable Security & Human Factors
27 papers this month on this topic
Two papers look at the human factor from opposite sides. On the attack side, LLM-driven fraudulent calls convince even people warned of the risk. On the defense side, embedded phishing training works as a periodic reminder more than as teaching. Together they suggest social engineering is becoming cheaper and more scalable, while awareness-based controls have measurable limits. Process controls, such as dual verification of payments, remain decisive.
Papers cited:
- On the Feasibility of Fully AI-automated Vishing Attacks
- Content, Nudges and Incentives: A Study on the Effectiveness and Perception of Embedded Phishing Training
Post-Quantum Cryptography
27 papers this month on this topic
In August 2024 NIST published the ML-KEM and ML-DSA standards, and by September research already focuses on their implementations. Fault injection attacks on Keccak, the hash both schemes use, show that keys can be recovered and signatures forged on low-cost microcontrollers. The lesson for migration is that choosing the algorithm is only the first step: physical protection of devices matters as much as the standard.
Papers cited:
How this digest is made
This issue belongs to the digest archive, reconstructed after the fact. The month's papers were collected from arXiv, the IACR ePrint archive and the main security conferences and journals; the selection also weighs the citations received since publication. An AI model assessed the candidates and wrote the summaries, which are based on the abstracts: the original paper is the reference for details.
Papers of the month assessed: 1213.