OpenAI has unveiled GPT-Red, an internal automated red-teaming model designed to systematically discover prompt injection vulnerabilities in its AI systems. This innovative approach aims to significantly enhance AI safety by identifying and rectifying potential security flaws before models are deployed to the public, marking a crucial step in the ongoing effort to bolster the robustness of large language models (LLMs).
The artificial intelligence company stated that GPT-Red has proven to be a “strong red-teamer,” capable of exploiting weaknesses in previous models with prompt injection attacks. OpenAI is leveraging GPT-Red to adversarially train its newer models, such as GPT-5.6, making them substantially more resilient to such malicious instructions. This proactive strategy addresses a persistent challenge for LLMs, where carefully crafted prompts can trick models into executing unintended or harmful actions.
GPT-Red: Automating AI Security Testing
GPT-Red operates by mimicking the process of human red-teamers. It sends a prompt to a target GPT model, meticulously observes the response, and iteratively refines its approach to achieve a malicious objective. These objectives can range from exfiltrating sensitive data to an external server to performing fraudulent transactions. The development comes at a critical time as agentic AI systems increasingly integrate with external data sources via web browsers, connected applications, and local files, thereby expanding the potential attack surface.
As AI agents become more interconnected, the pathways for malicious actors to influence model outcomes multiply. By embedding hidden or disguised instructions within seemingly innocuous input data—such as emails, web pages, tool responses, or code repositories—attackers can attempt to manipulate AI behavior. GPT-Red is designed to scale this adversarial testing process, enabling the identification of novel failure modes and the development of effective countermeasures before potential vulnerabilities can be exploited in real-world scenarios.
OpenAI reported that GPT-Red is integrated directly into the training pipeline for its production models. This integration has led to the development of GPT-5.6 Sol, which OpenAI claims is its most robust model to date against prompt injections. The company indicated that GPT-5.6 Sol exhibits six times fewer failures on direct prompt injection benchmarks compared to its predecessor, GPT-5.5, which was released just four months prior. This highlights the effectiveness of AI-driven red-teaming in improving model security.
The process involves training GPT-Red using self-play reinforcement learning. In this setup, GPT-Red and a set of diverse “defender” LLMs are trained concurrently across a wide array of red-teaming scenarios. GPT-Red is incentivized to successfully execute prompt injection attacks, while the defender models are rewarded for resisting these attacks and completing their intended tasks. This creates a dynamic feedback loop where the defender models become more robust, forcing GPT-Red to develop more sophisticated attack methods.
Examples of Prompt Injection Attempts
OpenAI shared several examples of prompt-injected conversations that GPT-Red has successfully exploited during its testing phases. These include, but are not limited to, attempts at internal directory exfiltration, fraudulent payment instructions, and the theft of Amazon Web Services (AWS) credentials. The model has also been tested against scenarios involving disabling two-factor authentication (2FA), uploading credentials files, injecting external scripts, forwarding API keys, and deploying malicious scraper scripts.
The continuous nature of this adversarial training suggests that as defender models improve, GPT-Red must evolve. OpenAI highlighted that GPT-Red has demonstrated proficiency in generating successful attacks against indirect prompt injections, even surpassing human red-teamers in certain scenarios when targeting earlier models like GPT-5.1. This ongoing evolution ensures that AI safety measures remain ahead of potential threats.
OpenAI emphasized that GPT-Red is maintained as a separate system to prevent its malicious capabilities from falling into the wrong hands. The constant vigilance against prompt injection attacks is crucial, as bad actors continually seek to circumvent AI models’ ethical and safety guidelines. By keeping GPT-Red isolated, OpenAI mitigates the risk of its own security tools being weaponized.
In a practical demonstration, OpenAI deployed GPT-Red against an AI-powered vending machine developed by Andon Labs. After simulated practice, the model successfully executed three objectives: reducing the price of a high-value item to the minimum of $0.50, ordering a $100 item for the same low price, and canceling another customer’s order. Following responsible disclosure practices, new safeguards are currently being tested for that system.
Another case study involved using GPT-Red to attack a Codex command-line agent, based on a smaller version of GPT-5.4. The AI successfully transmitted sensitive data in more instances than a GPT-5.5 baseline across ten specific data-exfiltration tasks. Furthermore, an early iteration of GPT-Red identified a new category of direct prompt injection attacks, known as Fake Chain-of-Thought (CoT) attacks. These attacks achieved over 95% success rates on GPT-5.1 but have been significantly reduced to below 10% on the more advanced GPT-5.6 Sol.
Reflecting on the robustness of their latest models, OpenAI stated that indirect prompt injection benchmarks, particularly those targeting developer tools and browsing capabilities, have seen performance saturation with their newest model, achieving over 97% accuracy. The company also noted substantial improvements in GPT-Red’s own robustness. Across a broad range of testing environments, GPT-Red’s attack success rates have shown a monotonic decrease over time. With the release of GPT-5.6 Sol, the model fails on a mere 0.05% of GPT-Red’s direct prompt injection attempts.
This announcement follows a recent audit by OpenAI of the SWE-Bench Pro benchmark, which revealed that approximately 30% of tasks within the benchmark were found to be flawed. This discovery has led OpenAI to retract its previous recommendation for adopting the benchmark as a measure of frontier coding capabilities. Earlier in February, the company had signaled its move away from SWE-bench Verified due to fundamental design and data contamination issues, underscoring the challenges of creating reliable AI evaluation tools.
OpenAI’s internal analysis has provided evidence of significant issues within a substantial portion of the SWE-Bench Pro dataset, with their data analysis pipeline flagging 27.4% of tasks as broken, and human annotation identifying 34.1%. The company reiterated the importance of evaluations that provide meaningful signals through benchmarks that are difficult to manipulate, easy to trust, and genuinely reflect model capabilities and alignment.
The ongoing development and implementation of advanced red-teaming methodologies like GPT-Red are critical for the responsible advancement of AI. As AI systems become more integrated into daily life and critical infrastructure, the ability to proactively identify and mitigate security risks will be paramount. The next steps will likely involve further refinements to GPT-Red and continued adversarial training to stay ahead of evolving threats, ensuring that AI technologies are both powerful and secure.