Adversarial Distillation: OpenAI Cuts Attack Rate

Network diagram representing defenses against adversarial distillation of AI models

OpenAI says updated protections on its GPT-6.2 Astra and Sol models sharply cut adversarial distillation, the unauthorized use of a model’s outputs to train a rival model. In its own testing, attacks that tried to pull out protected reasoning succeeded 54.0% of the time on average against GPT-6 Astra. Against GPT-6.2 Astra, that dropped to 0.2%.

The company has also published what it has learned about stopping the practice, in the hope that other providers can use it.

What adversarial distillation is

Distillation is a standard technique in which one model learns from another’s answers, reasoning or judgments. It becomes adversarial when an actor uses a frontier model’s outputs without authorization, then deploys the resulting model without comparable protections.

OpenAI’s concern goes beyond intellectual property. A student model can pick up advanced capabilities without the safeguards that limit how the original may use them. The risk is clearest in dual-use fields. A model that helps defenders find software flaws could help an attacker exploit them, and one that assists biological research could help someone build a harmful pathogen.

Two layers of anti-distillation safeguards. Model behavior, classifiers and API protections limit extraction in each interaction. Signals inform investigations across accounts and providers; investigation findings improve protections, supported by automated red-teaming.
Image: OpenAI

Why reasoning traces are the target

Attackers go after protected reasoning, the working a model does internally before it answers, also known as chain-of-thought. These traces can hold information the final answer leaves out, which makes them valuable training data.

OpenAI ran an internal study of how cyber capabilities transfer. Its baseline was what a normal user sees: final answers, reasoning summaries and tool use. then tested what extra capability moved over when the student also saw the teacher’s exact reasoning traces. Students started from the same pretrained model and were fine-tuned on trajectories from Astra-level teachers working on coding and defensive cybersecurity tasks.

After fine-tuning, the student trained on exact traces generally beat the one trained on summaries. After a light round of reinforcement learning, the exact-trace student kept its lead when given the biggest compute budget on both benchmarks. The summary-trained student did better on smaller budgets. OpenAI’s takeaway is that shielding reasoning traces can slow the spread of cyber skills, even if the teacher’s tasks were permitted under its usage rules.

Request-level defenses

At the request level, OpenAI combines model behavior, classifiers and API controls:

  • Adversarial training teaches models to spot extraction attempts and refuse to reveal protected reasoning while still answering legitimate requests for explanations. Examples come from red-teaming, investigations and outside researchers’ disclosures.
  • Input classifiers detect extraction attempts. Output classifiers catch leaks in assistant messages and tool-call arguments, in both streaming and non-streaming responses.
  • API-level limits control where protected reasoning can be carried between conversations and models.

Tool calls need coverage because a model might leak reasoning inside a tool’s arguments while keeping it out of its reply. Suspected leaks can be stopped before the caller sees them.

An earlier September disclosure covered attacks that copied encrypted reasoning into a different conversation, then asked the model to spell it out. OpenAI has since tightened the walls between users, workspaces, organizations and model families, and it shut a route that could have revealed another user’s encrypted reasoning.

With the 6.2 models, protected reasoning is tied to the conversation history that produced it. If an earlier message is rewritten, the API drops the reasoning that followed. Older model families with weaker safeguards also can’t read the new models’ protected reasoning.

Campaign-level enforcement

Individual requests may look harmless, so OpenAI also looks for patterns in request and account signals that suggest coordinated extraction. When it finds violations, it acts against the whole network of accounts, uses each investigation to improve defenses, and shares indicators of compromise with other model providers.

OpenAI’s consistent-standards policy sets shared safeguards and coordinated enforcement for providers offering its models. It says sharing findings across industry and government can help disrupt campaigns.

How the 54.0% to 0.2% figure was measured

The result came from an adaptive evaluation using strong automated red-teaming. Attackers tried to recover protected reasoning the model produced while answering an earlier prompt. OpenAI also runs static evaluations that replay previously successful attacks to see which still work.

These are the company’s own tests, not an outside audit, so the numbers show direction more than a guarantee.

What OpenAI plans to study next

OpenAI says it will keep studying how much capability transfers across different student models and training recipes. Expect continued attacks, and more shared indicators between providers as campaigns are found.

Source: OpenAI

Written by

Saksham Bhargava

Saksham Bhargava runs Alteroid and has been writing about consumer technology for over five years, covering iPhone, Android, Windows, AI tools and gaming. With an MCA and six years of hands-on web development experience, he tests the features he writes about on his own devices and focuses on clear, practical guides that solve real problems. Away from the screen, he enjoys travelling to places most tourists skip.