OpenAI Cuts Inference Costs in Half on Key Models

What did OpenAI's engineers actually discover?

OpenAI engineers cut inference costs by more than half on the models the optimization touched, according to The Information. Inference cost is the computational expense a model incurs when responding to a user query — the metric most directly tied to AI service pricing and margins.

The engineers shared the finding internally with colleagues earlier this month. OpenAI has not issued a public statement confirming the development, and the exact technical method has not been disclosed.

How did this show up in the real world?

The first measurable result appeared in logged-out ChatGPT traffic. After the optimization was applied to that segment, the number of GPUs needed to power it dropped to just a few hundred, as reported by The Information via Digg.

That is a striking reduction. Logged-out traffic covers every visitor using ChatGPT without a free or paid account — a meaningful slice of overall usage.

What is the optimization, exactly?

The source did not disclose specific technical details. Speculation from observers includes quantization, KV caching, smarter query batching, or routing simpler queries to cheaper models — but none of these have been confirmed.

You might also like

Meta Compute: Zuckerberg's Tens-of-Gigawatts AI

China's Humanoid Robots Stun at 2026 Spring Gala

Google Emissions Up 51% Since 2019 Despite Clean

Anthropic Launches Claude Science for Drug

What is confirmed: the gains came from improved utilization of existing server resources, not from deploying additional computing chips. That distinction matters. It means OpenAI squeezed more out of hardware it already owns.

Why does this matter for OpenAI's business?

Here's what we know so far about the financial context. OpenAI ended Q1 2026 with a 39% gross margin and has a stated target of 52% by year-end. Lower inference costs give the company room to improve margins, raise usage limits, or reduce API pricing pressure on developers.

The optimization has only been applied to logged-out ChatGPT traffic so far. How much headroom remains for paid traffic, the API, or other products is still an open question.

Who else is working on this?

OpenAI is not alone. Anthropic and Google are also working on server-level efficiency gains, according to Digg's reporting. No public benchmarks or technical details have emerged from either company.

For context on how rivals are competing: Claude paid users grew 75% since January 2026, and California partnered with Anthropic for state AI services — both signs that inference cost competitiveness has real commercial stakes.

What about OpenAI's hardware moves?

Separately, OpenAI showcased a keyboard called the Codex Micro at the AI Engineer World Fair in San Francisco on June 29. The device was co-developed with accessories company Work Louder. OpenAI spokesperson Dominik Kundel introduced it at the event, describing it as designed "to dramatically improve how people use Codex efficiently."

The Codex Micro debut is a separate development from the inference optimization story, but both arrived within days of each other. This also connects to broader moves in the developer tooling space — see Meta's ban on Claude Code & Codex for how rivals are responding to OpenAI's Codex push.

What are the limits of this news?

Several important caveats apply:

• OpenAI has not officially confirmed the inference cost reduction
• The information remains at the stage of "engineers sharing with select colleagues"
• The optimization has only been applied to logged-out ChatGPT traffic so far
• No timeline has been given for broader rollout or API pricing changes
• The exact technical method remains undisclosed

For developers watching OpenAI GPT model updates closely, the gap between an internal engineering win and a productized cost reduction is still significant.

Before and after: what the sources confirm

The most important confirmed fact: OpenAI engineers told colleagues this month that their optimization cut inference costs by more than half, and logged-out ChatGPT traffic is already running on just a few hundred GPUs as a result.