700 Tokens Per Second: The New Benchmark That Could Rattle Nvidia's Throne

Google’s custom silicon has officially crossed into one of its biggest competitors’ server rooms, marking a pivotal moment in the race for AI infrastructure dominance. This move signals that the era of a single, unassailable hardware standard may be ending, as major tech giants begin to diversify their compute strategies with aggressive new partnerships.

The announcement, made via Google’s developer blog, details the release of a PyTorch-native version of the SGLang inference engine. This software update allows Google’s Tensor Processing Units (TPUs) to serve AI workloads directly within the PyTorch framework, a standard widely adopted by the industry. Previously, running these chips required developers to use Google’s proprietary JAX framework, creating a significant barrier for teams invested in PyTorch. A matching open-source version of vLLM, another popular inference engine, was also unveiled to support this transition.

The performance figures accompanying the release are substantial and suggest a tangible competitive advantage in specific scenarios. According to data from Inferact, a startup founded by the creators of vLLM, a tuned setup serving the 2.8-trillion-parameter Kimi K3 model achieved over 700 tokens per second for a single user on the seventh-generation Ironwood TPU. This test utilized 16 Ironwood chips with speculative decoding. For comparison, the same workload on 16 Nvidia GB200s, using vLLM’s published setup, reached 452 tokens per second. Additionally, kernels for Kimi K3 and Qwen 3.8 27B reportedly deliver 1.4 to 2 times the throughput of the GB200 baseline at small batch sizes when speculative decoding is not used.

Zongwei Zhou, Meta’s lead for PyTorch-TPU, confirmed that the new SGLang-TorchTPU system is already running in production at Meta. Zhou, who works at RadixArk—the startup founded by the SGLang creators that collaborated with Google and Meta on the engine—described the result as a "remarkably mature, PyTorch-native serving stack." He noted that this achievement was the product of a "close three-way collaboration" between the three entities, highlighting the depth of the integration.

Jana van Greunen, director of PyTorch engineering at Meta, emphasized the strategic necessity of this shift in a post on Google’s TorchTPU page. She stated that Meta "cannot be constrained by a single hardware architecture" and that the new TorchTPU system "enables us to deploy our PyTorch workloads to Google TPUs." Google confirmed it co-developed TorchTPU specifically with Meta’s PyTorch team, indicating a targeted effort to address Meta’s specific software needs rather than a generic hardware release.

This is not the first time Meta has experimented with Google’s chips, but the current integration is significantly deeper. Google refers to the older PyTorch/XLA bridge as a "predecessor" to the new TorchTPU. Industry reports from SemiAnalysis indicated that Meta research teams used TPUs via this older bridge from 2020 to 2023 before canceling those contracts. The latest push appears to be a response to Meta’s reluctance to migrate its entire stack to JAX, aiming instead to make TPUs a native, low-friction option for PyTorch users.

The physical location of these chips remains somewhat opaque, though reports suggest a complex commercial arrangement. Talks last year reportedly involved Google Cloud TPU rentals in 2026 with direct purchases planned for 2027. Meta is also said to have signed a multi-year rental agreement. This hybrid approach suggests a cautious but committed adoption strategy, allowing Meta to scale usage without fully committing to a single vendor’s hardware procurement model.

The timing of the announcement is particularly telling. It came just one day after Amazon revealed it was open-sourcing TorchNeuron, a native PyTorch backend for its Trainium2 and Trainium3 chips. Both moves occurred at the same PyTorch Conference and appear designed to loosen the software lock-in that has long favored Nvidia. By ensuring their respective chips are first-class citizens in the most popular AI framework, both Google and Amazon are betting that software compatibility will drive hardware adoption more effectively than raw compute power alone.

The stakes for the broader market are high. If major players like Meta and Amazon can successfully decouple their AI workloads from Nvidia’s ecosystem, it could pressure the GPU giant to adjust its pricing or software support. Google, Meta, and RadixArk plan to share more details at the SGLang Summit 2026, scheduled for November 12–13 in San Francisco. In the meantime, the industry watches closely to see if this new benchmark of 700 tokens per second becomes the new standard for high-performance inference, potentially reshaping the data center landscape for years to come.