Google TPU throughput hits 700 tokens per second in first external Meta deployment
Meta is now running Google’s custom AI chips in production, a move that challenges Nvidia's software dominance and offers a high-speed alternative for large language model inference.
By Tunde Bailey ·
Google’s tensor processing units, which have traditionally served only internal applications, are now powering production AI workloads for Meta. This marks the first time the company has made its custom silicon available to an external customer in such a direct and integrated capacity. The shift signals a significant change in the competitive landscape for AI hardware, as major tech giants begin to diversify their chip dependencies beyond the current market leader.
The operational core of this development is a new version of the SGLang inference engine, known as SGLang-TorchTPU. Unlike previous iterations that required developers to use Google’s JAX framework, this new build is native to PyTorch. This allows engineers to run their existing code on Google hardware without extensive rewriting. Meta’s PyTorch-TPU lead, Zongwei Zhou, confirmed that the system is live at Meta today, describing the result as a remarkably mature serving stack born from a three-way collaboration.
Meta’s engineering leadership emphasized the strategic necessity of this flexibility. Jana van Greunen, director of PyTorch engineering at Meta, stated that the company cannot be constrained by a single hardware architecture. She noted that the new TorchTPU stack enables the deployment of PyTorch workloads directly onto Google TPUs. Google co-developed this integration with Meta’s team to ensure a smooth transition for developers who had previously resisted migrating away from the PyTorch ecosystem.
Performance metrics from the new setup suggest a compelling case for adoption. A tuned configuration of the vLLM engine, which was also released as an open source TorchTPU version, achieved speeds exceeding 700 tokens per second for a single user. This benchmark was run on seventh-generation Ironwood chips while serving the 2.8-trillion-parameter Kimi K3 model. The figures were generated by Inferact, a startup founded by the creators of vLLM, and they highlight the raw processing power available in this new configuration.
The performance gains are significant when compared to current industry standards. The test involved 16 Ironwood chips using speculative decoding, a technique that speeds up inference by using a smaller draft model. This setup outperformed the published benchmark for 16 Nvidia GB200 chips, which reached 452 tokens per second. For other models, such as Qwen 3.8 27B, the new kernels deliver throughput that is roughly 1.4 to 2 times higher than the GB200 baseline at small batch sizes.
This integration is not a new concept for Meta, but it represents a more mature and direct path. The company previously used TPUs through PyTorch/XLA from 2020 to 2023, a bridge that Google now considers a predecessor to the new system. Reports from last year indicated that Meta was in talks to rent Google Cloud TPUs in 2026 before making purchases in 2027. A subsequent multi-year rental agreement suggests that the hardware is already being deployed at scale, although the exact location of these production units remains unclear.
The timing of this announcement coincides with a broader industry trend to reduce reliance on Nvidia’s software stack. Just one day prior, Amazon announced the open sourcing of TorchNeuron, a native PyTorch backend for its Trainium2 and Trainium3 chips. Both moves appear designed to give developers more freedom in choosing their hardware. SGLang-TorchTPU will continue to expand its model support, starting with DeepSeek V4.1 Flash and Reflection Beam, with plans for day-zero support for new open models by the first quarter of 2027.
