TPU

A Tensor Processing Unit is an application-specific integrated circuit (ASIC) designed by Google to accelerate the mathematical operations that power neural networks and machine learning inference. Unlike general-purpose processors, a TPU contains specialized hardware for matrix multiplication and accumulation, the core operations of tensor computations. The first generation TPU, deployed in Google's data centers around 2016, delivered roughly 15 to 30 teraflops of performance depending on precision, a significant speedup over conventional CPUs and GPUs for certain workloads.

TPUs are built into Google's cloud infrastructure and sold through Google Cloud as a service; they are not sold as standalone chips to other manufacturers. Different generations have emerged: the TPU v2 and v3 increased memory bandwidth and floating-point performance, while later versions like TPU v4 and v5e introduced higher throughput and reduced cost per inference. Each revision targets specific bottlenecks in either training large models or serving predictions at scale. The chips operate at lower clock speeds than CPUs, typically 700 MHz to 1 GHz, but achieve throughput through massive parallelism.

A TPU excels at operations that fit its narrow instruction set: dense matrix operations, convolutions, and attention mechanisms common in transformers and convolutional networks. Its memory hierarchy, with on-chip SRAM and lower-latency interconnects to external memory, reduces the data movement penalty that hampers GPU performance on certain problems. However, TPUs are relatively inflexible; code must be compiled to their instruction set, and workloads that deviate from dense tensor operations may see no advantage or even degradation compared to a general-purpose GPU.

Where TPUs sit in the market

TPUs compete with NVIDIA GPUs and other AI accelerators in cloud machine learning platforms. GPU adoption remains broader because GPUs support a wider range of algorithms and frameworks run on diverse hardware, while TPU is tied to Google Cloud and optimized primarily for TensorFlow. TPUs have captured significant share in Google's own inference workloads, including search ranking, translation, and recommendation systems, where the economics of custom silicon justify development cost.

For industrial users, the choice between TPU and GPU depends on workload characteristics and infrastructure lock-in tolerance. TPUs offer lower cost per inference on appropriate workloads run continuously in Google Cloud; GPUs offer portability and a richer ecosystem. The term TPU has become generic in industry discussion of AI accelerators, though it refers specifically to Google's line; competitors like AWS (Trainium, Inferentia) and others have developed their own custom silicon with similar design goals.