neural processing unit

A neural processing unit (NPU) is a processor core or standalone chip optimized to perform the matrix multiplications and tensor operations that machine learning models require. Unlike a general-purpose CPU or GPU, an NPU is hardwired with data paths, memory hierarchies, and instruction sets tuned specifically to inference and training workloads, trading flexibility for raw throughput on those narrow, repetitive tasks.

NPUs appear in two main forms on the factory floor. Integrated NPUs sit alongside CPU and GPU cores in system-on-chip designs, found in edge devices like industrial cameras, robotics controllers, and embedded vision systems where latency and power matter more than peak performance. Discrete NPU cards slot into servers for datacenter inference, offering 50 to 200 trillion operations per second (TOPS) depending on architecture and precision. Most industrial NPUs operate on 8-bit or 16-bit fixed-point integers rather than 32-bit floats, which reduces memory bandwidth and accelerates computation without significant accuracy loss on trained models.

Where NPUs fit in the workflow

A factory deploying computer vision for quality inspection might run model training on a datacenter GPU, then export the optimized model to an edge NPU in a line camera or edge compute module. The NPU executes inference at 5 to 100 frames per second while consuming a few watts, whereas a full GPU would be overkill and generate heat in a tight enclosure. Automotive suppliers use NPUs in autonomous systems; packaging lines use them for defect detection; semiconductor fabs use them for wafer inspection.

Common pitfalls emerge when workers assume an NPU works like a GPU with simpler programming. Model quantization, layer fusion, and memory layout optimization become critical; a model that runs fine on FP32 may produce bad results on INT8 without retraining. Thermal throttling is rare but can occur in dense server racks. Supply chain risk is real: major NPU makers have limited capacity, and edge NPU availability often lags datacenter chips by six months to a year.

The term "neural processing unit" was formalized around 2017 as the market separated specialized AI accelerators from traditional GPUs. Competitors include Tensor Processing Units (TPUs, Google), dedicated accelerators from Qualcomm, MediaTek, and SambaNova, and field-programmable gate arrays (FPGAs) configured for inference. The category overlaps with "AI accelerator" and "inference engine," but NPU typically implies a fixed-function silicon rather than reconfigurable hardware.