Choosing Hardware for On-Device Inference: GPU, CPU, or NPU?
Picking the right processor for on-device machine learning affects latency, battery life, cost, and developer effort. This guide compares GPUs, CPUs, and NPUs across common workloads and gives concrete steps to deploy efficient models on edge devices.
- Quick summary of when to choose GPU, CPU, or NPU.
- Practical performance comparison and memory strategies (quantization, pruning).
- Deployment checklist, pitfalls, and a short FAQ for production readiness.
Quick answer
For throughput-heavy, parallelizable models (vision, large CNNs, transformer batching) choose a GPU; for general-purpose, latency-tolerant, or legacy code prefer a CPU; for strict power/latency and small-medium DNNs choose an NPU or edge accelerator. Use quantization and model compression to fit limits and pick the hardware that matches your latency, power, and integration constraints.
Pick GPU: when and why
GPUs excel at highly parallel workloads and batched inference. They offer massive matrix multiply throughput and are a natural fit for large CNNs, transformer inference with batching, and high-resolution image processing.
- Best for: high-throughput servers, desktop inference, on-device devices with strong thermal and power budgets (game consoles, some edge servers).
- Strengths: parallel FLOPS, mature software stack (CUDA, cuDNN), good mixed-precision support.
- Weaknesses: higher power draw, less efficient at tiny batch sizes or single-request real-time tasks, driver complexity on edge.
| Workload | When GPU is preferred |
|---|---|
| Image classification / detection | High-resolution or high-throughput streams |
| Batch transformer inference | Large batch sizes or sequence parallelism |
| Parallel preprocessing | Heavy augmentations or simultaneous pipelines |
Pick CPU: when and why
CPUs are the most flexible and easiest to integrate. They’re ideal for control-plane tasks, small models, or when hardware ubiquity and minimal driver dependencies matter.
- Best for: low-compute models, control logic, non-batched real-time inference, or devices lacking accelerators.
- Strengths: wide availability, single-request latency consistency, simpler deployment, good for integer-quantized models.
- Weaknesses: limited parallel FLOPS compared to GPUs/NPUs, inefficient for large matrix-heavy workloads.
Pick NPU/Edge accelerator: when and why
NPUs (or dedicated edge accelerators) are specialized for neural ops and optimized for throughput per watt. They often beat CPUs and GPUs on energy efficiency for common DNNs and are preferred for battery-powered devices requiring low latency.
- Best for: always-on inference, mobile vision tasks, voice wake-word, and other low-power, low-latency needs.
- Strengths: power efficiency, deterministic latency, hardware operators for quantized ops.
- Weaknesses: limited flexibility, vendor fragmentation, potential need to adapt models to supported ops or runtimes.
Compare performance: throughput, latency, and batch size
Performance must be evaluated across three axes: throughput (samples/sec), tail and median latency (ms), and batch size efficiency. Choose the hardware that matches your target point on these axes.
- Throughput: GPUs scale well with batch size; NPUs often offer best throughput-per-watt for supported ops.
- Latency: CPUs and NPUs provide predictable single-request latency; GPUs may have higher latency at batch size 1 due to kernel overhead.
- Batch size: if you can batch requests, GPUs often win; for single-shot real-time, NPUs or CPUs often win.
| Metric | GPU | CPU | NPU |
|---|---|---|---|
| Single-request latency | Medium | Low–Medium | Low |
| Batched throughput | High | Low | Medium–High |
| Power efficiency | Low | Medium | High |
Manage memory: model size, quantization, and pruning
Memory constraints often determine feasibility. Address them with model architecture choices, quantization, and pruning while validating accuracy trade-offs.
- Model size: prefer smaller architectures (MobileNet, EfficientNet-lite, TinyBERT) when memory or compute is limited.
- Quantization: 8-bit post-training quantization is usually the best first step; mixed precision (FP16/INT8) can cut memory and speed up inference on supported hardware.
- Pruning & distillation: structured pruning reduces compute and memory; knowledge distillation transfers accuracy to smaller models.
| Technique | Typical memory reduction | Notes |
|---|---|---|
| INT8 quantization | ~4x model weight size | Minor accuracy drop for many vision models |
| FP16 | ~2x | Requires hardware support for half precision |
| Structured pruning | 20–60% | Best when combined with fine-tuning |
Deploy and integrate: drivers, frameworks, and tooling
Deployment success hinges on matching your model and hardware to supported runtimes and drivers. Plan for toolchain compatibility early.
- Frameworks: TensorFlow Lite, ONNX Runtime, PyTorch Mobile, and vendor SDKs (e.g., NNAPI, Core ML, Arm Compute Library) cover most targets.
- Drivers & runtimes: verify NPU drivers and firmware for your target; GPU drivers (CUDA, ROCm) require correct versions for kernels and libraries.
- Tooling: use model conversion and optimization tools (TFLite converter, ONNX optimization passes, TensorRT, OpenVINO) to tune for hardware.
Example deployment flow:
- Train/export model to ONNX or SavedModel.
- Apply quantization/pruning and validate accuracy.
- Convert to target runtime (TFLite, TensorRT, NNAPI delegate) and test on device.
- Profile and iterate on batch size, threads, and scheduling.
Common pitfalls and how to avoid them
- Assuming all ops are supported: check operator coverage for NPUs; implement or replace unsupported ops with supported equivalents.
- Ignoring memory fragmentation: pre-allocate buffers or use memory pools to avoid runtime OOMs.
- Testing only on emulator: always validate on target hardware — performance and driver behaviors differ.
- Over-quantizing without validation: run representative datasets to measure accuracy loss after quantization.
- Neglecting power profile tests: measure energy per inference on battery-powered devices to ensure real-world viability.
Implementation checklist
- Define latency, throughput, power, and memory targets.
- Choose candidate hardware (CPU/GPU/NPU) based on target metrics.
- Select a compact model or apply compression (quantize/prune/distill).
- Convert model to target runtime and validate operator support.
- Profile on target device: latency, throughput, memory, power.
- Optimize batch size, threading, and memory allocation.
- Perform A/B tests to verify no critical accuracy regressions.
- Establish CI for model conversion and performance regression checks.
FAQ
- Q: Should I always quantize to INT8?
- A: Start with INT8 post-training quantization for best size/perf trade-off, but validate accuracy. Use PTQ or QAT if accuracy drops are unacceptable.
- Q: How do I decide between GPU and NPU for mobile?
- A: Use an NPU when power and deterministic latency matter; use GPU when you need higher throughput with larger models and the device supports efficient GPU compute.
- Q: Can all models run on NPUs?
- A: Not always. Many NPUs support common convolutional and linear ops; complex or custom ops may need fallback to CPU or model rework.
- Q: How much batching should I use?
- A: If low-latency single requests are required, use batch size 1. For server/edge aggregation, choose a batch size that maximizes throughput without violating latency SLOs.
- Q: Which profiling tools are recommended?
- A: Use vendor tools (NVIDIA Nsight, Android Systrace/Perfetto, Apple Instruments), framework profilers, and simple wall-clock/timeit measurements on target hardware.
