Abstract:Modern datacenters increasingly rely on low-power, single-slot inference accelerators to balance performance, energy efficiency, and rack density constraints. The NVIDIA T4 GPU has become widely deployed due to strong performance per watt and mature software support. Its successor, the NVIDIA L4 GPU, introduces improvements in Tensor Core throughput, cache capacity, memory bandwidth, and parallel execution capability. However, limited empirical evidence quantifies the practical inference performance gap between these two generations under controlled and reproducible conditions.
This work introduces DEEP-GAP, a systematic evaluation extending the GDEV-AI methodology to GPU inference. Using identical configurations and workloads, we evaluate ResNet18, ResNet50, and ResNet101 across FP32, FP16, and INT8 precision modes using PyTorch and TensorRT.
Results show that reduced precision significantly improves performance, with INT8 achieving up to 58x throughput improvement over CPU baselines. L4 achieves up to 4.4x higher throughput than T4 while reaching peak efficiency at smaller batch sizes between 16 and 32, improving latency-throughput tradeoffs for latency-sensitive workloads. T4 remains competitive for large batch workloads where cost or power efficiency is important.
DEEP-GAP provides practical guidance for selecting precision modes, batch sizes, and GPU architectures for modern inference deployments.
| Comments: | 16 pages, 42 figures. Evaluation of inference performance on NVIDIA T4 and L4 GPUs across precision modes (FP32, FP16, INT8) |
| Subjects: | Performance (cs.PF); Hardware Architecture (cs.AR); Machine Learning (cs.LG) |
| ACM classes: | C.4 |
| Cite as: | arXiv:2604.14552 [cs.PF] |
| (or arXiv:2604.14552v1 [cs.PF] for this version) | |
| https://doi.org/10.48550/arXiv.2604.14552 arXiv-issued DOI via DataCite |
From: Kathiravan Palaniappan [view email]
[v1]
Thu, 16 Apr 2026 02:32:58 UTC (7,425 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。