Linux's perf tool is essential for diagnosing performance bottlenecks, yet many developers treat it as a black box. This deep dive explains the internal architecture of perf, from event sources like hardware counters and tracepoints to the sampling mechanism that captures execution profiles. The author connects these fundamentals to real-world use cases, particularly in evaluating large language model inference performance on heterogeneous devices. Understanding perf's design helps engineers choose the right metrics, interpret profiling data correctly, and avoid common pitfalls. The post also touches on the distinction between CPU-bound and I/O-bound bottlenecks, which is crucial for optimizing modern workloads. For anyone serious about Linux performance engineering, grasping these internals transforms perf from a mysterious tool into a precise diagnostic instrument.
This post explains the principles behind Linux's perf tool, a critical utility for performance analysis. It covers how perf works under the hood, including sampling and hardware counters, and how it applies to evaluating AI model performance.