In-depth tracing often has the desirous side effect of having the developer or tester spot performance bottlenecks as well as debug systems-level latencies and issues. Linux has a plethora of frameworks and tools available for tracing, both at user space and at the level of the kernel; some of the more relevant ones are mentioned here:
- User space: ltrace(1) (trace library APIs), strace(1) (trace system calls; also try doing sudo perf trace), LTTng-ust, uprobes.
- Kernel space: LTTng, ftrace (plus several frontends such as tracecmd(1), kernelshark GUIm), Kprobes—(including Jprobes—up to Kernel Version 4.14), Kretprobes; SystemTaprm) eBPF.
