Analyzing container performance
Understanding, monitoring, profiling, and tracing Linux containers
Everybody is dockerizing, containerizing, and productionizing their applications. But just like any other abstraction, containers can make performance monitoring and analysis trickier. Because they are unable to separate container resource utilization from the host or fail to account for the container’s isolated file system view and networking namespace, many performance tools will lie to you.
In this three-hour hands-on training, Sasha Goldshtein teaches you how to monitor container resource utilization using standard Linux tools and open source frameworks. To understand your container’s CPU utilization and find bottlenecks, you’ll experiment with profiling and container CPU throttling using Linux perf, the versatile multitool for performance investigations developed as part of the kernel tree. You'll also get experience using cutting-edge tools based on BPF (the Berkeley Packet Filter) to optimize and monitor other kinds of resource consumption in your containers, tracing container disk accesses and network requests between containers. All the labs for the course are on GitHub and the environment is fully reproducible, so you can continue working on the labs in your own environment after the course is finished.
What you'll learn-and how you can apply it
By the end of this live online course, you’ll understand:
- How container isolation works and why it makes performance monitoring trickier
- Which container-level resources need to be monitored and analyzed
- How perf collects performance information and stack samples
- Why BPF offers a superior approach for tracing and performance investigation
And you’ll be able to:
- Apply a methodological approach for identifying container resource utilization
- Generate stack reports of container CPU usage
- Translate and propagate debug information from containers to the host
- Visualize and explore stack traces using flame graphs
- Trace and aggregate disk I/O, network, and database activity from containers
This training course is for you because...
- You're a Linux system administrator (SRE, production engineer, etc.) who needs to monitor container performance and pinpoint the root cause of performance issues.
- You're a Linux application developer who wants to identify bottlenecks in your application when it runs in a container.
- Experience with Linux system administration (e.g., the shell, pipes, very basic scripting)
- A basic understanding of operating systems concepts (e.g., processes, threads, scheduling, and memory)
- Familiarity with a container runtime such as Docker and the programming language used by your preferred Linux application
Materials or downloads needed in advance:
A machine with an up-to-date web browser (During the live training, you’ll get access to a virtual lab environment.)
"The USE Method" (article)
"CPU Flame Graphs" (article)
Assignment to be completed before the course:
Run stress(1) in a Docker container with various flags (e.g., CPU hog, I/O hog, and memory hog) and try to identify the container’s resource utilization from the host. You can use the progrium/stress Docker image.
About your instructor
Sasha Goldshtein is the CTO of Sela Group, a Microsoft MVP, Pluralsight author, and international consultant and trainer. Sasha is the author of two books and multiple online courses, and a prolific blogger. He is also an active open source contributor to projects focused on system diagnostics, performance monitoring, and tracing -- across multiple operating systems and runtimes. Sasha authored and delivered training courses on Linux performance optimization, event tracing, production debugging, mobile application development, and modern C++. Between his consulting engagements, Sasha speaks at international conferences world-wide.
The timeframes are only estimates and may vary according to how the class is progressing
Introduction (5 minutes)
- Lecture: Course overview; the lab environment
Container internals (10 minutes)
- Lecture: Overview of container technologies; who uses containers and what the key scenarios are; control groups—CPU, memory, and block I/O; namespaces—pid namespace, mount namespace, and network namespace; container resource caps; container-specific challenges with traditional performance monitoring tools
Resource monitoring and utilization (35 minutes)
- Lecture: The USE method for performance analysis; USE checklist for Linux systems; container monitoring tools—docker stats, docker top, systemd-cgtop, htop; getting monitoring data from within the container (nsenter, docker exec)
- Hands-on exercises: Monitor multiple CPU-hungry containers with resource caps; monitor multiple I/O-hungry containers
Break (10 minutes)
CPU profiling (50 minutes)
- Lecture: Running perf on the host (with -G and symbol mapping); profiling JIT-compiled runtimes (sharing perf maps from the container to the host); punning perf in the container in nonprivileged mode; identifying CPU throttling; generating flame graphs of CPU stacks; recording system tracepoints with perf (block I/O tracing and context switches)
- Hands-on exercises: Use perf to profile a C++ application in a container; use perf to profile a Java application with perf-map-agent
Break (10 minutes)
Introduction to BPF (the Berkeley Packet Filter) (10 minutes)
- Lecture: Challenges with perf-based workflows (lots of data copied to userspace); Ye Olde BPF; modern BPF—XDP, seccomp, event tracing, and aggregation
BCC tools (35 minutes)
- Lecture: The BCC tool collection (how to install and run, required kernel versions, etc.); block I/O and file I/O latency analysis and tracing; blocked time analysis (off-CPU); ad hoc tracing with trace, argdist, funccount, funclatency, and stackcount
- Hands-on exercises: Advanced container tracing—trace container file I/O and latency; trace container networking (including container-to-container)
Wrap-up and Q&A (15 minutes)
- Lecture: Overview of turnkey container monitoring solutions (SysDig, cAdvisor, etc.); further reading on container internals, deployment, and networking