In my journey as a software enthusiast, I've constantly sought innovative ways to unravel the intricacies of my Go applications. Recently, I stumbled upon a fascinating technology that has significantly transformed the way I observe and analyze Go processes.
In this blog post, We'll go on a journey of observing Go processes with eBPF. Let's first get to understand "what is eBPF?"
eBPF - The Substance X of Kernel Observability
eBPF is a technology that enables user a closer examination of how a kernel is behaving. This permits the attachment of lightweight & low-level programs called eBPF programs to specific events within the kernel. Think of it as setting up unobtrusive "listeners" throughout the system, each reporting valuable information.
With eBPF programs, users can observe Go process at the kernel level using tracing. Sysadmins, DevOps engineers, SREs, etc find this insightful because it is makes troubleshooting faster. As previously, it used to take days to figure out issues that require this level of visibility with current troubleshooting techniques.
There are two types of tracing available - static (tracepoints, USDT probes) and dynamic (kprobes, uprobes).
A Go Union with eBPF
If this works, you should be able to generate a Go trace from observed behaviour on a Linux machine. e.g. Is there a certain network path that shouldn’t be happening?
It is worth pointing out again that there are other ways to get this information, e.g. instrumentation, core dumps debugging, etc. This is for those who can’t modify the application, but also can’t hinder performance. It’s also useful for maintainers who want to do this without deploying code changes.
Go Lang doesn't need to support eBPF because it operates differently from languages and systems that benefit from eBPF. Stacks such as NodeJS doesn't support USDT probes nodejs/node#issue .
Go Lang creates executable files and has a runtime that allows for introspection, in line with its design principles. This native observability simplifies the need for eBPF-like techniques for monitoring and tracing. The way Go Lang is built and integrated with the system gives it enough visibility and control without requiring the additional support of eBPF.
Tracing Methods in eBPF
This section introduces tracing methods in eBPF and if they can be fit for this purpose. Let’s evaluate the different types of eBPF tracing methods available:
Tracepoints are eBPF probes attached to predefined kernel events, providing a way to observe fundamental system behaviours. Hence we can use them to trace kernel syscalls such as open(2), write(2), etc. You can learn by example "Go_ebpf_tracepoint"
USDT Probes (Static) are like tracepoints but instead created by user-space developers for their code.
Dynamic:- Kprobes allow us to attach probes to arbitrary kernel functions, while Uprobes do the same for user-space functions. A good sample project can be founded by "ebpf_Kprobe" by Cilium.
In Go processes, we can use USDT (User-Level Statically Defined Tracing) probes. These probes help us highlight specific parts of our application code.
This way, we can closely watch how our application runs and how it interacts with the system. This observation helps us understand how the program behaves and its connection to the system. First, we must register a USDT probe in the go code.