<div class="post-toc-block float-default with-border"> 
						<div class="post-toc-header">Table of Contents</div>
						<nav id="md-post-toc" class="md-post-toc"></nav>
						</div><h2>Introduction</h2>
<p>In my journey as a software enthusiast, I&#8217;ve constantly sought innovative ways to unravel the intricacies of my Go applications. Recently, I stumbled upon a fascinating technology that has significantly transformed the way I observe and analyze Go processes.</p>
<p>In this blog post, We&#8217;ll go on a journey of observing Go processes with eBPF. Let&#8217;s first get to understand &quot;<a href="https://www.brendangregg.com/blog/2015-05-15/ebpf-one-small-step.html">what is eBPF?</a>&quot;</p>
<h2>what is eBPF?</h2>
<p>eBPF tracing is a technology that enables user a closer examination of how a kernel is behaving. This permits the attachment of lightweight &amp; low-level programs called eBPF programs to specific events within the kernel. Think of it as setting up unobtrusive &quot;listeners&quot; throughout the system, each reporting valuable information.</p>
<p><img decoding="async" src="https://www.ebpf.top/post/ebpf_go_translation/imgs/xdp-xconnect.png" alt="【译】eBPF 和 Go 经验初探 -阿里云开发者社区" /></p>
<p>There are various ways to write eBPF programs, <a href="https://github.com/iovisor/bcc">BCC</a>, <a href="https://github.com/iovisor/bpftrace">bpftrace</a> and a new development called <a href="https://github.com/libbpf/libbpf">libbpf</a>.</p>
<p>With eBPF programs, users can observe Go process at the kernel level using tracing. Sysadmins, DevOps engineers, SREs, etc find this insightful because it is makes troubleshooting faster. As previously, it used to take days to figure out issues that require this level of visibility with current troubleshooting techniques.</p>
<p>There are two types of tracing available &#8211; static (tracepoints, USDT probes) and dynamic (kprobes, uprobes).</p>
<h2>How to get started with GO and eBPF?</h2>
<p>If this works, you should be able to generate a Go trace from observed behaviour on a <a href="https://keploy.io/docs/server/linux/installation">Linux</a> machine. e.g. Is there a certain network path that shouldn’t be happening?</p>
<p>It is worth pointing out again that there are other ways to get this information, e.g. instrumentation, core dumps debugging, etc. <em>This is for those who can’t modify the application, but also can’t hinder performance.</em> It’s also useful for maintainers who want to do this without deploying code changes.</p>
<p>Go Lang doesn&#8217;t need to support eBPF because it operates differently from languages and systems that benefit from eBPF. Stacks such as NodeJS doesn&#8217;t support USDT probes <a href="https://github.com/nodejs/node/issues/44550#issuecomment-1239416388">nodejs/node#issue</a> .</p>
<p>Go Lang creates executable files and has a runtime that allows for introspection, in line with its design principles. This native observability simplifies the need for eBPF-like techniques for monitoring and tracing. The way Go Lang is built and integrated with the system gives it enough visibility and control without requiring the additional support of eBPF.</p>
<h2>How does Tracing with eBPF work?</h2>
<p>This section introduces tracing methods in eBPF and if they can be fit for this purpose. Let’s evaluate the different types of eBPF tracing methods available:</p>
<ol>
<li><strong>Tracepoints</strong> are eBPF probes attached to predefined kernel events, providing a way to observe fundamental system behaviours. Hence we can use them to trace kernel syscalls such as <a href="https://man7.org/linux/man-pages/man2/open.2.html">open(2)</a>, <a href="https://man7.org/linux/man-pages/man2/write.2.html">write(2)</a>, etc. You can learn by example &#8220;<a href="https://github.com/cilium/ebpf/blob/main/examples/tracepoint_in_go/main.go">Go_ebpf_tracepoint</a>&#8220;<img decoding="async" src="https://cdn.jsdelivr.net/gh/b0xt/sobyte-images1/2022/07/20/984e8f0d69084c12a6d639f28a5d7282.png" alt="Go with Tracepoint" /></li>
<li><strong>USDT Probes (Static)</strong> are like tracepoints but instead created by user-space developers for their code.</li>
<li><strong>Dynamic:- Kprobes</strong> allow us to attach probes to arbitrary kernel functions, while <strong>Uprobes</strong> do the same for user-space functions. A good sample project can be founded by &#8220;<a href="https://github.com/cilium/ebpf/">ebpf_Kprobe</a>&#8221; by Cilium.</li>
</ol>
<p>In Go processes, we can use USDT (User-Level Statically Defined Tracing) probes. These probes help us highlight specific parts of our application code.</p>
<p>This way, we can closely watch how our application runs and how it interacts with the system. This observation helps us understand how the program behaves and its connection to the system. First, we must register a USDT probe in the go code.</p>
<pre><code class="language-go">if err := unix.Uprobe(nil, "your_probe_name", "your_probe_description");
err != nil {
    fmt.Printf("Error registering probe: %v\n", err)
    return
}</code></pre>
<p>Now that we&#8217;ve modified the Go code with USDT probes, we will use eBPF to trace those probes. Here&#8217;s an example of how you might write eBPF code to achieve this:</p>
<pre><code class="language-c">#include &lt;uapi/linux/ptrace.h&gt;

BPF_PERF_OUTPUT(events);

int trace_usdt(struct pt_regs *ctx) {
    char str[80];
    bpf_usdt_readarg(1, ctx, &amp;str, sizeof(str));

    events.perf_submit(ctx, &amp;str, sizeof(str));

    return 0;
}</code></pre>
<p>In this eBPF program:</p>
<ul>
<li><code>BPF_PERF_OUTPUT(events);</code> declares an output event named <code>events</code> that will be used to send data to user space.</li>
<li><code>int trace_usdt(struct pt_regs *ctx)</code> is the probe handler function. It reads the argument from the USDT probe and submits it to the <code>events</code> output.</li>
<li><code>bpf_usdt_readarg(1, ctx, &amp;str, sizeof(str));</code> reads the first argument of the USDT probe and stores it in the <code>str</code> variable.</li>
<li><code>events.perf_submit(ctx, &amp;str, sizeof(str));</code> submits the data to the user space for further processing.</li>
</ul>
<h2>Gathering Insights and Optimizing</h2>
<p>By attaching probes into a Go application, we can collect crucial runtime information. This includes details like function calls, system calls, and events related to memory allocation. The data collected from probes are extremely helpful, such as we can track metrics like memory allocation patterns, and network interactions.<br />
And this only helps us optimize performance as well as in identifying bottlenecks and unexpected behaviours.</p>
<h2>Conclusion:</h2>
<p>After exploring Go processes with eBPF, I can&#8217;t help but be amazed at the transformative power it holds. This journey has highlighted the significance of a deeper understanding of application behaviour, one that eBPF facilitates with grace. eBPF is a tool that stands at the crossroads of innovation and performance, to provide better metrics for optimization strategy.<br />
eBPF simplifies code execution, untangles bottlenecks, and improves application efficiency, making it a great tool for software enthusiasts. To improve Go applications, it is important to embrace technologies like eBPF in software development.</p>

         
        
                    

                    
                    <div class="pp-multiple-authors-boxes-wrapper pp-multiple-authors-wrapper pp-multiple-authors-layout-boxed multiple-authors-target-the-content box-post-id-16 box-instance-id-1 ppma_boxes_16"
                    data-post_id="16"
                    data-instance_id="1"
                    data-additional_class="pp-multiple-authors-layout-boxed.multiple-authors-target-the-content"
                    data-original_class="pp-multiple-authors-boxes-wrapper pp-multiple-authors-wrapper box-post-id-16 box-instance-id-1">
                                                                                    <h2 class="widget-title box-header-title">Author</h2>
                                                                            <span class="ppma-layout-prefix"></span>
                        <div class="ppma-author-category-wrap">
                                                                                                <span class="ppma-category-group ppma-category-group-">
                                                                                                                        <ul class="pp-multiple-authors-boxes-ul">
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <li class="pp-multiple-authors-boxes-li author_index_0 author_animesh has-avatar">
                                                                                                                                                                                                                                            <div class="pp-author-boxes-avatar">
                                                                                                                                            <img alt='Animesh Pathak' src='https://wp.keploy.io/wp-content/uploads/2024/09/WhatsApp-Image-2024-08-06-at-3.17.14-PM.jpeg' srcset='https://wp.keploy.io/wp-content/uploads/2024/09/WhatsApp-Image-2024-08-06-at-3.17.14-PM.jpeg' class='multiple_authors_guest_author_avatar avatar' height='80' width='80'/>                                                                                                                                    </div>
                                                            
                                                            <div class="pp-author-boxes-avatar-details">
                                                                                                                                                                                                <div class="pp-author-boxes-name multiple-authors-name">
                                                                        <a href="https://wp.keploy.io/author/animesh/" rel="author" title="Animesh Pathak" class="author url fn">Animesh Pathak</a> 
                                                                    </div>
                                                                                                                                                                                                                                                                    <p class="pp-author-boxes-description multiple-authors-description">
                                                                        Animesh is a founding DevRel Engineer at Keploy, focusing on technical content, demos, documentation, and community engagement. He has a B.Tech in Computer Science from KIIT and expertise in AI, cloud computing, and Kubernetes. Animesh is also an active contributor and leader in open-source and student communities.                                                                    </p>
                                                                                                                                <a class="ppma-author-user_url-profile-data ppma-author-field-meta ppma-author-field-type-url" aria-label="Website" href="https://linkedin.com/in/sonichigo" rel="dofollow" target="_blank"><span class="dashicons dashicons-admin-links"></span> </a>
                                                                                                                                
                                                                                                                            </div>
                                                                                                                                                                                                                                                                                                                                                                                                             </li>
                                                                                                                                                                                                                                    </ul>
                                                                            </span>
                                                                                    </div>
                    <span class="ppma-layout-suffix"></span>
                    </div>
                    
                    
                
                                <style>
                .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-author-boxes-avatar img { width: 80px !important; height: 80px !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-author-boxes-avatar img { border-style: none !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-author-boxes-avatar img { border-radius: 50% !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-author-boxes-meta a { background-color: #655997 !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-author-boxes-meta a { color: #ffffff !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-author-boxes-meta a:hover { color: #ffffff !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .ppma-author-user_email-profile-data { background-color: #655997 !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .ppma-author-user_email-profile-data { border-radius: 100% !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .ppma-author-user_email-profile-data { color: #ffffff !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .ppma-author-user_url-profile-data { background-color: #655997 !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .ppma-author-user_url-profile-data { border-radius: 100% !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .ppma-author-user_url-profile-data { color: #ffffff !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-author-boxes-recent-posts-title { border-bottom-style: dotted !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-multiple-authors-boxes-li { border-style: solid !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-multiple-authors-boxes-li { border-width: 1px !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-multiple-authors-boxes-li { border-color: #999 !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-multiple-authors-boxes-li { color: #3c434a !important; }             </style>

A Guide for Observing Go Process with eBPF tracing

Agile Testing Best Practices for Faster, Higher‑Quality Releases

Integration of E2E Testing in a CI/CD Pipeline

Tracking Multiple Requests Over a Single Connection with eBPF

What is Unit Testing?

Capturing Multiple Requests on the Same Connection with eBPF

Executing EBPF in Github Actions

Alpha vs Beta Testing: Key Differences & Best Practices

Building Keploy.io, an EBPF based open source framework to generate test cases and data stubs from API calls.

Neha Gupta

Cloud technology veteran and probably the youngest (globally) to have completed all 5 AWS certifications (including Solutions architect Professional and Dev­ops engineer professional)

A Guide For Observing Go Process With Ebpf Tracing

More Stories