<div class="post-toc-block float-default with-border"> 
						<div class="post-toc-header">Table of Contents</div>
						<nav id="md-post-toc" class="md-post-toc"></nav>
						</div><p>Kubernetes is an incredibly complex system that is used to manage and orchestrate containerized applications. It provides a way to run and manage your applications across multiple machines, abstracting away the underlying infrastructure complexities. With Kubernetes, you can easily deploy, scale, and update your applications, ensuring high availability, fault tolerance, and efficient resource utilization.</p>
<p>With so many moving parts, it can be difficult to pinpoint performance issues and troubleshoot problems when they arise. However, eBPF (extended Berkeley Packet Filter) has emerged as a powerful tool that can be used to trace system events in real-time and provide detailed insights into system behaviour. In this blog post, we will explore how eBPF can be used to Go tracing function arguments in a Kubernetes environment.</p>
<h2>What is eBPF?</h2>
<p><a href="_wp_link_placeholder" data-wplink-edit="true">eBPF</a> (extended Berkeley Packet Filter) is a revolutionary technology that allows you to safely and efficiently extend the capabilities of the Linux kernel without modifying its source code. It provides a programmable and highly efficient way to analyze and manipulate network packets, system calls, and various kernel events. With eBPF, you can write small, sandboxed programs that run directly within the kernel, enabling powerful networking, monitoring, and security applications. It has gained significant popularity in the Kubernetes ecosystem, empowering developers to build advanced networking and observability solutions for containerized environments.</p>
<h2><strong>Why Go Tracing Function Arguments?</strong></h2>
<p>In a Kubernetes environment, applications are often written in Go, a popular programming language for building containerized applications. When performance issues arise, it can be challenging to pinpoint the root cause. By tracing Go function arguments, we can gain valuable insights into the application&#8217;s behaviour and identify areas where improvements can be made.</p>
<h3><a href="https://keploy.io/blog/community/a-guide-for-observing-go-process-with-ebpf">Go Tracing</a> Function Arguments with eBPF</h3>
<p>eBPF is a powerful tool that can be used to trace system events in real-time. It allows us to insert probes into the kernel or user-space code and capture data about the system&#8217;s behavior.</p>
<p>First, make sure you have the bcc tool installed on your system. You can refer to the bcc documentation for installation instructions: <a href="https://github.com/iovisor/bcc"><strong><a href="https://github.com/iovisor/bcc">https://github.com/iovisor/bcc</a></strong></a></p>
<p>Let&#8217;s get started with the code, in the below example we are writing a code to add the number passed in the arguments.</p>
<pre><code class="language-go">//main.go
package main

import (
    "fmt"
)

func add(a, b int) int {
    return a + b
}

func main() {
    result := add(2, 3)
    fmt.Println("Result:", result)
}</code></pre>
<p>Now it&#8217;s time to write our eBPF program:</p>
<pre><code class="language-c">//trace_args.c
#include &lt;uapi/linux/ptrace.h&gt;

BPF_PERF_OUTPUT(events);

int trace_func_args(struct pt_regs *ctx) {
    int arg1 = PT_REGS_PARM1(ctx);
    int arg2 = PT_REGS_PARM2(ctx);

    bpf_trace_printk("arg1: %d, arg2: %d\\n", arg1, arg2);

    return 0;
}</code></pre>
<p>We are specifying the function we want to trace and capture the function arguments using the <code>arg1</code> and <code>arg2</code> variables. Save the above code to a file called <code>trace_args.c</code>.<br />
Compile the eBPF program using the bcc tool:</p>
<pre><code class="language-powershell">clang -O2 -target bpf -c trace_args.c -o trace_args.o</code></pre>
<p>Once we have written our eBPF program, let&#8217;s modify the Go code to load and attach the eBPF program:</p>
<pre><code class="language-go">//main.go
package main

import (
    "fmt"
    "os"
    "os/signal"
    "syscall"
    "unsafe"

    "golang.org/x/sys/unix"
)

const (
    perfEventTypeTracepoint = 10
    tracepointCategory      = "go"

    bpfProgramPath = "./trace_args.o"
)

func loadBPFProgram() int {
    fd, err := unix.BPF(unix.BPF_PROG_LOAD, bpfProgramPath)
    if err != nil {
        fmt.Fprintf(os.Stderr, "Failed to load BPF program: %v\n", err)
        return -1
    }
    return fd
}

func attachTracepoint(fd int, tracepoint string) int {
    attr := &amp;unix.PerfEventAttr{
        Type:   perfEventTypeTracepoint,
        Config: uint64(fd),
    }
    attr.SetSampleFreq(1)
    attr.SetWakeupEvents(1)

    tpName := fmt.Sprintf("%s:%s", tracepointCategory, tracepoint)
    cTpName := unix.ByteSliceFromString(tpName)
    attr.SetTracepoint(unix.BytePtrFromString(tpName))

    _, _, errno := unix.Syscall6(
        unix.SYS_PERF_EVENT_OPEN,
        uintptr(unsafe.Pointer(attr)),
        unix.Getpid(),
        -1,
        -1,
        unix.PERF_FLAG_FD_CLOEXEC,
    )
    if errno != 0 {
        fmt.Fprintf(os.Stderr, "Failed to attach to tracepoint %s: %v\n", tracepoint, errno)
        return -1
    }
    return 0
}

func main() {
    fd := loadBPFProgram()
    if fd &lt; 0 {
        os.Exit(1)
    }
    defer unix.Close(fd)

    if attachTracepoint(fd, "trace_func_args") &lt; 0 {
        os.Exit(1)
    }

    signalChan := make(chan os.Signal, 1)
    signal.Notify(signalChan, syscall.SIGINT, syscall.SIGTERM)
    &lt;-signalChan

    fmt.Println("Exiting...")
}</code></pre>
<p>With our eBPF program running, we can now send requests to our application and capture the function arguments in <code>real-time</code>. This allows us to gain valuable insights into our application&#8217;s behaviour and identify areas where improvements can be made.</p>
<p>Now, let&#8217;s compile and run the modified Go code:</p>
<pre><code class="language-powershell">go build -o main main.go
sudo ./main</code></pre>
<p>When you run the code, it will trace the <code>add</code> function arguments using the eBPF program and print them. Press Ctrl+C to stop the program.</p>
<p>The output will be similar to:</p>
<pre><code class="language-plaintext">arg1: 2, arg2: 3</code></pre>
<h2>Conclusion</h2>
<p>eBPF is a powerful tool that can be used to trace system events in <code>real-time</code> and provide detailed insights into system behaviour. By tracing Go function arguments in a Kubernetes environment, we can gain valuable insights into our application&#8217;s behaviour and identify areas where improvements can be made. We can improve the performance and reliability of our applications and ensure that they are running at their best, with the power of eBPF.</p>

         
        
                    

                    
                    <div class="pp-multiple-authors-boxes-wrapper pp-multiple-authors-wrapper pp-multiple-authors-layout-boxed multiple-authors-target-the-content box-post-id-16 box-instance-id-1 ppma_boxes_16"
                    data-post_id="16"
                    data-instance_id="1"
                    data-additional_class="pp-multiple-authors-layout-boxed.multiple-authors-target-the-content"
                    data-original_class="pp-multiple-authors-boxes-wrapper pp-multiple-authors-wrapper box-post-id-16 box-instance-id-1">
                                                                                    <h2 class="widget-title box-header-title">Author</h2>
                                                                            <span class="ppma-layout-prefix"></span>
                        <div class="ppma-author-category-wrap">
                                                                                                <span class="ppma-category-group ppma-category-group-">
                                                                                                                        <ul class="pp-multiple-authors-boxes-ul">
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <li class="pp-multiple-authors-boxes-li author_index_0 author_animesh has-avatar">
                                                                                                                                                                                                                                            <div class="pp-author-boxes-avatar">
                                                                                                                                            <img alt='Animesh Pathak' src='https://wp.keploy.io/wp-content/uploads/2024/09/WhatsApp-Image-2024-08-06-at-3.17.14-PM.jpeg' srcset='https://wp.keploy.io/wp-content/uploads/2024/09/WhatsApp-Image-2024-08-06-at-3.17.14-PM.jpeg' class='multiple_authors_guest_author_avatar avatar' height='80' width='80'/>                                                                                                                                    </div>
                                                            
                                                            <div class="pp-author-boxes-avatar-details">
                                                                                                                                                                                                <div class="pp-author-boxes-name multiple-authors-name">
                                                                        <a href="https://wp.keploy.io/author/animesh/" rel="author" title="Animesh Pathak" class="author url fn">Animesh Pathak</a> 
                                                                    </div>
                                                                                                                                                                                                                                                                    <p class="pp-author-boxes-description multiple-authors-description">
                                                                        Animesh is a founding DevRel Engineer at Keploy, focusing on technical content, demos, documentation, and community engagement. He has a B.Tech in Computer Science from KIIT and expertise in AI, cloud computing, and Kubernetes. Animesh is also an active contributor and leader in open-source and student communities.                                                                    </p>
                                                                                                                                <a class="ppma-author-user_url-profile-data ppma-author-field-meta ppma-author-field-type-url" aria-label="Website" href="https://linkedin.com/in/sonichigo" rel="dofollow" target="_blank"><span class="dashicons dashicons-admin-links"></span> </a>
                                                                                                                                
                                                                                                                            </div>
                                                                                                                                                                                                                                                                                                                                                                                                             </li>
                                                                                                                                                                                                                                    </ul>
                                                                            </span>
                                                                                    </div>
                    <span class="ppma-layout-suffix"></span>
                    </div>
                    
                    
                
                                <style>
                .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-author-boxes-avatar img { width: 80px !important; height: 80px !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-author-boxes-avatar img { border-style: none !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-author-boxes-avatar img { border-radius: 50% !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-author-boxes-meta a { background-color: #655997 !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-author-boxes-meta a { color: #ffffff !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-author-boxes-meta a:hover { color: #ffffff !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .ppma-author-user_email-profile-data { background-color: #655997 !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .ppma-author-user_email-profile-data { border-radius: 100% !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .ppma-author-user_email-profile-data { color: #ffffff !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .ppma-author-user_url-profile-data { background-color: #655997 !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .ppma-author-user_url-profile-data { border-radius: 100% !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .ppma-author-user_url-profile-data { color: #ffffff !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-author-boxes-recent-posts-title { border-bottom-style: dotted !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-multiple-authors-boxes-li { border-style: solid !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-multiple-authors-boxes-li { border-width: 1px !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-multiple-authors-boxes-li { border-color: #999 !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-multiple-authors-boxes-li { color: #3c434a !important; }             </style>

Go Tracing Function Arguments in Production

Tracking Multiple Requests Over a Single Connection with eBPF

Capturing Multiple Requests on the Same Connection with eBPF

Executing EBPF in Github Actions

eBPF for TLS Traffic Tracing: Secure & Efficient Observability

BitBucket Self-Hosting : Running ebpf/Privileged programs

eBPF, Service Mesh and Sidecar

Decoding HTTP/2 Traffic is Hard, but eBPF can help

Building Keploy.io, an EBPF based open source framework to generate test cases and data stubs from API calls.

Neha Gupta

Cloud technology veteran and probably the youngest (globally) to have completed all 5 AWS certifications (including Solutions architect Professional and Dev­ops engineer professional)

Go Tracing Function Arguments In Production

More Stories