Ebpf For Tls Traffic Tracing: Secure & Efficient Observability

Tracing TLS (Transport Layer Security) traffic is crucial for modern observability systems. It helps monitor encrypted communication, diagnose issues, and optimize application performance using insights from network traffic telemetry.
However, traditional methods like TLS proxying and packet capturing often come with significant performance overheads and security risks. They are not always the ideal solution, particularly for high-performance or security-sensitive environments.

This is where eBPF (Extended Berkeley Packet Filter) steps in. eBPF is a revolutionary technology that enables seamless TLS traffic tracing directly within the Linux kernel, minimizing performance impact and enhancing security. This article explores how eBPF works, its implementation, and its benefits for developers and DevOps teams.

Challenges with Traditional TLS Traffic Tracing

Tracing TLS traffic has always been an integral part of observability systems. Traditionally, these systems have relied on techniques such as:

• TLS Proxying

• Man-in-the-Middle (MITM) Proxies

• Packet Capturing and Decryption

• Logging TLS Handshakes

Unfortunately, these methods come with major drawbacks that prevent any one of them from becoming the dominant approach:

1. Performance Overhead: Techniques like TLS proxying and packet capturing can significantly tax the CPU due to the effort required for decryption and packet analysis.

2. Security Risks: TLS proxies often require creating and maintaining fake certificates or adding custom certificate authorities to the client’s system. This can lead to serious security vulnerabilities.

The eBPF Advantage

eBPF addresses these challenges by offering a state-of-the-art method for TLS traffic tracing. Here’s why it stands out:

• Minimal Performance Impact: eBPF operates directly within the Linux kernel, avoiding the CPU-intensive operations of traditional methods.

• Enhanced Security: It eliminates the need for fake certificates or risky man-in-the-middle techniques.

eBPF enables developers to attach hooks to system calls. For instance, a system call like sendto used for sending data over a UDP socket can be monitored. By attaching an eBPF program to sendto, you can capture critical information like the buffer, function descriptor, and buffer length as soon as the system call is invoked.
This makes TLS tracing seamless and secure, addressing the limitations of older methods. The next obvious question is how eBPF is able to achieve all of this? How does it work under the hood.

Another practical eBPF use case is redirecting DNS traffic in Docker containers using TC BPF programs – useful for testing and observability in containerized environments.

How does eBPF work?

eBPF programs are typically written in C and are then compiled into a intermediate bytecode representation. As mentioned above, these programs can be attached to various system calls, allowing them to be called when those calls are executed.
For storing data in eBPF, we first need to create special data structures called BPF maps to store the data. They can be declared as follows:

Therefore, to trace TLS data specifically, these hooks can be attached to the following system calls: sendto, recvfrom, socket, connect, accept, SSL_read, and SSL_write. These calls contain the actual buffer, its size, and the function descriptor before the request gets encrypted and after the response gets decrypted. Thus, they don’t need the session key to decrypt the responses coming from the server.

Sample application attaching hooks to SSL_read and SSL_write functions

After capturing the data in the kernel space, it needs to be sent to the user space. To do this, we write user space programs, in a variety of languages(Golang is more popular nowadays because of Cilium) where the eBPF hooks are attached to the system calls again, and the data is collected from the kernel space and shown to the user.

Sending data from the kernel space to the user space

Attaching hooks on the user side.

Bringing it all together, we can compile this into a complete program which will look something like this:

And to add these probes on the user side, we can use the Cilium library to write Go code as given below:

Now, to run this code, you need to compile the bpf files and generate the go structs and then run the code, for this, we can create a simple shell script as given below:

Now when you run this script, it will start tracing all the TLS calls that you make from your machine. The data for them will be shown on the same terminal window.

Outputting the data to the user.

It is because of these advantages that eBPF is gaining popularity by every passing day. But eBPF too, has some challenges that cannot be ignored.

1. For starters, writing and debugging eBPF programs can be a challenging task, especially for programmers who are not familiar with low-level programming.

2. Since eBPF code gets loaded onto the kernel, it needs to be bug-free, as any vulnerabilities in the eBPF code can lead to kernel crashes or security exploits.

3. Another constraint with eBPF programs is their kernel version dependency. They support different features across different kernel versions so porting eBPF code can be tough and can lead to compatibility issues.

4. Since eBPF is a relatively new technology the data structures used to write these programs(BPF maps) have certain limitations. For instance, their size cannot be changed dynamically and some map types do not support concurrent read and write operations.

Limitations Of This Approach

Not everything works all the time and no matter how robust, this approach has its limitations as well. Some of the limitations are as follows:

• Kernel Dependency: eBPF relies on specific kernel features that may not be present in older kernels. This limits its use to environments with up-to-date kernels, making it unsuitable for legacy systems.

• Kernel Stability: Bugs or changes in the kernel could impact the stability and performance of eBPF programs, requiring close tracking of kernel updates and patches.

• User-Space Encryption: If encryption and decryption occur entirely in user-space libraries (like OpenSSL or BoringSSL), eBPF might not capture all necessary data, as it primarily hooks into system calls and kernel functions.

• Sandbox Restrictions: eBPF programs run in a sandboxed environment with limited capabilities. While this enhances security, it also restricts what eBPF programs can do, requiring careful design to achieve the desired tracing functionality.

How Keploy uses eBPF for TLS Handling?

Keploy, a modern API testing platform, which leverages eBPF (Extended Berkeley Packet Filter) to efficiently trace TLS (Transport Layer Security) traffic. This approach addresses the challenges of monitoring encrypted data streams without compromising performance or security.

Key Benefits:

1. Seamless Traffic Tracing: Keploy uses eBPF to attach hooks to system calls such as SSL_read and SSL_write. These hooks capture encrypted and decrypted data before it reaches the application layer, enabling detailed traffic inspection.

2. Minimal Overhead: By operating directly in the Linux kernel, eBPF allows Keploy to monitor TLS traffic with negligible impact on system performance, making it suitable for high-load environments.

3. Improved Observability: The collected data helps Keploy deliver actionable insights for API testing, debugging, and observability. It ensures high test coverage and robust monitoring for applications with encrypted communications.

Conclusion

eBPF offers a mixed bag of power and limitations, but no one can really deny its potential, and as the technology gets more mature, it is certainly bound to leave a big impact on the tech world. As eBPF continues to evolve, it holds immense promise for shaping the future of observability, security, and performance monitoring.
While it does come with its own set of challenges, such as kernel dependencies and a learning curve, the benefits of eBPF in modern observability are undeniable. Its seamless integration with the Linux kernel, minimal overhead, and robust tracing capabilities make it an invaluable tool for debugging, optimizing, and ensuring the reliability of secure applications. Now is the perfect time to explore this groundbreaking technology and harness its potential for your systems.

FAQs

What is eBPF?

eBPF (Extended Berkeley Packet Filter) is a technology that allows developers to run sandboxed programs in the Linux kernel without modifying its source code. It’s widely used for networking, observability, and security purposes.

Why is tracing TLS traffic important?

Tracing TLS traffic is crucial for debugging, performance optimization, and understanding the behavior of secure applications. It helps identify bottlenecks, diagnose issues, and ensure compliance with security policies.

How does eBPF enable TLS traffic tracing?

eBPF attaches hooks to system calls like sendto, recvfrom, SSL_read, and SSL_write. These hooks capture data before encryption or after decryption, allowing seamless tracing without needing session keys or modifying certificates.

What are the advantages of using eBPF over traditional TLS tracing methods?

• Minimal performance overhead on the CPU.

• No need for fake certificates or proxy configurations.

• Enhanced security with a kernel-native approach.

• Compatibility with modern observability tools.

Are there any limitations to using eBPF?

Yes, eBPF has some limitations, including:

• Kernel dependency: It requires a modern Linux kernel.

• Steep learning curve: Writing and debugging eBPF programs can be challenging.

• Restricted capabilities: eBPF programs run in a sandboxed environment.

What programming languages can I use with eBPF?

eBPF programs are typically written in C but can be integrated with user-space programs written in languages like Golang, Python, and Rust.

How can I get started with eBPF?

Start by learning the basics of Linux system calls and kernel development. Use tools like BPFTrace or frameworks like Cilium for easier eBPF programming. Tutorials and eBPF books are also great resources.

What are some practical use cases of eBPF in observability?

• Tracing application network traffic.

• Debugging performance bottlenecks.

• Enhancing security through real-time monitoring.

• Capturing and analyzing TLS data without decryption keys.