<div class="post-toc-block float-default with-border"> 
						<div class="post-toc-header">Table of Contents</div>
						<nav id="md-post-toc" class="md-post-toc"></nav>
						</div><p>I&#8217;ve come across a particular challenge that many of us face: decoding HTTP/2 traffic. In this blog, I&#8217;ll share insights on why decoding HTTP/2 headers can be tricky, how HPACK adds a layer of complexity, and most importantly, how eBPF uprobes can come to the rescue.</p>
<p>It is crucial to gain visibility into the messages exchanged between services for a comprehensive understanding and effective troubleshooting of issues. Luckily, it is possible to track the traffic enabling you to effectively debug your HTTP/2 applications.</p>
<h2><strong>What does Wireshark do?</strong></h2>
<p><a href="https://www.wireshark.org/">Wireshark</a> is a popular open-source network protocol analyzer that allows you to capture and inspect the data conversing back and forth on a network in real time.</p>
<p>However, Wireshark sometimes fails to decode the HTTP/2. The issue stems from the binary framing of HTTP/2 packets, making it challenging for Wireshark to precisely decode headers. This challenge intensifies when dealing with encrypted traffic or intricate sequences of frames, leaving developers in a quandary. Let&#8217;s delve into a scenario where we attempt to inspect HTTP/2 traffic using Wireshark. We might encounter difficulties decoding headers due to the multiplexing of streams within a single connection. Traditional tools, designed for simpler protocols, may falter in providing a clear interpretation, emphasizing the need for a more sophisticated solution.</p>
<pre><code class="language-python"># Sample HTTP/2 binary framing
frame_data = b&#039;\x00\x00\x0c\x08\x00\x00\x00\x00\x00\x00\
x00\x01\x00\x00\x0c\x04\x00\x00\x00\x01Hello, HTTP/2!&#039;

# Decode HTTP/2 headers
decoded_headers = decode_http2_headers(frame_data)
print(decoded_headers)</code></pre>
<p>This snippet showcases the challenge of interpreting binary HTTP/2 frames, which can be a stumbling block for tools like Wireshark. Normally, we can create a function such as <code>decode_http2_headers</code> to determine the exact output of the above.</p>
<pre><code class="language-python">def decode_http2_headers(frame_data):
    # Assuming the frame_data follows the HTTP/2 binary framing format
    # Extract the frame type and payload length
    frame_type = frame_data[3]
    payload_length = int.from_bytes(frame_data[5:9], byteorder=&#039;big&#039;)

    # Check if it&#039;s a HEADERS frame (frame type 0x01 for HTTP/2)
    if frame_type == 0x01:
        # Extract the payload containing headers
        headers_payload = frame_data[9:]

        # Parse the headers payload (a more sophisticated parser is needed in a real-world scenario)
        headers = parse_headers_payload(headers_payload)

        return headers

    return None

def parse_headers_payload(payload):
    # This is a simplified parser; a complete parser would need to handle HPACK compression, etc.
    headers_list = payload.decode(&#039;utf-8&#039;).split(&#039;\r\n&#039;)

    # Convert headers to a dictionary
    headers_dict = {}
    for header in headers_list:
        if &#039;:&#039; in header:
            key, value = header.split(&#039;: &#039;, 1)
            headers_dict[key] = value

    return headers_dict

# Sample HTTP/2 binary framing
frame_data = b&#039;\x00\x00\x0c\x08\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x0c\x04\x00\x00\x00\x01Hello, HTTP/2!&#039;

# Decode HTTP/2 headers
decoded_headers = decode_http2_headers(frame_data)
print(decoded_headers)</code></pre>
<p>By running the above code snippet we can get our output:-</p>
<pre><code class="language-python">{&#039;Hello&#039;: &#039;HTTP/2!&#039;}</code></pre>
<p>But this is a highly simplified example, and a real-world HTTP/2 header decoding function would need to handle a variety of scenarios, including HPACK compression, binary encoding, and more. The actual output would depend on the structure and content of the HTTP/2 headers in the given <code>frame_data</code>.</p>
<h2><strong>How does eBPF solve the issue?</strong></h2>
<p>So if we can’t properly decode HTTP/2 traffic without knowing the state, what can we do?</p>
<p>Thankfully, with eBPF it becomes possible for us to observe HTTP/2 implementation to get the information that we need, without requiring state. By attaching uprobes to the HTTP/2 library APIs that take clear-text headers as input, the uprobes can directly read the header content from application memory.</p>
<p>The first thing I need to do is find a specific function in my code that holds all the important info about HTTP/2. This function should use a straightforward argument structure for easy data access within the eBPF code. The objective is to establish a reliable and adaptable foundation for observing and optimizing HTTP/2 interactions, this process entails strategically selecting a function that simplifies the manual pointer manipulation required for eBPF code.</p>
<pre><code class="language-python">from bcc import BPF
from datetime import datetime

# BPF program definition
bpf_code = &quot;&quot;&quot;
#include &lt;uapi/linux/ptrace.h&gt;
BPF_HASH(start, u32);
int trace_http2_send_request_headers(struct pt_regs *ctx) {
    u32 pid = bpf_get_current_pid_tgid();
    u64 ts = bpf_ktime_get_ns();
    start.update(&amp;pid, &amp;ts);
    return 0;
}
int trace_http2_recv_response_headers(struct pt_regs *ctx) {
    u32 pid = bpf_get_current_pid_tgid();
    u64 *tsp, delta;
    tsp = start.lookup(&amp;pid);
    if (tsp != 0) {
        delta = bpf_ktime_get_ns() - *tsp;
        bpf_trace_printk(&quot;HTTP/2 request took %lld ns\\n&quot;, delta);
        start.delete(&amp;pid);
    }
    return 0;
}
&quot;&quot;&quot;

# Attach BPF program to HTTP/2 functions
b = BPF(text=bpf_code)
b.attach_uprobe(name=&quot;your_http2_binary&quot;, sym=&quot;http2_send_request_headers&quot;, fn_name=&quot;trace_http2_send_request_headers&quot;)
b.attach_uprobe(name=&quot;your_http2_binary&quot;, sym=&quot;http2_recv_response_headers&quot;, fn_name=&quot;trace_http2_recv_response_headers&quot;)

# Print trace events
while True:
    try:
        task, pid, cpu, flags, ts, msg = b.trace_fields()
        print(f&quot;{datetime.utcfromtimestamp(ts).strftime(&#039;%Y-%m-%d %H:%M:%S&#039;)} PID {pid}: {msg}&quot;)
    except KeyboardInterrupt:
        break</code></pre>
<p>This is a simplified example of how we can do HTTP/2 tracing using eBPF uprobes. Now, let&#8217;s customize it so that the tracer is launched after the connection between the client and server is established.</p>
<pre><code class="language-python">bpf_code = &quot;&quot;&quot;
#include &lt;linux/sched.h&gt;
BPF_HASH(start, u32);
int trace_http2_headers(struct __sk_buff *skb) {
    u32 pid = bpf_get_current_pid_tgid();
    u64 ts = bpf_ktime_get_ns();
    start.update(&amp;pid, &amp;ts);
    return 0;
}
&quot;&quot;&quot;</code></pre>
<p>Instead of <code>trace_http2_recv_response_headers</code> and <code>trace_http2_send_request_headers</code>, we are using the <code>trace_http2_headers</code> function which is associated with the HTTP/2 headers, and prints a message when headers are received.</p>
<p>We are using <code>tcp_v{4,6}_connect</code> tracepoint, which is triggered when a TCP connection is established, and when this event occurs, it updates a timestamp in the BPF hash table. You can refer to the sample app code on <a href="https://github.com/Sonichigo/http_server/tree/main/server">GitHub</a>.</p>
<p>Now, when I run the Flask app and access it through my browser, I will get output on my terminal, which will look something like this:</p>
<pre><code class="language-yaml">2023-12-11 12:00:00 PID 1234: HTTP/2 headers received
2023-12-11 12:00:05 PID 5678: HTTP/2 headers received</code></pre>
<p>The messages indicate when HTTP/2 headers are received, and the associated PID helps identify the process of handling the HTTP/2 traffic.</p>
<h2><strong>Conclusion</strong></h2>
<p>Tracing HTTP/2 activity is hard because of a complicated compression method called HPACK. However, in this post, we showed a different method to catch messages. Instead of dealing with HPACK directly, we used eBPF Uprobes to track certain functions in the HTTP/2 library. This gives us a clearer way to see what&#8217;s happening with the messages in our HTTP/2 traffic.</p>
<p>The main advantage is the ability to trace messages regardless of when the tracer was deployed. In the end, our goal was to optimize for an approach that worked out of the box, regardless of the deployment order, which is what led us to the eBPF Uprobe-based approach.</p>

         
        
                    

                    
                    <div class="pp-multiple-authors-boxes-wrapper pp-multiple-authors-wrapper pp-multiple-authors-layout-boxed multiple-authors-target-the-content box-post-id-16 box-instance-id-1 ppma_boxes_16"
                    data-post_id="16"
                    data-instance_id="1"
                    data-additional_class="pp-multiple-authors-layout-boxed.multiple-authors-target-the-content"
                    data-original_class="pp-multiple-authors-boxes-wrapper pp-multiple-authors-wrapper box-post-id-16 box-instance-id-1">
                                                                                    <h2 class="widget-title box-header-title">Author</h2>
                                                                            <span class="ppma-layout-prefix"></span>
                        <div class="ppma-author-category-wrap">
                                                                                                <span class="ppma-category-group ppma-category-group-">
                                                                                                                        <ul class="pp-multiple-authors-boxes-ul">
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <li class="pp-multiple-authors-boxes-li author_index_0 author_animesh has-avatar">
                                                                                                                                                                                                                                            <div class="pp-author-boxes-avatar">
                                                                                                                                            <img alt='Animesh Pathak' src='https://wp.keploy.io/wp-content/uploads/2024/09/WhatsApp-Image-2024-08-06-at-3.17.14-PM.jpeg' srcset='https://wp.keploy.io/wp-content/uploads/2024/09/WhatsApp-Image-2024-08-06-at-3.17.14-PM.jpeg' class='multiple_authors_guest_author_avatar avatar' height='80' width='80'/>                                                                                                                                    </div>
                                                            
                                                            <div class="pp-author-boxes-avatar-details">
                                                                                                                                                                                                <div class="pp-author-boxes-name multiple-authors-name">
                                                                        <a href="https://wp.keploy.io/author/animesh/" rel="author" title="Animesh Pathak" class="author url fn">Animesh Pathak</a> 
                                                                    </div>
                                                                                                                                                                                                                                                                    <p class="pp-author-boxes-description multiple-authors-description">
                                                                        I'm a DevRel engineer who have 3+ year of working experience with AI and API. I am an active OSS Contributor and Tinker, who likes to try out emerging tech and build content around the same.                                                                    </p>
                                                                                                                                <a class="ppma-author-user_url-profile-data ppma-author-field-meta ppma-author-field-type-url" aria-label="Website" href="https://linkedin.com/in/sonichigo" rel="dofollow" target="_blank"><span class="dashicons dashicons-admin-links"></span> </a>
                                                                                                                                
                                                                                                                            </div>
                                                                                                                                                                                                                                                                                                                                                                                                             </li>
                                                                                                                                                                                                                                    </ul>
                                                                            </span>
                                                                                    </div>
                    <span class="ppma-layout-suffix"></span>
                    </div>
                    
                    
                
                                <style>
                .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-author-boxes-avatar img { width: 80px !important; height: 80px !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-author-boxes-avatar img { border-style: none !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-author-boxes-avatar img { border-radius: 50% !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-author-boxes-meta a { background-color: #655997 !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-author-boxes-meta a { color: #ffffff !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-author-boxes-meta a:hover { color: #ffffff !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .ppma-author-user_email-profile-data { background-color: #655997 !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .ppma-author-user_email-profile-data { border-radius: 100% !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .ppma-author-user_email-profile-data { color: #ffffff !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .ppma-author-user_url-profile-data { background-color: #655997 !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .ppma-author-user_url-profile-data { border-radius: 100% !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .ppma-author-user_url-profile-data { color: #ffffff !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-author-boxes-recent-posts-title { border-bottom-style: dotted !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-multiple-authors-boxes-li { border-style: solid !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-multiple-authors-boxes-li { border-width: 1px !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-multiple-authors-boxes-li { border-color: #999 !important; } .pp-multiple-authors-boxes-wrapper.box-post-id-16.pp-multiple-authors-layout-boxed.multiple-authors-target-the-content.box-instance-id-1 .pp-multiple-authors-boxes-li { color: #3c434a !important; }             </style>

Decoding HTTP/2 Traffic is Hard, but eBPF can help

Python Testing with Pytest: Features & Best Practices

eBPF for TLS Traffic Tracing: Secure & Efficient Observability

What is Postgres Wire Protocol

Finding Elements in a List using Python

Top 5 Best Python IDE to use in 2024

BitBucket Self-Hosting : Running ebpf/Privileged programs

Unit Testing in Python is way more convenient than you’ve thought

Building Keploy.io, an EBPF based open source framework to generate test cases and data stubs from API calls.

Neha

Cloud technology veteran and probably the youngest (globally) to have completed all 5 AWS certifications (including Solutions architect Professional and Dev­ops engineer professional)

Decoding Http/2 Traffic Is Hard, But Ebpf Can Help

More Stories