Caddy compatibility for zeroserve: 3x throughput and 70% lower latency

zeroserve is a high-performance HTTPS server that runs eBPF scripts in userspace (intro). Now it's got a Caddy-compat mode - when provided a Caddyfile, zeroserve JIT-compiles it to eBPF and then to native x86_64/ARM64 machine code, and runs it in an io_uringevent loop.

protocol	server	throughput	p50	p99	peak RSS
https	zeroserve-clang	38,948 req/s	1.45ms	3.91ms	30.9 MiB
https	zeroserve-tcc	36,653 req/s	1.67ms	4.00ms	34.2 MiB
https	caddy	12,529 req/s	4.74ms	13.11ms	67.4 MiB
https	nginx	37,424 req/s	1.57ms	4.24ms	25.7 MiB

HTTPS reverse proxy, 2 threads, AMD Ryzen 7 3700X. Check CI for original run result.

Try it with your Caddyfile:

curl -fL -o zeroserve https://github.com/losfair/zeroserve/releases/download/v0.2.11/zeroserve-$(uname -m)-linux
chmod +x zeroserve
./zeroserve --caddy /etc/caddy/Caddyfile
curl http://127.0.0.1:8080

zeroserve runs turing-complete eBPF and you can call custom code from your Caddyfile. For example, to reverse-proxy a path to an S3-compatible bucket with AWS SigV4 auth, grab io.su3.aws-sigv4.c and then:

# zeroserve --plugin io.su3.aws-sigv4.c --caddy Caddyfile

example.com {
  route /s3/* {
    uri strip_prefix /s3
    rewrite * /my-bucket{uri}
  
    # Call the `sign_request` method in the eBPF middleware `io.su3.aws-sigv4.o`
    zeroserve_call io.su3.aws-sigv4 sign_request {
      access_key_id "minioadmin"
      secret_access_key "minioadmin"
    }
  
    reverse_proxy http://127.0.0.1:9000
  }
}