Auth service slows down
  → goroutines pile up waiting
  → shared thread/connection pool exhausted
  → Payment service also fails (never touched auth)
  → Entire service down

Auth service slows down
  → only auth's semaphore/pool fills up
  → auth calls get fast 503s
  → Payment service unaffected
  → Ship stays afloat

sem := semaphore.NewWeighted(20) // max 20 concurrent calls

func call(ctx context.Context) error {
    // TryAcquire = instant rejection if full (true bulkhead)
    // Acquire    = waits up to QueueTimeout for a slot
    acquireCtx, cancel := context.WithTimeout(ctx, 50*time.Millisecond)
    defer cancel()

    if err := sem.Acquire(acquireCtx, 1); err != nil {
        return fmt.Errorf("bulkhead full: %w", err) // fast 503
    }
    defer sem.Release(1)

    // make the actual call
}

transport := &http.Transport{
    MaxConnsPerHost:     20, // hard ceiling on total connections
    MaxIdleConnsPerHost: 20, // warm connections kept alive
    MaxIdleConns:        20, // global idle pool
}

client := &http.Client{
    Transport: transport,
    Timeout:   600 * time.Millisecond, // absolute end-to-end deadline
}

transport := &http.Transport{
    MaxIdleConnsPerHost:   N,
    MaxConnsPerHost:       N,
    MaxIdleConns:          N,
    DialContext:           ...,
    TLSHandshakeTimeout:   ...,
    ResponseHeaderTimeout: ...,
    IdleConnTimeout:       ...,
}
client := &http.Client{
    Transport: transport,
    Timeout:   ...,
}

What:    Maximum total TCP connections (active + idle) to a single host.
         New requests BLOCK when this is hit (until a connection is freed or context times out).
Effect:  This IS the TCP-level bulkhead. Beyond this number, no new connections are opened.

MaxConnsPerHost: cfg.MaxConcurrent // keep these equal

What:    How many idle (reusable) connections to keep open to a single host after a request completes.
Effect:  Avoids TCP+TLS handshake overhead on the next request. Higher = faster reuse, more FDs held open.

MaxIdleConnsPerHost = MaxConnsPerHost       // internal, high-throughput
MaxIdleConnsPerHost = MaxConnsPerHost / 2   // external, slow/bursty

What:    Total idle connections across ALL hosts combined.
Effect:  If this is too low, connections for one host get evicted to make room for another,
         causing unexpected TCP reconnects even though per-host limits haven't been hit.

// If you have 3 bulkheads: auth(20) + payment(10) + inventory(15)
MaxIdleConns: 20 + 10 + 15, // = 45, avoids cross-host eviction

What:    Time allowed between sending the request body and receiving the first byte of response headers.
Effect:  The most important timeout for catching slow/hung servers.
         Does NOT include time to read the response body.

External API p99 = 500ms → ResponseHeaderTimeout = 600ms
Internal svc p99 = 5ms   → ResponseHeaderTimeout = 20ms
Database p99     = 20ms  → ResponseHeaderTimeout = 50ms

What:    Time from request initiation to reading the last byte of response body.
         Includes: DNS + TCP dial + TLS + write request + wait for headers + read body.
Effect:  The outer hard deadline. Cancels the entire request if exceeded.

ResponseHeaderTimeout = 600ms  (detect slow server)
body read estimate    = 200ms  (for your expected payload size)
Client.Timeout        = 800ms  (slightly above their sum)

What:    Time allowed to establish the TCP connection itself.
Effect:  Catches unreachable hosts fast. Completely independent of request latency.

DialContext: (&net.Dialer{
    Timeout:   2 * time.Second,  // TCP dial hard limit
    KeepAlive: 30 * time.Second, // TCP keepalive probes
}).DialContext,

What:    Time allowed for TLS handshake after TCP is established.
Effect:  Catches TLS issues (expired certs, overloaded TLS terminator) separately from request latency.

TLSHandshakeTimeout: 5 * time.Second,

What:    How long an idle connection is kept in the pool before being closed.
Effect:  Prevents holding open TCP connections that the server has already closed (common after 60–90s).

IdleConnTimeout: 90 * time.Second, // safe default

What:    How long a request waits for a semaphore slot before being rejected.
Effect:  Controls the queue depth in time. Even if a slot frees in 10ms, you might not want to wait.

Your SLA:              500ms total
Downstream p99:        200ms
Budget for queuing:    500 - 200 - 50(overhead) = 250ms
QueueTimeout:          ~100ms (conservative, leaves room for retries)

L = λ × W

L = concurrent requests in-flight (what your semaphore controls)
λ = throughput (requests per second you want to sustain)
W = average time per request (latency in seconds)

Target: 100 rps

Auth svc  p99=100ms → L = 100 × 0.1 = 10 concurrent  (fast, slots free quickly)
Payment   p99=500ms → L = 100 × 0.5 = 50 concurrent  (slow, slots held longer)

Payment svc p99=500ms, target 100rps, but MaxConcurrent=10:

Max throughput = 10 / 0.5 = 20 rps   ← you're throttling yourself to 20%

Blast radius grows: more goroutines can pile up waiting on a misbehaving dep.
Blast radius is controlled by TIMEOUT, not concurrency.

min_concurrent = target_rps × p99_latency_seconds
semaphore_limit = min_concurrent × 1.3   // +30% headroom for bursts

IO call timeline:
[send ~50µs] [====== blocked waiting ~200ms ======] [read ~50µs]
                        ↑
              goroutine is PARKED here, OS thread runs other work

→ semaphore limit is a downstream capacity question, not a CPU question

package bulkhead

import (
    "context"
    "fmt"
    "net"
    "net/http"
    "sync/atomic"
    "time"

    "golang.org/x/sync/semaphore"
)

// Config defines the bulkhead parameters for one downstream dependency.
type Config struct {
    Name           string
    MaxConcurrent  int64         // semaphore size — derived from Little's Law: target_rps × p99_latency
    MaxConnections int           // TCP connection ceiling — keep equal to MaxConcurrent
    RequestTimeout time.Duration // absolute end-to-end HTTP deadline (blast radius knob)
    QueueTimeout   time.Duration // how long to wait for a semaphore slot before rejecting
}

// Metrics holds live counters. All fields use atomic ops — safe to read from any goroutine.
type Metrics struct {
    Accepted atomic.Int64 // requests that acquired a semaphore slot
    Rejected atomic.Int64 // requests rejected because bulkhead was full
    Errors   atomic.Int64 // requests that acquired a slot but the HTTP call failed
}

// Bulkhead combines a per-dependency semaphore and an isolated HTTP client.
// Each instance is independent — a saturated bulkhead has zero effect on others.
type Bulkhead struct {
    cfg     Config
    sem     *semaphore.Weighted // Gate 1: concurrency cap
    client  *http.Client        // Gate 2: connection pool cap + timeouts
    Metrics Metrics
}

// New creates a Bulkhead. Call once per downstream dependency at startup.
func New(cfg Config) *Bulkhead {
    transport := &http.Transport{
        // --- Gate 2: TCP-level bulkhead ---
        MaxConnsPerHost:     cfg.MaxConnections,        // hard ceiling: new reqs block when hit
        MaxIdleConnsPerHost: cfg.MaxConnections,        // warm connections: avoids TCP+TLS overhead
        MaxIdleConns:        cfg.MaxConnections,        // global pool: set >= sum of all per-host

        // --- TCP dial ---
        DialContext: (&net.Dialer{
            Timeout:   2 * time.Second,  // TCP handshake hard limit (unreachable host detection)
            KeepAlive: 30 * time.Second, // TCP keepalive probe interval
        }).DialContext,

        // --- TLS ---
        TLSHandshakeTimeout: 5 * time.Second, // TLS negotiation limit

        // --- Request lifecycle ---
        // ResponseHeaderTimeout: time between sending request and receiving first response byte.
        // This is the "server is hung" detector. Set to p99 × 1.2.
        ResponseHeaderTimeout: cfg.RequestTimeout,

        // Evict idle connections before the server closes them (~90s on most servers).
        IdleConnTimeout: 90 * time.Second,
    }

    return &Bulkhead{
        cfg: cfg,
        sem: semaphore.NewWeighted(cfg.MaxConcurrent),
        client: &http.Client{
            Transport: transport,
            // Absolute deadline covering DNS+TCP+TLS+write+headers+body.
            // Always set — without this, goroutines can leak indefinitely.
            Timeout: cfg.RequestTimeout,
        },
    }
}

// Do executes an HTTP request through the bulkhead.
// Returns an error immediately if the semaphore is full (after QueueTimeout).
func (b *Bulkhead) Do(ctx context.Context, req *http.Request) (*http.Response, error) {
    // --- Gate 1: Semaphore ---
    // Use a child context with QueueTimeout so we don't wait forever for a slot.
    acquireCtx, cancel := context.WithTimeout(ctx, b.cfg.QueueTimeout)
    defer cancel()

    if err := b.sem.Acquire(acquireCtx, 1); err != nil {
        b.Metrics.Rejected.Add(1)
        return nil, fmt.Errorf("bulkhead [%s] rejected (semaphore full after %s): %w",
            b.cfg.Name, b.cfg.QueueTimeout, err)
    }
    defer b.sem.Release(1) // always release, even on HTTP error

    b.Metrics.Accepted.Add(1)

    // --- Gate 2: HTTP client (connection pool + timeouts) ---
    resp, err := b.client.Do(req.WithContext(ctx))
    if err != nil {
        b.Metrics.Errors.Add(1)
        return nil, fmt.Errorf("bulkhead [%s] call failed: %w", b.cfg.Name, err)
    }

    return resp, nil
}

// Stats returns a human-readable snapshot of metrics.
func (b *Bulkhead) Stats() string {
    return fmt.Sprintf("[%s] accepted=%d rejected=%d errors=%d",
        b.cfg.Name,
        b.Metrics.Accepted.Load(),
        b.Metrics.Rejected.Load(),
        b.Metrics.Errors.Load(),
    )
}

authBulkhead.sem   ──► controls only auth calls
paymentBulkhead.sem ──► controls only payment calls

Auth saturated:
  authBulkhead.sem full → auth calls rejected
  paymentBulkhead.sem unaffected → payment calls proceed normally

package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "time"

    "yourmodule/bulkhead"
)

// Declare one bulkhead per downstream dependency at package level.
// These are long-lived, safe to use concurrently.
var (
    authBulkhead    = bulkhead.New(authConfig())
    paymentBulkhead = bulkhead.New(paymentConfig())
)

func HandleOrder(ctx context.Context) error {
    // Auth call — isolated behind its own semaphore + connection pool
    req, _ := http.NewRequest("GET", "https://auth.internal/verify", nil)
    resp, err := authBulkhead.Do(ctx, req)
    if err != nil {
        // Bulkhead full or auth down — fail this request fast.
        // Payment bulkhead is completely unaffected.
        return fmt.Errorf("auth unavailable: %w", err)
    }
    defer resp.Body.Close()
    io.Copy(io.Discard, resp.Body) // always drain body to return connection to pool

    // Payment call — completely isolated from auth
    req2, _ := http.NewRequest("POST", "https://payments.internal/charge", nil)
    resp2, err := paymentBulkhead.Do(ctx, req2)
    if err != nil {
        return fmt.Errorf("payment unavailable: %w", err)
    }
    defer resp2.Body.Close()
    io.Copy(io.Discard, resp2.Body)

    return nil
}

min_concurrent  = target_rps × p99_latency_seconds
semaphore_limit = min_concurrent × 1.3   // +30% burst headroom
MaxConnections  = semaphore_limit        // keep aligned
RequestTimeout  = p99_latency × 1.2     // slightly above p99, blast radius knob
QueueTimeout    = your_SLA - p99 - overhead_budget

// Little's Law: 30 × 0.5 = 15, +30% = 20
// Blast radius: controlled by tight RequestTimeout (600ms), NOT by low concurrency
// Slower dep → more slots needed to sustain throughput
func externalAPIConfig() bulkhead.Config {
    return bulkhead.Config{
        Name:           "external-auth-api",
        MaxConcurrent:  20,                       // L = 30rps × 0.5s = 15, +30% = 20
        MaxConnections: 20,
        RequestTimeout: 600 * time.Millisecond,   // p99(500ms) × 1.2 — blast radius knob
        QueueTimeout:   100 * time.Millisecond,   // don't queue long for slow dep
    }
}

// Little's Law: 500 × 0.005 = 2.5 → floor at ~10 (connection overhead)
// Faster dep → naturally fewer slots needed, slots free in milliseconds
func internalServiceConfig() bulkhead.Config {
    return bulkhead.Config{
        Name:           "inventory-internal",
        MaxConcurrent:  10,                       // L = 500rps × 0.005s = 2.5, floor at 10
        MaxConnections: 10,
        RequestTimeout: 20 * time.Millisecond,    // tight: it's internal, same DC
        QueueTimeout:   5 * time.Millisecond,     // fail fast: if pool busy, something is wrong
    }
}

// Little's Law: 200 × 0.02 = 4 per instance
// Hard constraint: Postgres max_connections=100, shared across 5 instances → 20 per instance
// Use: max(Little's Law result × 1.3, safety floor) but never exceed hard cap
func postgresConfig() bulkhead.Config {
    return bulkhead.Config{
        Name:           "postgres",
        MaxConcurrent:  10,                       // L=4, ×1.3=5.2, safety floor=10, hard cap=20 ✓
        MaxConnections: 10,
        RequestTimeout: 100 * time.Millisecond,   // DB should be fast; slow DB = problem
        QueueTimeout:   20 * time.Millisecond,
    }
}

// Little's Law: 2000 × 0.002 = 4 concurrent
// Kafka batches internally — keep TCP connections low, semaphore has headroom for bursts
func kafkaConfig() bulkhead.Config {
    return bulkhead.Config{
        Name:           "kafka-producer",
        MaxConcurrent:  10,                       // L=4, headroom for burst
        MaxConnections: 3,                        // Kafka multiplexes internally; few TCP conns enough
        RequestTimeout: 50 * time.Millisecond,
        QueueTimeout:   5 * time.Millisecond,
    }
}

Little's Law          → sets MaxConcurrent
  slow dep = MORE slots (slots held longer)
  fast dep = FEWER slots (slots released quickly)

Blast Radius Control  → sets RequestTimeout + QueueTimeout
  NOT concurrency — tight timeouts release slots fast
  even if MaxConcurrent is high, a 600ms timeout limits damage

Normal:                         Auth saturated:

User → Auth    ✓               User → Auth    ✗ (semaphore full → instant 503)
User → Payment ✓               User → Payment ✓ (own semaphore, unaffected)
User → S3      ✓               User → S3      ✓ (own semaphore, unaffected)

Layer                   | Protects against
────────────────────────┼──────────────────────────────────────────
Semaphore               | Goroutine pile-up from slow/hung deps
Connection pool         | TCP connection exhaustion from one host
ResponseHeaderTimeout   | Server that accepts connection but never responds
Client.Timeout          | Goroutine leak from infinite response body reads
QueueTimeout            | Cascading slowdowns from queue buildup

transport := &http.Transport{
    MaxConnsPerHost:     20,               // bulkhead ceiling
    MaxIdleConnsPerHost: 20,               // full connection reuse
    IdleConnTimeout:     90 * time.Second, // evict before server does

    DialContext: (&net.Dialer{
        Timeout:   2 * time.Second,
        KeepAlive: 30 * time.Second,
    }).DialContext,
    TLSHandshakeTimeout:   5 * time.Second,
    ResponseHeaderTimeout: cfg.RequestTimeout,
}