The problem mirroring solves

You have rewritten a service. The old one is Python, the new one is Go, the test suite is green, and you still do not trust it. Green tests prove the new version handles the inputs you thought of. Production carries the ones you did not: the client still sending a field you deprecated two years ago, a date in a format nobody documented, an encoding that one old mobile release produces and nothing else does.

Traffic mirroring answers what tests cannot. Istio sends every real request to the current version and a copy of it to the new one. The copy's response is discarded, the real one goes back to the user, and nobody outside the cluster can tell. Unreleased code gets a stream of genuine production traffic with no blast radius.

The same shape covers four other jobs:

• Load testing with real traffic. A generator produces uniform requests. Mirroring reproduces the actual mix of paths, payload sizes and cache hit ratios.
• Debugging without touching production. Point the copy at a pod running verbose logging, a profiler or a debugger, none of which you would enable on the live path.
• Validating a dependency upgrade. New database driver, new TLS library, new base image. Run it against a day of real traffic and watch for errors and latency drift.
• Scoring a new model. Feed real requests to a new fraud or ranking model and compare its decisions with the current one before it is allowed to make any.

All five end in the same place. Two versions have now handled the same request, and you need to know whether they agreed. That is where mirroring stops helping.

Why that turns out to be hard

Both workloads log what they served. For any single request you want to line up its two log entries and diff them, and nothing in the output obviously connects the pair.

Teams arriving from a hand-rolled mirroring gateway usually had this solved. The gateway stamped a unique header before duplicating the request, both copies carried it, and the log join was a grep. Istio appears to take that hook away.

It does not. The join key is already there, and it is already in Istio's default access log format. One thing genuinely breaks it, one hop downstream. Four further behaviours will surprise you along the way, including one that Istio's own documentation still describes incorrectly.

Everything below runs on a throwaway k3d cluster, and every command and output is from that cluster. The lab mirrors echo-v1 to echo-v2, which stand in for the old and new implementations above; echo-v2 carries one seeded behaviour change so the comparison at the end has something real to catch.

Key Takeaways

• x-request-id is the join key. Envoy copies request headers into the mirrored stream untouched, so both sides carry the same value. No configuration needed
• One hop downstream it collides. The real path and the shadow path reach the next service carrying an identical x-request-id, and nothing built in separates them
• The -shadow Host suffix is gone. Istio 1.28 flipped the default. Istio's mirroring docs still describe the old behaviour
• The source proxy logs nothing for the mirror. One request produces one access log line at the caller, for the primary only
• Istio discards the mirrored response, so comparing v1 against v2 is code you write

---

What Istio mirroring does

Mirroring is an Envoy shadow policy attached to a route. When the router filter handles a matching request, it copies the request headers, opens a second upstream stream to the mirror cluster, and sends the copy there. The response from that second stream is read and thrown away.

Two implementation details drive everything later. The copy goes out through Envoy's async client rather than the HTTP connection manager that writes access logs. And it is fire and forget: the comment in Envoy's shadow_writer_impl.cc reads "This is basically fire and forget. We don't handle cancelling." The caller never waits for the mirror and never learns what happened to it.

The header copy is almost untouched. getClusterAndPreprocessHeadersAndOptions() rewrites the authority and nothing else, and only when the suffix option is on. x-request-id arrives at the shadow byte for byte, which is what makes the whole correlation approach possible.

Building the lab

Four workloads: echo-v1 as the primary, echo-v2 as the mirror target, echo-b as a downstream dependency that both of them call, and a curl pod as the client. The echo-b hop exists because that is where the interesting failure lives.

Cluster and Istio

k3d cluster create istio-mirror \
  --agents 1 \
  --k3s-arg "--disable=traefik@server:*" \
  --wait --timeout 300s
kubectl config use-context k3d-istio-mirror

Traefik is disabled because Istio's ingress gateway takes that role. Istio 1.31 installs on the Kubernetes 1.35 that k3d ships without complaint; istioctl x precheck returns "No issues found".

ISTIO_VERSION=1.31.0
ISTIO_ARCH=osx-arm64      # or linux-amd64, linux-arm64, osx-amd64
BASE="https://github.com/istio/istio/releases/download/${ISTIO_VERSION}"

curl -sSL -O "${BASE}/istioctl-${ISTIO_VERSION}-${ISTIO_ARCH}.tar.gz"
curl -sSL -O "${BASE}/istioctl-${ISTIO_VERSION}-${ISTIO_ARCH}.tar.gz.sha256"
shasum -a 256 -c "istioctl-${ISTIO_VERSION}-${ISTIO_ARCH}.tar.gz.sha256"
tar xzf "istioctl-${ISTIO_VERSION}-${ISTIO_ARCH}.tar.gz"

./istioctl install --set profile=default -y

Access logging, which the default profile leaves off

A pod in the mesh runs two containers, and each one produces a different kind of log.

The application container logs whatever your code prints. The istio-proxy sidecar logs something else entirely: an access log, one line per HTTP request that passes through the proxy, recording the method, path, response code, which upstream it picked, how long it took, and a handful of headers. Your application never sees that line and cannot produce it, because it describes the proxy's view of the request rather than the application's.

You read the two streams separately, with -c selecting the container:

kubectl -n shadow-lab logs <pod> -c echo          # what the application printed
kubectl -n shadow-lab logs <pod> -c istio-proxy   # the proxy's access log

This matters here because the access log is where the correlation lives. Istio's default line format already contains x-request-id, so once logging is on, the identifier that ties a request to its mirrored copy is sitting in the output of every proxy in the path, with no code change anywhere.

It is off by default. istioctl install --set profile=default leaves meshConfig.accessLogFile unset, and an unset value means the sidecars write no access log lines at all. Skip this step and every kubectl logs ... -c istio-proxy later in the article returns nothing, which looks exactly like a request that never arrived.

The Telemetry API switches it on for the whole mesh. A Telemetry object named mesh-default in istio-system applies to every workload; providers: [{ name: envoy }] selects Istio's built-in provider, which writes Envoy's standard text format to the sidecar's stdout so kubectl logs can read it:

kubectl apply -f - <<'EOF'
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  accessLogging:
    - providers:
        - name: envoy
EOF

$ kubectl -n istio-system get telemetry mesh-default
NAME           AGE
mesh-default   4s

The object exists immediately. Actual log lines only start appearing once workloads are running and receiving traffic, which is the next step.

The application

A single Python file. It logs one JSON line per request with the headers that matter and a hash of the response body, and it forwards trace headers downstream the way Istio asks applications to. BEHAVIOUR_DRIFT changes one response field, standing in for a bug introduced during a rewrite.

cat > app.py <<'EOF'
import hashlib, json, os, sys, urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

VARIANT = os.environ.get("VARIANT", "unknown")
DRIFT = os.environ.get("BEHAVIOUR_DRIFT") == "1"
DOWNSTREAM = os.environ.get("DOWNSTREAM_URL")

# The headers Istio asks applications to forward so a trace survives a hop.
PROPAGATE = ["x-request-id", "x-b3-traceid", "x-b3-spanid", "x-b3-parentspanid",
             "x-b3-sampled", "x-b3-flags", "traceparent", "tracestate",
             "x-ot-span-context", "x-shadow-run"]

class Handler(BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"
    def log_message(self, *a): pass

    def do_GET(self):
        h = self.headers
        downstream_status = None
        # Health probes must not fan out, or they drown the access log.
        if DOWNSTREAM and self.path != "/health":
            req = urllib.request.Request(DOWNSTREAM)
            for name in PROPAGATE:
                if h.get(name) is not None:
                    req.add_header(name, h.get(name))
            try:
                with urllib.request.urlopen(req, timeout=5) as r:
                    downstream_status = r.status
            except Exception as exc:
                downstream_status = "error: %s" % exc

        body = {"variant_neutral": True, "path": self.path,
                "order_date": "07/09/2026" if DRIFT else "2026-09-07"}
        raw = json.dumps(body, sort_keys=True).encode()

        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(raw)))
        self.send_header("x-variant", VARIANT)
        self.end_headers()
        self.wfile.write(raw)

        record = {"variant": VARIANT, "path": self.path, "authority": h.get("host"),
                  "x_request_id": h.get("x-request-id"),
                  "x_shadow_run": h.get("x-shadow-run"),
                  "traceparent": h.get("traceparent"),
                  "x_b3_traceid": h.get("x-b3-traceid"),
                  "x_b3_spanid": h.get("x-b3-spanid"),
                  "status": 200, "body_sha256": hashlib.sha256(raw).hexdigest()[:16],
                  "body": body}
        if DOWNSTREAM:
            record["downstream_status"] = downstream_status
        sys.stdout.write(json.dumps(record, sort_keys=True) + "\n")
        sys.stdout.flush()

ThreadingHTTPServer(("0.0.0.0", 8080), Handler).serve_forever()
EOF

kubectl create namespace shadow-lab
kubectl label namespace shadow-lab istio-injection=enabled
kubectl -n shadow-lab create configmap echo-app --from-file=app.py=app.py

Workloads

Service/echo selects both versions by app: echo; the version label is what the subsets key on later.

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Service
metadata: { name: echo, namespace: shadow-lab, labels: { app: echo } }
spec:
  selector: { app: echo }
  ports: [{ name: http, port: 8080, targetPort: 8080 }]
---
apiVersion: v1
kind: Service
metadata: { name: echo-b, namespace: shadow-lab, labels: { app: echo-b } }
spec:
  selector: { app: echo-b }
  ports: [{ name: http, port: 8080, targetPort: 8080 }]
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: echo-v1, namespace: shadow-lab }
spec:
  replicas: 1
  selector: { matchLabels: { app: echo, version: v1 } }
  template:
    metadata: { labels: { app: echo, version: v1 } }
    spec:
      containers:
        - name: echo
          image: python:3.13-alpine
          command: ["python3", "-u", "/app/app.py"]
          env:
            - { name: VARIANT,        value: "echo-v1" }
            - { name: DOWNSTREAM_URL, value: "http://echo-b:8080/ledger" }
          ports: [{ containerPort: 8080 }]
          volumeMounts: [{ name: app, mountPath: /app }]
          readinessProbe: { httpGet: { path: /health, port: 8080 }, initialDelaySeconds: 2 }
      volumes: [{ name: app, configMap: { name: echo-app } }]
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: echo-v2, namespace: shadow-lab }
spec:
  replicas: 1
  selector: { matchLabels: { app: echo, version: v2 } }
  template:
    metadata: { labels: { app: echo, version: v2 } }
    spec:
      containers:
        - name: echo
          image: python:3.13-alpine
          command: ["python3", "-u", "/app/app.py"]
          env:
            - { name: VARIANT,         value: "echo-v2" }
            - { name: BEHAVIOUR_DRIFT, value: "1" }
            - { name: DOWNSTREAM_URL,  value: "http://echo-b:8080/ledger" }
          ports: [{ containerPort: 8080 }]
          volumeMounts: [{ name: app, mountPath: /app }]
          readinessProbe: { httpGet: { path: /health, port: 8080 }, initialDelaySeconds: 2 }
      volumes: [{ name: app, configMap: { name: echo-app } }]
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: echo-b, namespace: shadow-lab }
spec:
  replicas: 1
  selector: { matchLabels: { app: echo-b } }
  template:
    metadata: { labels: { app: echo-b } }
    spec:
      containers:
        - name: echo
          image: python:3.13-alpine
          command: ["python3", "-u", "/app/app.py"]
          env: [{ name: VARIANT, value: "echo-b" }]
          ports: [{ containerPort: 8080 }]
          volumeMounts: [{ name: app, mountPath: /app }]
          readinessProbe: { httpGet: { path: /health, port: 8080 }, initialDelaySeconds: 2 }
      volumes: [{ name: app, configMap: { name: echo-app } }]
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: client, namespace: shadow-lab }
spec:
  replicas: 1
  selector: { matchLabels: { app: client } }
  template:
    metadata: { labels: { app: client } }
    spec:
      containers:
        - name: curl
          image: curlimages/curl:8.11.1
          command: ["sleep", "infinity"]
EOF

kubectl -n shadow-lab wait --for=condition=available --timeout=300s deploy --all

Check injection before going further:

$ kubectl -n shadow-lab get pods
NAME                       READY   STATUS    RESTARTS   AGE
client-dddcdc8f5-q8qlj     2/2     Running   0          48s
echo-b-757699dc8d-6bjhf    2/2     Running   0          48s
echo-v1-b487d56cb-mznlk    2/2     Running   0          48s
echo-v2-78f756bf7f-2kps7   2/2     Running   0          48s

2/2 is the signal to trust. On Kubernetes 1.29 and later Istio injects istio-proxy as a native sidecar, which puts it in .spec.initContainers with restartPolicy: Always, not in .spec.containers:

$ kubectl -n shadow-lab get pod -l version=v1 \
    -o jsonpath='{.items[0].spec.initContainers[*].name}'
istio-init istio-proxy

Any injection check that greps .spec.containers[*].name reports a false negative on a perfectly healthy pod.

Turning on mirroring

Two objects. The DestinationRule defines the subsets, and the VirtualService cannot reference subset: v1 without it. Skip it and every request fails with no healthy upstream.

kubectl apply -f - <<'EOF'
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata: { name: echo, namespace: shadow-lab }
spec:
  host: echo
  subsets:
    - name: v1
      labels: { version: v1 }
    - name: v2
      labels: { version: v2 }
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata: { name: echo, namespace: shadow-lab }
spec:
  hosts: ["echo"]
  http:
    - route:
        - destination: { host: echo, subset: v1 }   # every real response comes from v1
      mirrors:
        - destination: { host: echo, subset: v2 }   # v2 gets a copy, response discarded
          percentage: { value: 100.0 }
EOF

Note mirrors, plural. The API has two spellings and the reference documentation uses the older one. In virtual_service.proto, mirror is field 9 and takes a single destination, mirrors is field 22 and takes a list with per-destination percentages. Neither is deprecated. What is deprecated is mirror_percent, the integer field, superseded by the double mirror_percentage. Use mirrors for anything new.

Confirm the routing changed. Before these objects existed, Service/echo round-robined across both versions:

$ CLIENT=$(kubectl -n shadow-lab get pod -l app=client -o jsonpath='{.items[0].metadata.name}')
$ for i in $(seq 1 6); do
    kubectl -n shadow-lab exec "$CLIENT" -c curl -- \
      curl -s -D- -o /dev/null http://echo:8080/orders/$i | grep -i '^x-variant'
  done
x-variant: echo-v1
x-variant: echo-v1
x-variant: echo-v1
x-variant: echo-v1
x-variant: echo-v1
x-variant: echo-v1

Every response now comes from v1. Mirroring changes nothing about what the caller receives.

The join key

Send one request and read both application logs:

$ kubectl -n shadow-lab exec "$CLIENT" -c curl -- \
    curl -s http://echo:8080/orders/corr-1788786710 > /dev/null

$ kubectl -n shadow-lab logs -l version=v1 -c echo --tail=5 | grep corr-1788786710
{"variant":"echo-v1","authority":"echo:8080","x_request_id":"87934116-4953-4113-a47e-62df1e0b0b98","path":"/orders/corr-1788786710"}

$ kubectl -n shadow-lab logs -l version=v2 -c echo --tail=5 | grep corr-1788786710
{"variant":"echo-v2","authority":"echo:8080","x_request_id":"87934116-4953-4113-a47e-62df1e0b0b98","path":"/orders/corr-1788786710"}

Same x-request-id on both sides. That is the correlation the old gateway provided, available with no configuration. Istio's default text access log carries it too: EnvoyTextLogFormat in telemetry_logging.go contains "%REQ(X-REQUEST-ID)%" followed immediately by "%REQ(:AUTHORITY)%".

Where the join key actually breaks

One hop later. Both echo-v1 and echo-v2 call echo-b, forwarding x-request-id exactly as Istio's documentation instructs applications to do. The fork re-converges:

Measured:

$ BPOD=$(kubectl -n shadow-lab get pod -l app=echo-b -o jsonpath='{.items[0].metadata.name}')
$ kubectl -n shadow-lab logs "$BPOD" -c echo | grep -c d80d3228-c36d-4408-abc1-786f595de2ac
2

$ kubectl -n shadow-lab logs "$BPOD" -c echo | grep d80d3228-c36d-4408-abc1-786f595de2ac
{"variant":"echo-b","path":"/ledger","x_request_id":"d80d3228-c36d-4408-abc1-786f595de2ac","authority":"echo-b:8080"}
{"variant":"echo-b","path":"/ledger","x_request_id":"d80d3228-c36d-4408-abc1-786f595de2ac","authority":"echo-b:8080"}

Two arrivals, one id, identical path and authority. At echo-b's sidecar the access log lines differ only by the caller's pod IP. The SNI is identical because echo-b has no subsets, and the mTLS identity is identical because both deployments run under the default ServiceAccount.

So anything past the first hop cannot separate production from shadow, and an IP-to-pod lookup that churns on every restart is not a join key. Your dashboards and traces are double-counting, and the second copy looks exactly like real traffic.

If mirroring is already live in your mesh, this is worth two minutes right now: take one request id from a shadow log line and search every downstream service for it. Two hits means the double-counting has been there since the day mirroring went in. Fixing it needs a marker that Istio does not give you by default, which is the section after next.

Four behaviours that are not what the docs say

The -shadow Host suffix is gone

Look at authority in the join key output again. It reads echo:8080 on both lines. Istio's mirroring task page says otherwise:

"When traffic gets mirrored, the requests are sent to the mirrored service with their Host/Authority headers appended with -shadow. For example, cluster-1 becomes cluster-1-shadow."

That is Envoy's default and it is how Istio behaved through 1.27. It is not what Istio 1.31 does:

$ istioctl proxy-config route "$CLIENT.shadow-lab" --name 8080 -o json \
  | jq '.[].virtualHosts[] | select(.name|test("^echo\\.")) | .routes[].route.requestMirrorPolicies'
[
  {
    "cluster": "outbound|8080|v2|echo.shadow-lab.svc.cluster.local",
    "runtimeFraction": { "defaultValue": { "numerator": 1000000, "denominator": "MILLION" } },
    "traceSampled": false,
    "disableShadowHostSuffixAppend": true
  }
]

Istio sets that flag from pilot/pkg/features/pilot.go:

DisableShadowHostSuffix = env.Register("DISABLE_SHADOW_HOST_SUFFIX", true,
    "If disabled, the shadow host suffix will be added to the hostnames of the mirrored requests.").Get()

It defaults to true, meaning the suffix is not added. Envoy's own default for disable_shadow_host_suffix_append is false, so Istio inverts Envoy here. Checking each release branch shows where it changed:

Istio	DISABLE_SHADOW_HOST_SUFFIX	-shadow appended
1.24 to 1.27	flag absent	yes
1.28 to 1.31	true	no

Setting it to false brings the suffix back, which confirms the direction:

$ kubectl -n istio-system set env deploy/istiod DISABLE_SHADOW_HOST_SUFFIX=false
$ # wait for the new route to reach the sidecar, then send one request
{"variant":"echo-v1","authority":"echo:8080","x_request_id":"00d65de9-915a-44af-aad9-4ff3f1bb6e30"}
{"variant":"echo-v2","authority":"echo-shadow:8080","x_request_id":"00d65de9-915a-44af-aad9-4ff3f1bb6e30"}

Give the config time to land. A request fired fifteen seconds after the istiod rollout still used the old route.

Istio's 1.28 change notes state the inverse, that true adds the suffix. Issue 58855 reported the inversion in January 2026 and was closed in May, but the published text still reads the wrong way round. If your shadow detection keys on -shadow, it stopped matching anything the day you upgraded past 1.27.

Three fields do still separate primary from shadow at the destination:

Field	Primary	Shadow
%REQUESTED_SERVER_NAME%	outbound_.8080_.v1_.echo...	outbound_.8080_.v2_.echo...
%REQ(X-FORWARDED-FOR)%	-	10.42.1.6
%DOWNSTREAM_REMOTE_ADDRESS%	10.42.1.6:42716	10.42.1.6:0

The SNI carries the subset name, which makes it the most direct signal. The other two follow from how the copy is dispatched: the mirrored request is a fresh stream from the async client rather than a continuation of the client's connection, so Envoy has no downstream socket to report and fills the port with 0, and it records the original caller in x-forwarded-for instead.

The source proxy logs nothing for the mirror

One client request produces exactly one access log line at the caller's sidecar, for outbound|8080|v1|.... The shadow goes out through the async client, which never passes through the connection manager that writes access logs. Grepping the caller for evidence that a mirror happened returns nothing, by design.

Per-cluster Envoy stats do not rescue you either. Istio's default stats matcher drops them: pilot-agent request GET stats on a sidecar returns about 220 lines, and the only upstream_rq_* counters belong to cluster.xds-grpc.

Trace context is copied verbatim, span id included

Send a request with an explicit trace header and both sides receive the identical traceparent, down to the same parent span id. The shadow gets no trace context of its own, so an application-level span it emits attaches to the production trace with nothing marking it. Istio does set traceSampled: false on the mirror policy, visible in the route dump above, which suppresses the sidecar's own span. That has no effect on what your application does with the headers it was handed.

Percentages make the join lossy

With percentage.value: 30 over 300 requests, 88 were mirrored, or 29.3%. The field is honest. The consequence is that most primary log lines have no shadow counterpart, and a missing counterpart is indistinguishable from a shadow that failed. Treat "unmatched" as an expected outcome rather than an error.

One caution on measuring this: an earlier 60-request run showed 24 mirrored, or 40%, which was sampling noise. Do not size a mirror-rate check on 60 requests.

Stamping a marker on the mirrored copy only

To fix the downstream collision you want a header on the mirrored copy and not on the primary. A VirtualService cannot do it, because route-level header mutations run before Envoy copies the headers, so they land on both branches.

Envoy can. Its RequestMirrorPolicy in route_components.proto carries a request_headers_mutations field:

message RequestMirrorPolicy {
  string cluster = 1;
  string cluster_header = 5;
  core.v3.RuntimeFractionalPercent runtime_fraction = 3;
  google.protobuf.BoolValue trace_sampled = 4;
  bool disable_shadow_host_suffix_append = 6;
  repeated common.mutation_rules.v3.HeaderMutation request_headers_mutations = 7;
  string host_rewrite_literal = 8;
}

Istio's HTTPMirrorPolicy exposes only destination and percentage, so reaching that field means an EnvoyFilter. Remove mirrors from the VirtualService first: MERGE concatenates repeated fields, so leaving both in place gives you two mirror policies and copies every request twice.

kubectl -n shadow-lab patch virtualservice echo --type=json \
  -p='[{"op":"remove","path":"/spec/http/0/mirrors"}]'

kubectl apply -f - <<'EOF'
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata: { name: mirror-stamp, namespace: shadow-lab }
spec:
  workloadSelector:
    labels: { app: client }
  configPatches:
    - applyTo: HTTP_ROUTE
      match:
        context: SIDECAR_OUTBOUND
        routeConfiguration:
          vhost:
            name: "echo.shadow-lab.svc.cluster.local:8080"
      patch:
        operation: MERGE
        value:
          route:
            request_mirror_policies:
              - cluster: "outbound|8080|v2|echo.shadow-lab.svc.cluster.local"
                runtime_fraction:
                  default_value: { numerator: 100, denominator: HUNDRED }
                disable_shadow_host_suffix_append: true
                request_headers_mutations:
                  - append:
                      header: { key: "x-shadow-run", value: "true" }
                      append_action: OVERWRITE_IF_EXISTS_OR_ADD
EOF

The marker lands on the mirrored copy only:

$ kubectl -n shadow-lab logs -l version=v1 -c echo --tail=5 | grep sfxfix
{"variant":"echo-v1","authority":"echo:8080","x_request_id":"f6b434dc-b8d6-4eea-a7d2-5d7471dbf47e","x_shadow_run":null}

$ kubectl -n shadow-lab logs -l version=v2 -c echo --tail=5 | grep sfxfix
{"variant":"echo-v2","authority":"echo:8080","x_request_id":"f6b434dc-b8d6-4eea-a7d2-5d7471dbf47e","x_shadow_run":"true"}

A hand-written mirror policy does not inherit Istio's suffix override, which is why disable_shadow_host_suffix_append: true is set explicitly above. Drop that line and the authority becomes echo-shadow:8080 again.

The collision at echo-b is now resolvable, because the application forwards x-shadow-run alongside the trace headers:

$ kubectl -n shadow-lab logs "$BPOD" -c echo | grep f6b434dc-b8d6-4eea-a7d2-5d7471dbf47e
{"variant":"echo-b","path":"/ledger","x_request_id":"f6b434dc-...","x_shadow_run":"true"}
{"variant":"echo-b","path":"/ledger","x_request_id":"f6b434dc-...","x_shadow_run":null}

Istio propagates nothing across a hop for you. That forwarding is the application's job, exactly as it is for traceparent.

Two caveats. kubectl apply prints the standard warning that EnvoyFilter exposes internal implementation details and deserves care across upgrades, and it applies here: the patch is pinned to a generated vhost name and to Envoy's API shape. Second, give the shadow deployment its own ServiceAccount. It costs nothing and gives the mirrored traffic a distinct mTLS identity that AuthorizationPolicy can match, which a header cannot do on its own.

Comparing responses

Istio compares nothing. The mirrored response is read and discarded, so response diffing is code you write. The cheap version is a log join: both variants already log the request id and a hash of what they returned.

# compare.py, joining the two streams on x-request-id
def load(selector):
    pods = subprocess.run(["kubectl", "-n", NS, "get", "pods", "-l", selector,
        "-o", "jsonpath={range .items[*]}{.metadata.name}{'\\n'}{end}"],
        capture_output=True, text=True, check=True).stdout.split()
    out = defaultdict(list)
    for pod in pods:
        raw = subprocess.run(["kubectl", "-n", NS, "logs", pod, "-c", "echo"],
                             capture_output=True, text=True).stdout
        for line in raw.splitlines():
            if line.startswith("{"):
                rec = json.loads(line)
                if rec.get("x_request_id"):
                    out[rec["x_request_id"]].append(rec)
    return out

primary, shadow = load("app=echo,version=v1"), load("app=echo,version=v2")
for rid, precs in primary.items():
    srecs = shadow.get(rid)
    if not srecs:
        continue                      # not mirrored: expected below 100 percent
    p, s = precs[0], srecs[0]
    if p["body_sha256"] != s["body_sha256"]:
        fields = sorted(set(p["body"]) | set(s["body"]))
        print(rid, {f: (p["body"].get(f), s["body"].get(f))
                    for f in fields if p["body"].get(f) != s["body"].get(f)})

Run against 40 requests with the drift switched on:

primary requests observed : 40
  identical response      : 0
  DIFFERENT response      : 40
  never mirrored          : 0

field-level differences (primary -> shadow):
  18a45422-10d8-4d4e-bf4e-01bd7cd40f65  /orders/1
    order_date: '2026-09-07'  ->  '07/09/2026'

That is the seeded bug in echo-v2, a date format change. With the drift off the same 40 requests report 40 identical and the script exits 0. A date format flip is exactly the class of defect a rewrite introduces and a unit test with fixed fixtures misses, which is the argument for mirroring in the first place.

Two properties matter more than the code. Never-mirrored requests are reported separately rather than counted as failures, which is required once percentage drops below 100. And the diff is field by field, not "bodies differ", because a hash mismatch alone tells you nothing actionable.

One trap worth naming: reading kubectl logs during a rollout will happily read across two pod generations at once. An intermediate run here reported 404 requests and 116 mismatches that were entirely an artifact of old and new ReplicaSet pods both matching the label selector. Bound the join by time or by pod generation. Running it in your log backend over a time window is where this belongs once the lab is over, and it avoids the problem for free.

That is the last thing the lab is needed for, so tear it down:

k3d cluster delete istio-mirror

Before you point this at production traffic

The mirrored response is discarded. Writes the shadow performs are not. Give the shadow its own database, or run it read-only, and stub anything that sends mail, charges cards, or calls a third party. Verify that by reading the connection string the shadow actually starts with, not the config you believe it inherits.

The shadow also carries a full copy of production load, so size it accordingly or start at 5 to 10% with percentage. On EKS, x-request-id originates at the first Envoy in the path: an ALB or NLB in front of the ingress gateway does not generate one, so the gateway does. If a client supplies its own, decide deliberately whether to trust it before building a join on top of it.

Ambient mode was not tested here. Mirroring is L7 routing and ztunnel is L4 only, so a namespace without a waypoint proxy has nothing that can execute a mirror policy. Verify it on your own cluster before relying on it.

What Istio still does not hand you

Two of the three pieces come free. x-request-id joins a request to its mirror with no configuration at all, and Istio's default access log already prints it next to the authority. The third piece, a marker that survives the next hop, does not. HTTPMirrorPolicy exposes destination and percentage and nothing else, so the one field that resolves the collision sits behind an EnvoyFilter pinned to a generated vhost name and to Envoy's API shape.

That is the fragile part of this setup and the first thing to re-test after an Istio upgrade. It is also the thing to watch for in the release notes: request_headers_mutations has been on Envoy's RequestMirrorPolicy for several versions now, and the day Istio surfaces it on mirrors is the day the EnvoyFilter above can be deleted.