In the previous article, everything happened in the inner loop. We connected a laptop to a live Kubernetes cluster, had a coding agent build and validate a feature against real dependencies, and debugged a cross-service bug with traffic recording and local overrides. All of that was you (or your agent), working from your laptop before the commit.

But code ships from a pull request. The moment you open a PR, you want the same guarantees applied automatically, for every PR and every developer on the team.

The outer loop applies those same guarantees after you push. CI creates a sandbox for every GitHub PR, with an optional live preview for reviewers. Even traditionally scary PRs with database schema changes get a disposable database branch instead of a dedicated environment. Tests run inside the cluster against the PR's code, and closing or merging the PR cleans everything up.

From the inner loop to the outer loop

The inner loop ends at the commit; the outer loop starts when the pull request opens.

There are three building blocks that make the outer loop work at scale.

Sandbox template

A sandbox template is a sandbox spec with variables, checked into the repo right next to the code. CI fills in the name, image, and PR number at apply time.

Scope the sandbox to the service the PR changed. Other microservices, databases, and message queues stay shared from the baseline. With one changed service per PR, fifty sandboxes add about fifty pods to one cluster. Fifty full environments duplicate the entire stack fifty times.

This walkthrough focuses on one changed service, but a sandbox can contain multiple forks. In a monorepo, CI can detect which services changed, build each image, and include all affected services in the same PR sandbox. The cost then follows the number of changed services, not the size of the full environment.

The GitHub Actions lifecycle

One GitHub Actions workflow owns the PR lifecycle. On open, synchronize, or reopen, it builds the PR image, creates or updates the sandbox, runs the in-cluster test, and posts one sticky comment with the preview details. On close, it deletes the PR-derived sandbox and updates that comment. There is no separate GitHub App in this setup.

Jobs

Jobs put the tests inside the cluster, where they can use normal Kubernetes DNS names with sandbox routing applied. A Job combines a script, a pool of runner pods in your cluster, and a routing context that adds the sandbox's routing key to every request.

Prerequisites

This builds on the same environment as the previous article: HotROD in the hotrod namespace, the Signadot operator installed, and the CLI authenticated. On top of that, you'll need:

• A Signadot account and the Signadot CLI (signadot --version)
• A GitHub account and a fork of the HotROD demo repository you control
• A Docker Hub account with a personal access token (CI pushes a per-PR image)
• For the database section, a Neon account, neonctl installed and authenticated, and psql
• jq for inspecting output

Part 1: A sandbox for every pull request

Open a PR and a sandbox running that PR's code appears in the shared cluster, with its status and routing details attached to the PR for reviewers. A preview URL already includes the routing context, but reviewers can instead activate the sandbox with the browser extension. The complete workflow, sandbox template, Job template, and Job Runner Group spec are checked into the repository.

File 1: the sandbox template

The HotROD repo already ships the sandbox template at .signadot/sbx-gh-template.yaml:

name: "@{name}"
spec:
  description: PR sandbox for the HotROD route service
  cluster: "@{cluster}"
  labels:
    signadot/github-repo: "@{github-repo}"
    signadot/github-pull-request: "@{github-pr}"
    branch: "@{branch}"
  ttl:
    duration: 2d
    offsetFrom: updatedAt
  forks:
    - forkOf:
        kind: Deployment
        name: route
        namespace: "@{namespace}"
      customizations:
        images:
          - container: hotrod
            image: "@{image}"
  defaultRouteGroup:
    endpoints:
      - name: hotrod
        target: "http://frontend.@{namespace}.svc:8080"

The sandbox spec forks the route deployment and overrides its image. The @{...} placeholders are template variables that CI fills in, including the cluster and namespace.

File 2: the GitHub Actions workflow

The complete GitHub Actions workflow is checked in at .github/workflows/sandbox.yaml. It reads the Signadot organization, API key, and Docker Hub credentials from repository secrets instead of hardcoding them. Its lifecycle is:

on:
  pull_request:
    types: [opened, synchronize, reopened, closed]

permissions:
  contents: read
  pull-requests: write

env:
  SANDBOX_NAME: hotrod-pr-${{ github.event.pull_request.number }}

jobs:
  create-or-update:
    if: github.event.action != 'closed'
    # Build route-pr-<sha>, apply sandbox, run Job, post sticky comment.

  cleanup:
    if: github.event.action == 'closed'
    # Run: signadot sandbox delete "${SANDBOX_NAME}"

The workflow installs the Signadot CLI, builds and pushes a per-commit image, applies the sandbox template, submits the route Job, and posts the result. The sandbox runs the per-commit image built from this PR.

The image tag includes the commit SHA, but the sandbox name comes from the PR number. Every push updates the same sandbox for the lifetime of the PR instead of creating another sandbox for every commit. hotrod-pr-<number> identifies the long-lived PR sandbox, while route-pr-<sha> identifies the specific route-service image running inside it.

Sandbox names cap out at 30 characters. This repository fits comfortably as hotrod-pr-<number>. For longer repository names, truncate the readable prefix and append a short deterministic hash so the name remains stable and collision-resistant for the lifetime of the PR.

The Signadot API key and registry credentials are repository secrets. The cluster and HotROD namespace are repository variables. The workflow itself is the integration; no separate App installation is required.

Open a real PR

Push a small, visible change to a branch (I tweaked the route service response), then run:

gh pr create --head demo/route-tweak \
  --title "Tweak route service response" \
  --body "Demo PR for sandbox-per-PR workflow"

The workflow kicks off and builds the image. The Signadot dashboard then shows a sandbox named hotrod-pr-<number>, with a forked route service running the commit-specific route-pr-<sha> image and wired into the shared cluster. Fifty open PRs create fifty sandboxes, each about one pod.

The workflow follows this lifecycle:

Back on the PR page, the workflow's sticky comment reports the sandbox name, commit-specific image, preview URL, routing key, and passing Job. Whether reviewers use the preview or activate the sandbox with the browser extension, they see this PR's version of the app running against real upstream and downstream services.

Cleanup is automatic

Once you merge or close the PR, the workflow receives the closed event and runs signadot sandbox delete for the stable PR-derived name. It should disappear from the dashboard shortly afterward; the exact delay depends on event delivery and cleanup time. The workflow updates the same sticky comment to show that cleanup completed.

Database isolation with Neon branching

So far, every sandbox has shared the baseline database, which works for most changes. A schema-changing PR can't run a migration there without breaking the baseline and every other sandbox. Spinning up a full environment avoids the collision but duplicates the rest of the stack.

For this example, I'll use Neon, a serverless Postgres service with copy-on-write branching that's similar to Git branches, but for your database.

A Neon branch is a fully isolated Postgres environment. It starts with the parent's schema and data, along with its databases, roles, and extensions. Changes made within the branch remain isolated from the parent and every other branch. Branch creation takes seconds, and the branch shares storage with its parent until you write to it. A Signadot resource plugin can create a branch when a sandbox starts and destroy it when the sandbox is deleted.

Let's build a small service to demonstrate how this works. The service is a deliberately tiny users API with about 50 lines of Express, one table, and three endpoints. It reads its Postgres connection string from a single DATABASE_URL environment variable, which the sandbox will override later. The Signadot Neon branching example contains a complete version of the service, Docker build, manifests, resource plugin, and sandbox spec; the snippets below keep the project fixed to main and neondb to make the lifecycle easier to follow.

One table, two seed rows:

-- schema.sql

CREATE TABLE IF NOT EXISTS users (
  id SERIAL PRIMARY KEY,
  name TEXT NOT NULL,
  email TEXT UNIQUE NOT NULL
);

INSERT INTO users (name, email) VALUES
  ('Ada Lovelace', 'ada@example.com'),
  ('Grace Hopper', 'grace@example.com')
ON CONFLICT (email) DO NOTHING;

Create a Neon project and load the schema into its main branch:

neonctl projects create --name users-demo

psql "$(neonctl connection-string main \
  --project-id <project-id> \
  --database-name neondb)" \
  -f schema.sql

Note the project ID from the output.

Deploy it to the cluster

The Kubernetes side has one Deployment and one Service:

# k8s/users-service.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: users-service
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: users-service
  template:
    metadata:
      labels:
        app: users-service
      annotations:
        sidecar.signadot.com/inject: "true"
    spec:
      containers:
        - name: users-service
          image: users-service:demo
          imagePullPolicy: Never # The image is side-loaded into kind.
          ports:
            - containerPort: 3000
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: users-db-credentials
                  key: DATABASE_URL
---
apiVersion: v1
kind: Service
metadata:
  name: users-service
  namespace: default
spec:
  selector:
    app: users-service
  ports:
    - port: 3000
      targetPort: 3000

Build the image and load it into the cluster. I'm using kind, but you can alternatively push it to a registry. Next, create the two secrets and deploy:

docker build -t users-service:demo .
kind load docker-image users-service:demo --name <kind-cluster-name>

# The baseline's connection string (Neon main branch).
kubectl create secret generic users-db-credentials \
  --from-literal=DATABASE_URL="$(neonctl connection-string main \
    --project-id <project-id> \
    --database-name neondb)"

# Neon API key for the plugin. Plugin runners execute in the signadot namespace.
kubectl -n signadot create secret generic neon-api-credentials \
  --from-literal=NEON_API_KEY=<your-neon-api-key>

kubectl apply -f k8s/users-service.yaml
kubectl get pods -l app=users-service # 2/2 Running: app + devmesh sidecar

The resource plugin

A resource plugin has two lifecycle scripts with typed inputs and outputs. One runs when a sandbox that requests the resource is created, and one runs when it's deleted. The full plugin is:

# neon-branch-plugin.yaml
name: neon-branch
spec:
  description: Creates and deletes a Neon database branch per sandbox
  runner:
    image: node:20-alpine
    namespace: signadot
    podTemplateOverlay: |
      spec:
        containers:
          - name: main
            env:
              - name: NEON_API_KEY
                valueFrom:
                  secretKeyRef:
                    name: neon-api-credentials
                    key: NEON_API_KEY
  create:
    - name: createbranch
      inputs:
        - name: project-id
          valueFromSandbox: true
          as:
            env: NEON_PROJECT_ID
      script: |
        #!/bin/sh
        set -e
        npm install -g neonctl
        # Branch named after the sandbox (Neon branch names: no hyphens).
        SAFE_NAME=$(echo "${SIGNADOT_SANDBOX_NAME}" | tr -d '-')
        BRANCH_NAME="sandbox${SAFE_NAME}"
        neonctl branches create \
          --project-id "${NEON_PROJECT_ID}" \
          --name "${BRANCH_NAME}" \
          --parent main
        CONNECTION_STRING=$(neonctl connection-string "${BRANCH_NAME}" \
          --project-id "${NEON_PROJECT_ID}" \
          --database-name neondb)
        mkdir -p /outputs
        echo -n "${BRANCH_NAME}" > /outputs/branch-name
        echo -n "${CONNECTION_STRING}" > /outputs/connection-string
      outputs:
        - name: branch-name
          valueFromPath: /outputs/branch-name
        - name: connection-string
          valueFromPath: /outputs/connection-string
  delete:
    - name: deletebranch
      inputs:
        - name: project-id
          valueFromSandbox: true
          as:
            env: NEON_PROJECT_ID
        - name: branch-name
          valueFromStep:
            name: createbranch
            output: branch-name
          as:
            env: BRANCH_NAME
      script: |
        #!/bin/sh
        set -e
        npm install -g neonctl
        neonctl branches delete "${BRANCH_NAME}" --project-id "${NEON_PROJECT_ID}"

The runner is a plain node:20-alpine pod in the signadot namespace with the Neon API key injected from the secret we created. The create step takes the project ID from the sandbox spec, names a branch after the sandbox (Signadot injects SIGNADOT_SANDBOX_NAME automatically), creates it with neonctl, and writes the branch name and connection string to files under /outputs. Writing those files publishes the step outputs. The delete step reads the branch name from the create step's output via valueFromStep and removes it.

Install it:

signadot resourceplugin apply -f neon-branch-plugin.yaml

The sandbox spec

The sandbox spec requests the branch and injects its connection string into the fork:

# users-sandbox.yaml
name: "@{sandbox-name}"
spec:
  description: users-service fork with an isolated Neon database branch
  cluster: "@{cluster}"
  resources:
    - name: usersDb
      plugin: neon-branch
      params:
        project-id: "@{neon-project-id}"
  forks:
    - forkOf:
        kind: Deployment
        namespace: default
        name: users-service
      customizations:
        env:
          - name: DATABASE_URL
            valueFrom:
              resource:
                name: usersDb
                outputKey: createbranch.connection-string
  defaultRouteGroup:
    endpoints:
      - name: users-api
        target: http://users-service.default.svc:3000

The resources block requests a neon-branch from the plugin, so the branch gets created before the fork starts. The fork's DATABASE_URL comes from the plugin's createbranch.connection-string output, overriding the secret-backed value used by the baseline. The application still reads the same environment variable; only the value changes.

Create the sandbox and watch the branch appear

signadot sandbox apply -f users-sandbox.yaml \
  --set sandbox-name=schema-change \
  --set cluster=<your-cluster-name> \
  --set neon-project-id=<project-id>

Immediately afterward:

neonctl branches list --project-id <project-id>

Within seconds, the sandboxschemachange branch exists as a copy-on-write clone with production-shaped data, without waiting for a nightly dump restore. In the dashboard, the sandbox's Resources tab shows the plugin run steps and their outputs.

Prove the isolation

Write a user through the sandbox. The preview URL routes to the forked service, which talks to the branch. Grab an API key from the dashboard:

curl -s -X POST \
  -H "signadot-api-key: <key>" \
  -H "Content-Type: application/json" \
  -d '{"name":"Sandbox-Only User","email":"sandbox@example.com"}' \
  "https://users-api--schema-change.preview.signadot.com/users" | jq

Read it back through the sandbox to confirm it's there:

curl -s \
  -H "signadot-api-key: <key>" \
  "https://users-api--schema-change.preview.signadot.com/users" | jq

Now read from the baseline via signadot local connect:

curl -s "http://users-service.default.svc:3000/users" | jq

It's not there. Same cluster, same service name, two completely isolated databases. The PR can run its migration, mutate data, or drop tables; the baseline never notices.

Cleanup is automatic here too

signadot sandbox delete schema-change

neonctl branches list --project-id <project-id>

The plugin's delete step removed the branch, leaving just main. Nothing to remember and no cost leaking from forgotten branches.

Jobs running tests inside the cluster

CI created the sandbox from outside the cluster. The tests still need to run inside, use normal service names, and carry the sandbox routing context. Signadot Jobs handle that.

Create a Job Runner Group

A Job Runner Group is a pool of runner pods in your own cluster. You pick the image, namespace, and number of pods. Tests run on your infrastructure, next to your services, instead of on a GitHub-hosted VM reaching in from the internet. The repository includes the complete Job Runner Group spec.

# .signadot/jobrunnergroup.yaml
name: hotrod-tests
spec:
  cluster: "@{cluster}"
  namespace: signadot-jobs
  image: golang:1.22-bookworm
  jobTimeout: 15m
  scaling:
    manual:
      desiredPods: 1

kubectl create namespace signadot-jobs
signadot jobrunnergroup apply \
  --set cluster=<your-cluster-name> \
  -f .signadot/jobrunnergroup.yaml
signadot jobrunnergroup list # Wait until ready.

The Job template

The repository also includes the Job template at .signadot/job-route-test.yaml:

spec:
  namePrefix: route-api-test-
  runnerGroup: hotrod-tests
  routingContext:
    sandbox: "@{sandbox}"
  script: |
    #!/bin/bash
    set -euo pipefail
    git clone --depth 1 --single-branch \
      --branch "@{branch}" \
      "https://github.com/@{repo}.git" \
      /tmp/hotrod
    cd /tmp/hotrod
    TEST_ROUTE_ADDR="route.@{namespace}.svc:8083" \
      go test -run '^TestRouteClient$' -v ./services/route 2>&1 \
      | tee /tmp/route-test.log
    grep -q -- '--- PASS: TestRouteClient' /tmp/route-test.log
    echo "PASS: route gRPC test reached sandbox @{sandbox}"
  uploadArtifact:
    - path: /tmp/route-test.log

routingContext.sandbox adds the sandbox routing key to requests from the Job. HotROD's existing Go test connects to the ordinary gRPC address route.<namespace>.svc:8083, and routing sends that connection to the forked version. The test does not need a per-PR hostname.

Submit and attach

With a target sandbox running (reuse a PR sandbox from Part 1, or create one from the template), run:

signadot job submit -f .signadot/job-route-test.yaml \
  --set sandbox=hotrod-pr-<number> \
  --set branch=demo/route-tweak \
  --set repo=<owner>/hotrod \
  --set namespace=<hotrod-namespace> \
  --attach

With --attach, the logs stream straight to your terminal until the Job finishes, and the Job's exit code propagates to the caller. Logs and artifacts land in the dashboard, attached to the run. The test's captured response body is available there as a downloadable artifact.

Slot it into CI

Add this step to the create-or-update job from earlier:

- name: Run API test against sandbox
  run: |
    signadot job submit --attach \
      --set sandbox=${SANDBOX_NAME} \
      --set branch=${{ github.head_ref }} \
      --set repo=${{ github.repository }} \
      --set namespace=${HOTROD_NAMESPACE} \
      -f .signadot/job-route-test.yaml

signadot job submit --attach propagates the Job's exit code to GitHub Actions, so a failed in-cluster test fails the PR check. When the PR closes, the workflow deletes the sandbox, and resource plugins delete any database branches they created.

Conclusion

Every PR now gets the same live-cluster validation as the inner loop: one stable sandbox, a commit-specific image, in-cluster tests, and a preview reviewers can open before merge. Schema-changing PRs get their own Neon branch, so migrations and test data never touch the baseline. With one changed service per PR, fifty open PRs add roughly fifty pods to the shared cluster; closing a PR removes its pod and database branch.