Your deploy pipeline went green at 2 p.m. By 3 p.m. the error rate graph is a staircase. The on-call engineer rolls back to the previous release and the fire goes out, but the commit that caused it is still somewhere in the two hundred changes that landed since Monday. Finding it by hand means checking out an old commit, building, running the reproduction, noting the result, and repeating. At ten minutes per iteration, that is thirty-three hours of worst-case work. Most teams do not have thirty-three hours. They have a blameless postmortem in the morning and a promise to “add more tests.”

Git bisect is the tool that turns that linear nightmare into a binary search. Instead of testing every commit, you test log2(N) commits. Two hundred changes becomes eight tests. And if you write a small script that can detect the bug automatically, git bisect run does those eight tests while you make coffee. This post is the exact workflow, the script template, the edge cases that trip people up, and the CI integration that turns bisect from a firefighting tool into a preventative one.

The shape of the failure

You know the pattern. A refactor landed on Tuesday. A dependency upgrade landed Wednesday. A feature branch merged Thursday. The rollback fixed the symptom, but the root cause could be any of those three, or an interaction between two of them. The ticket says “investigate regression,” which really means “find the commit so we can revert or patch it without reverting the entire release.”

The naive approach is to check out the commit from just before the release, verify it is good, then walk forward one commit at a time. That is O(n). With a large merge commit history, it is also confusing. Merge commits have two parents, and “forward” is not a single line. You end up retesting the same code paths through different history shapes, and half your time is spent figuring out which commit to check out next.

Git bisect solves this by treating commit history as a sorted array. It assumes there is a contiguous block of good commits, followed by a contiguous block of bad commits, and it finds the boundary using binary search. The only requirement is that you can identify one good commit and one bad commit. Everything in between is bisected automatically.

Manual bisect: the five-command workflow

Start by finding a known-good commit. Often this is the tag from the previous release. Then find a known-bad commit, usually the tip of main after the broken deploy.

# Start the session
git bisect start

# Mark the current HEAD as broken
git bisect bad

# Mark the last known good release
git bisect good v1.42.0

Git checks out a commit roughly halfway between the two. You build, run your reproduction, and tell Git the result.

# If the bug is present at this commit
git bisect bad

# If the bug is NOT present
git bisect good

Git picks the next midpoint. Repeat. Eight iterations later, Git prints the first bad commit hash, author, date, and message. Run git bisect reset to return to your original branch.

This alone saves hours, but it still requires you to sit at the keyboard and babysit the build. The real power is automation.

Automating the hunt with git bisect run

git bisect run executes a script at each midpoint and uses the exit code to decide whether the commit is good, bad, or should be skipped. The contract is simple:

• Exit 0: this commit is good (bug not present).
• Exit 1 through 127: this commit is bad (bug present).
• Exit 125: this commit should be skipped (build breaks, test is flaky, unrelated infrastructure is down).

The script must be deterministic. If it depends on a database that has since been migrated, an external API with rate limits, or a test that fails 10% of the time, bisect will either crash or produce garbage. The fix is to make the script self-contained.

Here is a template for a Node.js project that uses Docker to provide a predictable environment.

#!/usr/bin/env bash
set -euo pipefail

# Clean up any previous build artifacts
rm -rf node_modules dist

# Install dependencies using the lockfile from this commit
npm ci

# Build if needed
npm run build

# Start a throwaway Postgres container for integration tests
docker run -d --name bisect-pg-$BASHPID \
  -e POSTGRES_PASSWORD=bisect \
  -p 5433:5432 postgres:15-alpine > /dev/null

# Wait for Postgres to accept connections
sleep 2
until docker exec bisect-pg-$BASHPID pg_isready -U postgres > /dev/null 2>&1; do
  sleep 1
done

# Run migrations against the test port
DATABASE_URL=postgres://postgres:bisect@localhost:5433/postgres npm run db:migrate

# Run the specific reproduction test
# This test must return 0 when the bug is absent, non-zero when present
npm run test:reproduction || true
EXIT_CODE=$?

# Tear down the database
docker stop bisect-pg-$BASHPID > /dev/null
docker rm bisect-pg-$BASHPID > /dev/null

exit $EXIT_CODE

Save this as scripts/bisect.sh, make it executable, and run:

git bisect start
git bisect bad HEAD
git bisect good v1.42.0
git bisect run ./scripts/bisect.sh

Git will check out commits, run the script, parse the exit code, and print the bad commit. The $BASHPID trick ensures that if you run multiple bisects in parallel (for example, on different machines), their database containers do not collide.

What makes a good bisect script

The most common failure mode is not a broken script. It is a script that is too slow. If your full test suite takes fifteen minutes, and bisect needs eight iterations, that is two hours. The reproduction test should be as minimal as possible. It should compile only what is necessary, run only the one test that detects the bug, and avoid network calls to external services.

If the bug is an API returning the wrong status code, the script can be a ten-line curl command piped to jq.

#!/usr/bin/env bash
set -euo pipefail

npm ci
npm run build
npm start &
SERVER_PID=$!
sleep 3

# The bug: this endpoint should return 200, but after the bad commit it returns 500
STATUS=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:3000/api/health)

kill $SERVER_PID
wait $SERVER_PID 2>/dev/null || true

if [ "$STATUS" -eq 200 ]; then
  exit 0  # good
else
  exit 1  # bad
fi

This script is fast because it skips the entire test suite and checks only the symptom. If the server fails to start at a particular commit (maybe a dependency was temporarily broken), the script will exit non-zero and bisect will mark that commit as bad even though it is not the regression. That is where exit 125 matters.

Skipping commits that cannot be tested

Sometimes the build is broken at a midpoint. A dependency update broke npm install, or a migration file was temporarily malformed, or the TypeScript config had a syntax error for one commit. If your script returns 1, bisect assumes the bug is present and keeps searching in the older half of the history. If the bug is actually newer, bisect will wander into the weeds and report the wrong commit.

Exit 125 tells bisect “I cannot determine good or bad at this commit, skip it.” Git will treat the skipped commit as untested and narrow around it.

# Inside bisect.sh
if ! npm run build > /dev/null 2>&1; then
  echo "Build failed at this commit, skipping"
  exit 125
fi

The risk is that if you skip too many commits, bisect may be unable to isolate a single bad commit. It will report a range instead. That range is still useful, it narrows the search from two hundred commits to maybe five, but it means your history has too much noise. Treat frequent skips as a signal that your main branch is not reliably buildable, which is a problem worth fixing on its own.

Bisecting through merge commits

On a team that uses merge commits, bisecting can feel non-linear because merge commits have two parents. By default, git bisect will sometimes step into a feature branch that was merged, testing commits that never landed on main individually. This is technically correct but often irrelevant. You usually care about which merged branch introduced the bug, not which specific commit inside that branch.

Add --first-parent to bisect only along the main line.

git bisect start --first-parent
git bisect bad HEAD
git bisect good v1.42.0
git bisect run ./scripts/bisect.sh

This treats merge commits as single units. If a merged branch is bad, bisect identifies the merge commit itself as the first bad commit. That is usually what you want for a postmortem. The follow-up investigation happens inside that branch, but the boundary on main is clear.

Bisecting performance regressions

Not every regression is a crash or a wrong status code. Sometimes the regression is latency. The API still returns 200, but the p95 doubled between releases. Bisect works here too, as long as your script can measure the performance and return an exit code based on a threshold.

#!/usr/bin/env bash
set -euo pipefail

npm ci
npm run build
npm start &
SERVER_PID=$!
sleep 3

# Warmup
curl -s http://localhost:3000/api/heavy-endpoint > /dev/null

# Measure
curl -s -w "%{time_total}" -o /dev/null http://localhost:3000/api/heavy-endpoint > /tmp/bisect-timing.txt
LATENCY=$(cat /tmp/bisect-timing.txt)

kill $SERVER_PID
wait $SERVER_PID 2>/dev/null || true

# Threshold in seconds
if (( $(echo "$LATENCY < 0.5" | bc -l) )); then
  exit 0  # good
else
  exit 1  # bad
fi

Performance bisecting is more fragile than functional bisecting. Machine load, Docker networking, and background processes can add noise. Run the measurement three times and take the median if your latency budget is tight. If the noise is too high, return 125 and let bisect narrow around the uncertain commits.

Integrating bisect into CI

The highest-leverage use of bisect is not firefighting. It is prevention. When a test fails on main after a merge, most teams look at the merged PR and assume the PR is at fault. But if main was already broken by an earlier commit that did not trigger the test (maybe because the test was added later), blaming the most recent PR sends the wrong engineer on a wild goose chase.

A CI pipeline can automate bisect to find the true first bad commit. On main branch failure, the pipeline checks out the code, finds the last known green commit (from the previous successful CI run), and runs git bisect run with the failing test as the script. It posts the bad commit SHA to Slack or creates a ticket.

Here is a GitHub Actions job that does this.

  bisect-regression:
    runs-on: ubuntu-latest
    if: failure()
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Find last green commit
        id: last-green
        run: |
          # In practice, query your CI API or check git tags
          LAST_GREEN=$(git rev-list -n 1 --before="2 hours ago" origin/main)
          echo "commit=$LAST_GREEN" >> "$GITHUB_OUTPUT"

      - name: Run bisect
        run: |
          git bisect start
          git bisect bad HEAD
          git bisect good ${{ steps.last-green.outputs.commit }}
          git bisect run npm run test:ci -- --grep "regression-suite"

The --grep flag limits the test to the specific suite that failed, keeping bisect fast. You will want to cache node_modules between iterations if your test suite is heavy, but be careful: caching across commits can hide dependency changes. A clean install per iteration is slower but safer.

The edge cases that waste your time

Submodules. If your repo uses Git submodules, bisect will not automatically update them to the correct commit for the checked-out midpoint. Your script must run git submodule update --init --recursive before building, or you will test the wrong code.

Generated files. If your build pipeline generates files that are not in .gitignore, bisect may report a dirty working tree and refuse to switch commits. Run git clean -fdx at the start of every bisect script to ensure a pristine state.

Environment variables. If your reproduction depends on an environment variable that changes between commits (for example, a feature flag that was toggled manually during the bisected period), the script must set that variable explicitly. Bisect assumes the only variable is the code itself.

Non-deterministic tests. A flaky test that fails 5% of the time will eventually return 1 on a good commit and send bisect down the wrong path. If you know a test is flaky, run it multiple times inside the script and return 125 unless all runs agree.

FAILS=0
for i in {1..3}; do
  npm run test:reproduction || FAILS=$((FAILS + 1))
done

if [ "$FAILS" -eq 0 ]; then
  exit 0
elif [ "$FAILS" -eq 3 ]; then
  exit 1
else
  exit 125
fi

The takeaway

Finding a regression by hand is linear, error-prone, and demoralizing. Git bisect is logarithmic, mechanical, and automatable. The investment is one reproduction test script. The return is a ten-minute automated search that finds the exact commit, even across merge commits, even in a noisy history.

Write the script before you need it. Keep it fast, self-contained, and deterministic. Add --first-parent if your team uses merge commits. Return 125 when the build breaks. And integrate it into CI so the next time main goes red, the pipeline tells you which commit broke it before the standup starts.

The next time production breaks after a multi-day sprint, you will not be hand-checkout-testing two hundred commits. You will be running one command and reading the bad commit hash while your coffee brews.

A note from Yojji

The teams that ship reliably are not the ones that never break production. They are the ones that can pinpoint the breaking change, revert or patch it precisely, and prevent the same class of failure from recurring. Automated bisect workflows, deterministic reproduction scripts, and CI integrations that catch regressions at the commit boundary are standard practice in the backend engineering culture Yojji builds with its partners.

Yojji is an international custom software development company founded in 2016, with teams across Europe, the US, and the UK. They specialize in the JavaScript ecosystem (React, Node.js, TypeScript), cloud platforms (AWS, Azure, Google Cloud), and the kind of disciplined debugging workflows that turn production fires into routine maintenance.