The 2 AM Phone Call Nobody Wants

Last year I got pulled into an incident at a previous gig. A junior engineer had pushed a .env file to a public repo. Within 11 minutes, automated scanners had grabbed the AWS access keys and started spinning up GPU instances for crypto mining. By the time we rotated credentials, the bill was already four figures.

This stuff happens constantly. The recent news cycle around government cloud keys ending up in public GitHub repos isn't unique — it's just the latest reminder that credential leaks remain one of the most boring, preventable, and absolutely catastrophic failures in our industry.

Let's talk about why this keeps happening, how to fix it when it does, and how to make it nearly impossible to happen on your team.

Why Smart People Leak Credentials

The root cause is almost never "the developer was careless." It's usually one of these:

• Local-only files drift into commits. Someone adds a config file to test something, forgets to gitignore it, and git add . sweeps it up.
• Credentials hardcoded "temporarily." A quick test that becomes permanent. We've all done it.
• History contamination. Someone removes the secret in a follow-up commit, thinking that's enough. It isn't — the secret lives forever in git history.
• Wrong repo, wrong remote. Pushing a private fork's contents to a public mirror by mistake.

The automated scanning bots that watch GitHub's public event stream are fast. Research from various security teams has shown leaked AWS keys can be exploited within minutes of being pushed. You don't have time to react manually.

Step 1: Detect Before You Push

The single most effective layer of defense is a pre-commit hook that scans for secrets. I've been using gitleaks across three projects this year and it has caught two genuine near-misses.

Install it locally:

# macOS
brew install gitleaks

# Or grab a release binary from the official repo:
# https://github.com/gitleaks/gitleaks/releases

Then wire it into a pre-commit hook. If you're using the pre-commit framework (https://pre-commit.com/), drop this into .pre-commit-config.yaml:

repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.18.0  # pin to a real release tag
    hooks:
      - id: gitleaks
        # runs on every staged file before commit completes

Then:

pre-commit install
# verify it works by trying to commit a fake AWS key in a test file

If you don't want a framework, a raw git hook works too. Create .git/hooks/pre-commit:

#!/usr/bin/env bash
# Block commits that contain detected secrets.
# Exits non-zero on a finding, which aborts the commit.

if ! gitleaks protect --staged --redact --verbose; then
  echo "Secrets detected. Commit aborted."
  echo "If this is a false positive, add it to .gitleaksignore"
  exit 1
fi

Make it executable with chmod +x .git/hooks/pre-commit and you're done.

Step 2: Detect Things That Already Slipped Through

Local hooks help, but they only protect commits made on machines where the hook is installed. You also need server-side scanning.

GitHub has built-in secret scanning, and it's free for public repos. For private repos you'll need to check the current pricing on their docs (https://docs.github.com/en/code-security/secret-scanning). If you can't or won't use that, a GitHub Action running on every push works fine:

name: secret-scan
on: [push, pull_request]

jobs:
  gitleaks:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0  # need full history for scanning
      - uses: gitleaks/gitleaks-action@v2
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

The fetch-depth: 0 is important — without it you only get the latest commit, which means a sneaky push that hides a secret behind an innocent-looking final commit will slip past you.

Step 3: When You've Already Leaked

Okay, the worst happened. Here's the order of operations, and the order matters:

1. Rotate the credential immediately. Not after you clean up the repo. Right now. Assume the credential is compromised the moment it touches a public commit, even for one second.

For AWS specifically:

# Deactivate the leaked key first (faster than deletion, reversible)
aws iam update-access-key \
  --user-name <username> \
  --access-key-id AKIA... \
  --status Inactive

# Then create a replacement
aws iam create-access-key --user-name <username>

# Once everything is migrated, delete the old key entirely
aws iam delete-access-key \
  --user-name <username> \
  --access-key-id AKIA...

2. Audit what the key did. Check CloudTrail for any API calls made by that access key ID since the leak. Look for unusual regions, unfamiliar resource creation, or IAM changes (attackers love adding their own users).

3. Purge the secret from git history. Just deleting the file in a new commit is not enough. You need to rewrite history with git filter-repo (the modern replacement for filter-branch):

# Install: pip install git-filter-repo
git filter-repo --invert-paths --path config/secrets.env

# Or to scrub a specific string across all history:
git filter-repo --replace-text <(echo 'AKIAIOSFODNN7EXAMPLE==>REDACTED')

Then force-push (after coordinating with your team — this rewrites shared history):

git push origin --force --all
git push origin --force --tags

4. Tell your security team. If this is a regulated environment, there are likely disclosure obligations. Don't try to bury it.

Prevention That Actually Sticks

A few habits that have served me well:

• Use short-lived credentials wherever possible. For AWS, that means IAM Roles, IAM Identity Center, or OIDC federation from your CI provider. A leaked role assumption token that expires in an hour is dramatically less dangerous than a permanent access key.
• Keep a strict .gitignore from day one. Add *.env, *.pem, *.key, credentials, .aws/, and any framework-specific secret paths. Commit the .gitignore before you commit anything else.
• Separate dev and prod credentials physically. Different accounts, different IAM principals, different machines if possible. Blast radius matters.
• Run scans against history periodically. New patterns get added to scanners all the time. Re-scanning old repos occasionally surfaces things that weren't detectable when they were committed.

The Boring Truth

Credential hygiene isn't glamorous and nobody gets promoted for it. But the cost of getting it wrong scales from "embarrassing Slack message" to "front page of Krebs" depending on what leaked and who you work for.

Set up the hooks. Use short-lived credentials. Assume any string that looks like a secret in your codebase has already been exfiltrated, and act accordingly. Future-you, woken up at 2 AM, will be grateful.