ost secrets scanners work the same way.

They maintain a list of regex patterns — one for AWS access keys, one for GitHub personal access tokens, one for Stripe keys, one for JWT headers — and they scan your code looking for matches. When a pattern fires, they report a finding. When it doesn't, they stay silent.

This works well for secrets that have distinctive, consistent formats. An AWS access key always starts with AKIA followed by 16 uppercase alphanumeric characters. A GitHub PAT has a recognisable prefix. A private key has a PEM header. Regex catches these reliably.

But it's only part of the problem. And the part it misses is exactly where real breaches happen.

This is the story of why I built a machine learning secrets detector — what the existing approaches get wrong, what ML adds, and what the combined system catches that neither approach catches alone.

The Two Failure Modes of Existing Tools

Before building anything, I spent time understanding where the leading tools fail. TruffleHog, detect-secrets, and Gitleaks are all excellent tools. They're also all vulnerable to the same two failure modes in different proportions.

Failure Mode 1: The Regex Gap

Regex-only scanners miss secrets that don't match a known pattern.

The most dangerous class of missed secrets is the generic hardcoded credential — a password, database URL, or internal API key that doesn't follow any publicly documented format because it was generated internally.

# No regex pattern catches this reliably
DB_PASSWORD = "Tr0ub4dor&3"
INTERNAL_API_KEY = "prod-backend-service-key-2019"
SMTP_PASSWORD = "companyname_mail_2018!"

These are real secrets. They're low entropy by the standards of a cryptographically random key. They don't match any known service's key format. A regex scanner walks past them silently.

This is not a theoretical concern. A significant proportion of credential exposures in real breaches involve exactly this type of secret — human-chosen passwords and internal tokens that were never designed to be detected by pattern matching.

Failure Mode 2: The Entropy False Positive Flood

Some tools compensate by flagging anything with high Shannon entropy — the reasoning being that secrets are random, and random strings have high entropy.

This is directionally correct and practically unusable in many codebases.

High-entropy strings that are not secrets appear constantly in normal code:

# UUID — high entropy, not a secret
session_id = "550e8400-e29b-41d4-a716-446655440000"

# SHA-256 hash — very high entropy, not a secret
expected_checksum = "d8e8fca2dc0f896fd7cb4cb0031ba249"

# Base64-encoded image data — extremely high entropy, not a secret
avatar_placeholder = "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJ..."

# Package integrity hash — high entropy, not a secret
integrity = "sha512-abc123def456..."

A pure entropy scanner flags all of these. In a Node.js project with a package-lock.json, an entropy scanner generates thousands of findings from integrity hashes alone. Engineers learn to ignore it within a week.

What ML Adds: Context-Aware Classification

The insight that drove the ML approach is that whether a string is a secret depends on context, not just the string itself.

d8e8fca2dc0f896fd7cb4cb0031ba249 is either a secret or a benign hash depending on what variable contains it. A human security engineer can tell these apart instantly by reading the surrounding code. A regex scanner and an entropy scanner cannot.

The question I asked was: can I teach a classifier to do what a human engineer does — look at the full context of a string and make a judgment about whether it's a secret?

The answer turned out to be yes, with a 26-dimensional feature vector that captures what a human eye actually processes when making that judgment.

Here's the comparison that drove the design:

The ML classifier doesn't replace regex — it adds a second layer. Known-format secrets (AWS keys, GitHub PATs, JWTs) are still caught by pattern flags that are part of the feature vector. Generic hardcoded credentials that no regex would catch are caught by the combination of entropy, character distribution, and — most importantly — the variable name context.

The Feature That Changed Everything: Key Name Risk

When I looked at feature importances after training the initial model, one feature stood above all others: key_name_risk, with an importance score of 0.28 out of 1.0.

That's the variable name. Not the value — the name of the variable holding the value.

This makes intuitive sense once you see it. These two lines of code contain the same string value:

checksum = "d8e8fca2dc0f896fd7cb4cb0031ba249"
password = "d8e8fca2dc0f896fd7cb4cb0031ba249"

A human engineer looks at these and immediately knows: the first is almost certainly a hash, the second is almost certainly a secret. The string itself carries no information about its purpose. The variable name carries everything.

I built a risk scoring function that assigns numerical scores to variable names based on their semantic association with sensitive data:

• password, passwd, secret, private_key → score 1.0
• api_key, token, credential, auth → score 0.9
• access_key, client_secret, bearer → score 0.85
• config, setting, value → score 0.1
• checksum, hash, version, id → score 0.0 The classifier learns to combine this score with the entropy and character distribution features to make decisions that mirror what a human reviewer would make.

The result: password = "abc123" gets flagged despite low entropy. checksum = "d8e8fca2dc0f896fd7cb4cb0031ba249" gets passed despite high entropy. Neither outcome is achievable with regex or entropy alone.

Why Random Forest, Not a Neural Network

When people hear "ML classifier," they often assume deep learning. I chose Random Forest deliberately, and it's worth explaining why.

Interpretability. A Random Forest tells you exactly why it made a decision — which features contributed how much to a particular classification. When an engineer asks "why did the scanner flag this?", I can show them the feature breakdown: high entropy (0.82), key name risk (0.95), matches JWT pattern (true). A neural network produces a probability with no explanation.

Size. The trained model is approximately 1MB as a pickle file. It ships with the tool, requires no internet connection, and adds negligible overhead to a scan. A neural network of sufficient sophistication would be orders of magnitude larger.

Training speed. The model trains on 6,000 labeled samples in seconds on a standard laptop CPU. No GPU required. This matters enormously for the retraining feature — teams can add their own training samples and retrain in their local environment without specialist infrastructure.

No overfitting on small data. With 6,000 training samples — which is small by deep learning standards — Random Forest generalises better than a neural network would. The structured feature engineering does the heavy lifting; the model itself doesn't need to be sophisticated.

The tradeoff is ceiling accuracy. A neural network operating on raw token sequences would likely achieve higher peak accuracy given sufficient data. But for a tool that needs to be deployable, explainable, and retrainable by a team without ML expertise, Random Forest is the right choice.

Synthetic Training Data: The Ethical Constraint

One early design decision shaped everything else: I would not train on real leaked secrets from public repositories.

The alternative — scraping GitHub for accidentally committed credentials and using them as positive training examples — is technically straightforward and has been done. It's also legally and ethically problematic. Those credentials belong to real people and organisations. Even if the data is technically public, using it to train a commercial tool raises questions I didn't want to answer.

Instead, I built a synthetic data generator that produces realistic examples of both secrets and benign high-entropy strings:

Secrets (label=1): Algorithmically generated AWS access keys, GitHub PAT formats, JWT structures, OpenAI key formats, Slack tokens, database connection strings, and — critically — synthetically generated "human-chosen" passwords that follow common patterns without being anyone's real password.

Benign (label=0): UUIDs, MD5 and SHA-256 hashes, version strings, base64-encoded image data fragments, color hex codes, package integrity hashes, lorem ipsum text fragments.

The synthetic approach has one significant advantage beyond ethics: I can generate unlimited training data and precisely control the class distribution. The 6,000 sample baseline can be scaled to 50,000 samples with a single command, which meaningfully improves model accuracy on edge cases.

The Three-Layer Detection Architecture

The final tool combines three detection mechanisms, each compensating for the others' weaknesses:

Layer 1 — Pattern matching flags. Sixteen binary features in the feature vector correspond to known secret formats (AWS, GitHub, JWT, OpenAI, Slack, database URLs, private key headers, and so on). These fire on known formats with near-zero false positives and form the backbone of high-confidence detections.

Layer 2 — Entropy and character analysis. Shannon entropy, character class ratios, repetition ratio, longest run of repeated characters — these features capture the statistical "shape" of a secret without requiring a specific format match. High entropy combined with a high-risk key name is a strong signal even when no pattern matches.

Layer 3 — Key name risk scoring. The variable name context that neither regex nor entropy captures. This is what allows the classifier to catch password = "simple123" despite its low entropy and lack of a recognisable format.

A finding is reported when the classifier's confidence exceeds a configurable threshold (default: 0.7). Findings include the confidence score, the matched pattern if any, and — for CI/CD integration — an exit code that can gate builds.

What This Actually Catches

I ran the tool against a collection of test cases designed to stress each approach. Results that illustrate the gap:

Caught by all approaches: AWS_KEY = "AKIAIOSFODNN7EXAMPLE" — known format, high entropy, high-risk key name. Every tool gets this.

Caught only by ML: DB_PASS = "Winter2019!" — low entropy, no known format, but the key name DB_PASS scores 1.0 and the classifier flags it at 89% confidence. Regex misses it. Entropy misses it.

False positive in entropy tools, not in ML: expected_hash = "d8e8fca2dc0f896fd7cb4cb0031ba249" — high entropy, but key name scores 0.0 and the ML classifier correctly passes it. A pure entropy scanner flags it; the ML classifier does not.

False positive in regex tools, not in ML: An internal test file with TEST_TOKEN = "fake-token-for-testing" annotated with # secrets-ignore — the suppression annotation is respected, and the low-entropy value combined with a test file context (another feature) keeps the confidence below threshold even without the annotation.

Where This Fits in a Security Programme

A secrets detector — even an ML-powered one — is one layer of a defence-in-depth approach, not a complete solution.

It catches secrets at the point of scanning. It doesn't prevent secrets from being created in the first place (that's developer education and code review). It doesn't rotate compromised credentials (that's incident response). It doesn't enforce secrets management policies (that's your secrets manager — Vault, AWS Secrets Manager, Azure Key Vault).

What it does well: systematically surface secret exposure across a codebase and git history, prevent new secrets from reaching the repository via pre-commit hooks, and provide a measurable baseline for "how many secret exposures exist in our codebase right now."

That baseline matters more than most teams realise — you can't improve what you can't measure.

The full source, including the feature extractor, trainer, and pre-commit hook, is at github.com/pgmpofu/secrets-detector.

Next up: a deep dive into the 26-dimensional feature vector — exactly what the model sees when it evaluates a candidate secret, and how each feature contributes to the final decision.