惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

月光博客
月光博客
SecWiki News
SecWiki News
爱范儿
爱范儿
Jina AI
Jina AI
钛媒体:引领未来商业与生活新知
钛媒体:引领未来商业与生活新知
Apple Machine Learning Research
Apple Machine Learning Research
Vercel News
Vercel News
S
SegmentFault 最新的问题
freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More
GbyAI
GbyAI
V
V2EX
博客园 - 司徒正美
WordPress大学
WordPress大学
Y
Y Combinator Blog
B
Blog RSS Feed
H
Help Net Security
C
Check Point Blog
P
Proofpoint News Feed
Google DeepMind News
Google DeepMind News
Application and Cybersecurity Blog
Application and Cybersecurity Blog
B
Blog
Help Net Security
Help Net Security
罗磊的独立博客
让小产品的独立变现更简单 - ezindie.com
让小产品的独立变现更简单 - ezindie.com
H
Heimdal Security Blog
大猫的无限游戏
大猫的无限游戏
Security Latest
Security Latest
Cisco Talos Blog
Cisco Talos Blog
Blog — PlanetScale
Blog — PlanetScale
A
Arctic Wolf
T
The Blog of Author Tim Ferriss
P
Proofpoint News Feed
The Register - Security
The Register - Security
F
Fortinet All Blogs
S
Securelist
Microsoft Security Blog
Microsoft Security Blog
O
OpenAI News
P
Privacy & Cybersecurity Law Blog
C
Cybersecurity and Infrastructure Security Agency CISA
The GitHub Blog
The GitHub Blog
云风的 BLOG
云风的 BLOG
AWS News Blog
AWS News Blog
H
Hackread – Cybersecurity News, Data Breaches, AI and More
I
InfoQ
T
Threat Research - Cisco Blogs
Martin Fowler
Martin Fowler
D
Docker
C
Cisco Blogs
C
CERT Recently Published Vulnerability Notes
CTFtime.org: upcoming CTF events
CTFtime.org: upcoming CTF events

DEV Community

Authentication Security Deep Dive: From Brute Force to Salted Hashing (With Java Examples) Why AI Systems Don’t Fail — They Drift Spilling beans for how i learn for exam😁"Reinforcement Learning Cheat Sheet" I Replaced Chrome with Safari for AI Browser Automation. Here's What Broke (and What Finally Worked) How Python Borrows Other People's Work The $40 Architecture: Processing 1 Billion API Requests with 99.99% Uptime Vibe Coding: A Workflow Guide (From Zero to SaaS) Most webhook security guides protect the wrong side. The scary part is delivery. Headless CMS for TanStack Start: Build a Blog with Cosmic EU Age Verification App "Hacked in 2 Minutes" — What Actually Happened Comfy Cloud’s delete function does not actually remove files Running AI Models on GPU Cloud Servers: A Beginner Guide Event-driven media intelligence with AWS Step Functions and Bedrock I scored 500 AI prompts across 8 quality dimensions — here's what broke How to Call Google Gemini API from Next.js (Free Tier, No Backend Needed) The Portal Protocol: Reclaiming Human Connection in the Age of AI How to Fix Your Team's Scattered Knowledge Problem With a Self-Hosted Forum Intro to tc Cloud Functors: A Graph-First Mental Model for the Modern Cloud Designing Multi-Tenant Backends With Both Ownership and Team Access I Built a Neumorphic CSS Library with 77+ Components — Here's What I Learned PostgreSQL Performance Optimization: Why Connection Pooling Is Critical at Scale Cómo construí un SaaS multi-rubro para gestionar expensas en Argentina con FastAPI + Vue 3 🚀 I Built an Ethical Hacking Scanner Tool – Open Source Project I Replaced /usage and /context in Claude Code With a Single Statusline A Pythonic Way to Handle Emails (IMAP/SMTP) with Auto-Discovery and AI-Ready Design I Collected 8.9 Million Polymarket Price Points — Here's What I Found About How Markets Really Move EcoTrack AI — Carbon Footprint Tracker & Dashboard Everyone's Using AI. No One Agrees How. 5 self-hosted ebook managers worth trying in 2026 Building Your First AI Agent with LangChain: From Chatbot to Autonomous Assistant Common SOC 2 Failures (Real World) Stop Vibe-Checking Your AI App: A Practical Guide to Evals How to Use SonarQube and SonarScanner Locally to Level Up Your Code Quality Your Next To-Do App Is Dead — I Replaced Mine with an OpenClaw AI Sign a Nostr event in 60 lines of Python using coincurve — no nostr-sdk, no nbxplorer, no rust toolchain ITGC Audit Explained Like You’re in Big 4 Patch Tuesday abril 2026: Microsoft parcha 163 vulnerabilidades y un zero-day en SharePoint Stop scraping everything: a better way to track competitor price changes Listing on MCPize + the Official MCP Registry while routing payments OUTSIDE the marketplace — how I kept 100% of my x402 revenue Building an AI-Powered Risk Intelligence System Using Serverless Architecture Why We Ripped Function Overloading Out of Our AI Toolchain Testing AI-Generated Code: How to Actually Know If It Works SaaS Churn Is Killing Your Business. Here Is What to Do About It (Without a Support Team) The Speed of AI Is No Longer Linear - And Self-Improving Models Are Why How to Implement RBAC for MCP Tools: A Practical Guide for Engineering Teams From Standard Quote to Persuasive Proposal: AI Automation for Arborists I built a CLI that scaffolds complete multi-tenant SaaS apps Axios CVE-2025–62718: The Silent SSRF Bug That Could Be Hiding in Your Node.js App Right Now The dashboard that ended our friendship Data Pipelines Explained Simply (and How to Build Them with Python) The Hidden Cost of AI Systems Nobody Talks About. undefined vs undeclared, and how typeof behaves Switching from file-based jobs to NATS/Kafka in Rust without changing code io_uring Adventures: Rust Servers That Love Syscalls Why Agentic AI is Killing the Traditional Database The POUR principles of web accessibility for developers and designers Quantum Neural Network 3D — A Deep Dive into Interactive WebGL Visualization How To Install Caveman In Codex On macOS And Windows Automation Pipeline Reliability: Why Your Workflow Breaks When Nobody Is Watching I Built an 'Open World' AI Coding Agent — It Works From ANY Folder From Freelancing to Product: A Tech Service Company's SaaS Transformation China's AI Giants: Adding Tencent Hunyuan & ByteDance Doubao to AI University (74 Providers) On the Vibe Coders and Their Lies clerk: Auto-Summarize Your Claude Code Sessions AI Weekly — 2026/04/10–04/17 | The Model Lockdown Is Here, but the Toolchain Is the Real Battleground AI 週報 — 2026/04/10–2026/04/17 模型封鎖潮來了,但工具鏈才是真戰場 Maybe this is how Open-Source apps are born... 🚀 Fine-Tune LLMs with LoRA and QLoRA: 2026 Guide tRPC v11 + Next.js App Router: End-to-End Type Safety Without the Boilerplate ShadCN UI in 2026: Why I Stopped Installing Component Libraries and Started Owning My Components SaaS Billing in React Server Components: Stripe + Supabase Without a Single `useEffect` Join our DEV Weekend Challenge — $1,000 in Prizes Across TEN winners! Submissions Due April 20 at 6:59 AM UTC. Implementing FSRS Spaced Repetition in Flutter + Supabase — Adding Memory Science to an AI Learning App "I Texted My Localhost From the Train — Claude Code Fixed the Bug Before I Got Home" I Built a Sales Prep AI and It Went Deeper Than Expected Design to Code #2: One JSON, Eleven Outputs Solving the 100M-Row Problem: A Summary Table Pattern for High-Volume Push Notification Logs Flutter Web With Wasm: What Actually Changes For Developers I Built 50 Royalty-Free Soundtracks for My Side Project in a Weekend Using AI Music Generation The Vibe Coding Security Checklist: 7 Things to Check Before You Ship Stop Letting Googlebot Guess Fix Your React App's SEO Right Desconstruindo o Streaming do LinkedIn: Como Criar um Engine de Extração de Vídeo de Alta Performance com HLS e FFmpeg (EDA Part-1) EDA (Exploratory Data Analysis) Explained With Real Life — Why Looking at Your Data Is the Most Important Step in Machine Learning Brand Relationship Management at Scale: Our 4-Touch Outreach System for 200+ Brands Why String.fromEnvironment() Might Return an Empty String in Dart JGuardrails 1.0.0 — Hardening Java LLM Apps Against Jailbreaks, Toxicity, and Prompt Injection Plan and Schedule a Full Week of Threads Content From One Claude Conversation Coding Cat Oran Ep3, Five Tables Changed Everything Updated: BFF Pattern I'm done watching freelancers get buried by 200 proposals. So I'm building the alternative. This is my first post BFS Algorithm in Java Step by Step Tutorial with Examples Tracking LLM Pricing Monthly: An Open Dataset for 22 AI Models How We Measure Content ROI on a Comparison Site: Revenue Attribution Without Perfect Data Introducing Nova AI Ops: The AI-Native Operating System for SRE Teams I built a free desktop video downloader for Windows — Grabbit How Talkie OCR Helps Vision-Impaired & Dyslexic Users Read the World Around Them VRCFaceTracking安装和iPhone面捕配置教程,有bug Even CrowdStrike Can't See Your Agents The Automation Gold Rush: What n8n Workflows and Claude Are Opening Up for Developers Right Now
API Error Codes: A Test Suite Pattern I Stole from Stripe
Smeet Gohel · 2026-06-26 · via DEV Community

Read Stripe's API reference for an hour and you'll notice every endpoint has a complete enumerated list of error codes with example payloads. Then look at your own API.

The contrast is hard to ignore.

Stripe's API documentation treats errors as first-class citizens. Every endpoint clearly documents not only the happy path but also every expected failure, complete with structured error codes, descriptions, HTTP status codes, and example responses.

Now compare that to many APIs in production.

You might find a generic list of HTTP status codes somewhere in the documentation, but business-specific errors are often buried inside controller logic, scattered across wiki pages, or simply undocumented. The test suite isn't much better—there are dozens of happy-path tests, but only a handful of negative scenarios.

That imbalance creates problems for everyone involved:

  • Developers don't know which errors are expected.
  • Frontend teams can't reliably handle failures.
  • QA engineers miss important negative cases.
  • Refactoring accidentally changes error responses without anyone noticing.

A few years ago, I borrowed a simple idea from Stripe's documentation and turned it into a testing strategy.

Instead of treating error responses as exceptions, we created an error-code catalog and made it the foundation of our negative test suite.

The result wasn't just better API error code testing—it also improved documentation, simplified maintenance, and made API contracts far more consistent.

Here's how the pattern works.


Why Error Responses Are Part of the API Contract

When people think about API testing, they naturally focus on successful responses.

Typical assertions include:

  • HTTP 200 OK
  • HTTP 201 Created
  • Correct JSON payload
  • Required fields
  • Business calculations

Negative testing often gets much less attention.

Maybe there are a few tests for:

  • Invalid authentication
  • Missing required fields
  • Unknown resources

Beyond that, many APIs rely on manual testing or hope that the framework handles everything correctly.

The problem is that real users encounter failures just as often as successful requests.

Examples include:

  • Customer account locked
  • Payment declined
  • Coupon expired
  • Inventory unavailable
  • Duplicate registration
  • Subscription canceled
  • Rate limit exceeded

These aren't exceptional scenarios.

They're expected business outcomes.

Treating them as first-class API contracts changes how you design both documentation and tests.


The Error-Code Catalog as a Test Input

The first step is creating a centralized catalog of every business error the API can intentionally return.

A simplified example might look like this:

errors:
  USER_NOT_FOUND:
    httpStatus: 404
    message: User not found

  EMAIL_ALREADY_EXISTS:
    httpStatus: 409
    message: Email already exists

  INVALID_TOKEN:
    httpStatus: 401
    message: Invalid authentication token

  PAYMENT_DECLINED:
    httpStatus: 402
    message: Payment declined

  ORDER_ALREADY_SHIPPED:
    httpStatus: 409
    message: Order cannot be modified

This catalog becomes far more than documentation.

It becomes an executable specification.

Instead of asking:

"What errors should this endpoint return?"

the answer already exists in one authoritative location.

Every new business error must be added here before it reaches production.

That single requirement dramatically improves consistency.


Why a Catalog Helps

Without a catalog:

  • Documentation drifts.
  • Tests become incomplete.
  • Frontend teams discover errors by accident.
  • Reviewers overlook breaking changes.

With a catalog:

  • Every error is documented.
  • Every error becomes testable.
  • Every API consumer sees the same contract.

The catalog becomes the foundation for automation.


One Test per Error Code, Generated from the Catalog

Once the catalog exists, generating negative tests becomes surprisingly straightforward.

Rather than manually writing dozens of repetitive tests, a generator simply iterates through every defined error.

Conceptually:

for (const errorCode of catalog) {
    generateNegativeTest(errorCode);
}

Each generated test validates four things:

  • The expected HTTP status
  • The error code
  • The error message
  • The response schema

Consider EMAIL_ALREADY_EXISTS.

The generated scenario might:

  1. Create a user.
  2. Attempt to create the same user again.
  3. Verify the response:
{
  "code": "EMAIL_ALREADY_EXISTS",
  "message": "Email already exists"
}

The implementation differs depending on the framework, but the testing philosophy remains the same:

Every documented error deserves exactly one corresponding test.

As new error codes are introduced, new tests appear automatically.

No engineer has to remember to write them.


Why This Scales Better

Imagine your API exposes:

  • 150 endpoints
  • 90 business error codes

Maintaining those manually quickly becomes tedious.

Generation solves two maintenance problems simultaneously:

  • Missing tests
  • Duplicate effort

Instead of asking developers to remember every negative case, the catalog guarantees baseline coverage.

Engineers can then focus on more complex business workflows rather than repetitive validation tests.


The Shape Assertion That Prevents Silent Error Drift

One lesson we learned very early was this:

Checking only the HTTP status is almost useless.

Imagine an endpoint originally returns:

{
  "code": "USER_NOT_FOUND",
  "message": "User not found",
  "requestId": "abc123"
}

Months later, someone refactors the global exception handler.

The response becomes:

{
  "error": "User not found"
}

The HTTP status is still:

404

Many tests still pass.

But every client expecting the original response contract is now broken.

This is known as silent error drift.

Nothing appears wrong until consumers start failing.


The Solution: Shape Assertions

Every negative test also validates the response structure.

Example:

expect(response.body).toEqual({
    code: expect.any(String),
    message: expect.any(String),
    requestId: expect.any(String)
});

Notice that we're not only validating values.

We're validating the schema itself.

That single assertion protects every API consumer from accidental response changes.


Why This Matters

Consumers often depend on:

  • Error codes
  • Localization keys
  • Correlation IDs
  • Documentation URLs

Removing any of these fields can become a breaking API change even though the HTTP status remains correct.

Schema validation catches those problems immediately.


Keeping the Catalog in Sync with the Code (Code Generation)

The obvious concern is maintenance.

If engineers must manually update both:

  • Source code
  • Error catalog

the catalog eventually becomes outdated.

The solution is code generation.

Most applications already define errors centrally.

For example:

export enum ErrorCode {
    USER_NOT_FOUND,
    INVALID_TOKEN,
    PAYMENT_DECLINED,
    EMAIL_ALREADY_EXISTS
}

A simple generation step can produce:

  • API documentation
  • OpenAPI components
  • Markdown reference tables
  • Test inputs
  • SDK constants

All from the same source.

Now there's only one place where error definitions live.

Everything else is generated automatically.


Benefits of Codegen

This approach creates several advantages:

Documentation Never Falls Behind

As soon as a new error appears in code, documentation updates automatically.

Generated Tests Stay Current

No manual synchronization required.

API Consumers Stay Aligned

Client SDKs can reference the same constants used by the server.

Code Reviews Become Easier

Adding a new business error becomes highly visible because it affects generated documentation and tests.


The Two Error Codes We Deliberately Don't Test (And Why)

Although our negative suite covers nearly every business error, there are two categories we intentionally exclude.

1. Generic Internal Server Errors

Example:

500 Internal Server Error

These represent unexpected failures.

They're not part of normal business behavior.

Rather than intentionally triggering every possible internal exception, we verify:

  • Sensitive details aren't exposed
  • Generic messages are returned
  • Correlation IDs exist
  • Logging occurs correctly

Testing every possible server failure adds little value.

Testing the response contract provides much greater return.


2. Infrastructure Failures

Examples include:

  • Database unavailable
  • Network partition
  • DNS outage
  • Message broker failure
  • Cloud storage unavailable

These failures belong to resilience testing rather than standard API automation.

They are better validated using:

  • Chaos engineering
  • Fault injection
  • Infrastructure testing
  • Disaster recovery exercises

Mixing infrastructure scenarios into routine API negative tests usually creates unstable pipelines.

Keeping them separate results in cleaner and more reliable automation.


Additional Benefits We Didn't Expect

Once the catalog became part of our development process, several unexpected improvements appeared.

More Consistent APIs

Every endpoint used the same response format.

Better Frontend Development

Frontend teams no longer guessed which errors could occur.

Simpler Documentation

Error references stayed synchronized automatically.

Cleaner Pull Requests

Adding a new error became an explicit design decision rather than an implementation detail.

Better QA Coverage

Negative scenarios became just as visible as successful ones.


Final Thoughts

Most engineering teams invest heavily in testing successful requests while treating failures as secondary concerns.

Stripe demonstrates a different philosophy.

Errors are documented, standardized, and treated as an integral part of the public API contract.

Building an error-code catalog allowed us to adopt that same mindset.

Instead of manually maintaining dozens of repetitive error response testing scenarios, we generated them from a single source of truth.

Combined with response schema validation and code generation, the approach dramatically reduced maintenance while increasing confidence that every documented failure behaved exactly as expected.

If your API already has a growing collection of business errors, consider creating a centralized catalog before the list becomes unmanageable.

The investment is relatively small, but the payoff in documentation quality, test coverage, and long-term maintainability is substantial.

If you'd like to explore how automated API testing can support this approach, you can spin up a free trial to try this catalog pattern and see how generated negative tests, schema validation, and API contracts work together in practice.