惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

Microsoft Azure Blog
Microsoft Azure Blog
博客园_首页
Forbes - Security
Forbes - Security
WordPress大学
WordPress大学
P
Proofpoint News Feed
T
Threat Research - Cisco Blogs
L
LINUX DO - 热门话题
L
Lohrmann on Cybersecurity
Spread Privacy
Spread Privacy
D
Darknet – Hacking Tools, Hacker News & Cyber Security
大猫的无限游戏
大猫的无限游戏
博客园 - 三生石上(FineUI控件)
P
Privacy International News Feed
A
About on SuperTechFans
T
Tailwind CSS Blog
I
InfoQ
S
Securelist
云风的 BLOG
云风的 BLOG
罗磊的独立博客
Recent Announcements
Recent Announcements
T
The Exploit Database - CXSecurity.com
B
Blog RSS Feed
V
Visual Studio Blog
Know Your Adversary
Know Your Adversary
The GitHub Blog
The GitHub Blog
Jina AI
Jina AI
腾讯CDC
Cyberwarzone
Cyberwarzone
有赞技术团队
有赞技术团队
AWS News Blog
AWS News Blog
博客园 - 【当耐特】
cs.AI updates on arXiv.org
cs.AI updates on arXiv.org
F
Full Disclosure
S
Secure Thoughts
博客园 - 司徒正美
J
Java Code Geeks
Y
Y Combinator Blog
Google Online Security Blog
Google Online Security Blog
GbyAI
GbyAI
N
News and Events Feed by Topic
Help Net Security
Help Net Security
freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More
Project Zero
Project Zero
T
Tenable Blog
cs.CV updates on arXiv.org
cs.CV updates on arXiv.org
T
Tor Project blog
MyScale Blog
MyScale Blog
Scott Helme
Scott Helme
小众软件
小众软件
K
Kaspersky official blog

DEV Community

Authentication Security Deep Dive: From Brute Force to Salted Hashing (With Java Examples) Why AI Systems Don’t Fail — They Drift Spilling beans for how i learn for exam😁"Reinforcement Learning Cheat Sheet" I Replaced Chrome with Safari for AI Browser Automation. Here's What Broke (and What Finally Worked) How Python Borrows Other People's Work The $40 Architecture: Processing 1 Billion API Requests with 99.99% Uptime Vibe Coding: A Workflow Guide (From Zero to SaaS) Most webhook security guides protect the wrong side. The scary part is delivery. Headless CMS for TanStack Start: Build a Blog with Cosmic EU Age Verification App "Hacked in 2 Minutes" — What Actually Happened Comfy Cloud’s delete function does not actually remove files Running AI Models on GPU Cloud Servers: A Beginner Guide Event-driven media intelligence with AWS Step Functions and Bedrock I scored 500 AI prompts across 8 quality dimensions — here's what broke How to Call Google Gemini API from Next.js (Free Tier, No Backend Needed) The Portal Protocol: Reclaiming Human Connection in the Age of AI How to Fix Your Team's Scattered Knowledge Problem With a Self-Hosted Forum Intro to tc Cloud Functors: A Graph-First Mental Model for the Modern Cloud Designing Multi-Tenant Backends With Both Ownership and Team Access I Built a Neumorphic CSS Library with 77+ Components — Here's What I Learned PostgreSQL Performance Optimization: Why Connection Pooling Is Critical at Scale Cómo construí un SaaS multi-rubro para gestionar expensas en Argentina con FastAPI + Vue 3 🚀 I Built an Ethical Hacking Scanner Tool – Open Source Project I Replaced /usage and /context in Claude Code With a Single Statusline A Pythonic Way to Handle Emails (IMAP/SMTP) with Auto-Discovery and AI-Ready Design I Collected 8.9 Million Polymarket Price Points — Here's What I Found About How Markets Really Move EcoTrack AI — Carbon Footprint Tracker & Dashboard Everyone's Using AI. No One Agrees How. 5 self-hosted ebook managers worth trying in 2026 Building Your First AI Agent with LangChain: From Chatbot to Autonomous Assistant Common SOC 2 Failures (Real World) Stop Vibe-Checking Your AI App: A Practical Guide to Evals How to Use SonarQube and SonarScanner Locally to Level Up Your Code Quality Your Next To-Do App Is Dead — I Replaced Mine with an OpenClaw AI Sign a Nostr event in 60 lines of Python using coincurve — no nostr-sdk, no nbxplorer, no rust toolchain ITGC Audit Explained Like You’re in Big 4 Patch Tuesday abril 2026: Microsoft parcha 163 vulnerabilidades y un zero-day en SharePoint Stop scraping everything: a better way to track competitor price changes Listing on MCPize + the Official MCP Registry while routing payments OUTSIDE the marketplace — how I kept 100% of my x402 revenue Building an AI-Powered Risk Intelligence System Using Serverless Architecture Why We Ripped Function Overloading Out of Our AI Toolchain Testing AI-Generated Code: How to Actually Know If It Works SaaS Churn Is Killing Your Business. Here Is What to Do About It (Without a Support Team) The Speed of AI Is No Longer Linear - And Self-Improving Models Are Why How to Implement RBAC for MCP Tools: A Practical Guide for Engineering Teams From Standard Quote to Persuasive Proposal: AI Automation for Arborists I built a CLI that scaffolds complete multi-tenant SaaS apps Axios CVE-2025–62718: The Silent SSRF Bug That Could Be Hiding in Your Node.js App Right Now The dashboard that ended our friendship Data Pipelines Explained Simply (and How to Build Them with Python) The Hidden Cost of AI Systems Nobody Talks About. undefined vs undeclared, and how typeof behaves Switching from file-based jobs to NATS/Kafka in Rust without changing code io_uring Adventures: Rust Servers That Love Syscalls Why Agentic AI is Killing the Traditional Database The POUR principles of web accessibility for developers and designers Quantum Neural Network 3D — A Deep Dive into Interactive WebGL Visualization How To Install Caveman In Codex On macOS And Windows Automation Pipeline Reliability: Why Your Workflow Breaks When Nobody Is Watching I Built an 'Open World' AI Coding Agent — It Works From ANY Folder From Freelancing to Product: A Tech Service Company's SaaS Transformation China's AI Giants: Adding Tencent Hunyuan & ByteDance Doubao to AI University (74 Providers) On the Vibe Coders and Their Lies clerk: Auto-Summarize Your Claude Code Sessions AI Weekly — 2026/04/10–04/17 | The Model Lockdown Is Here, but the Toolchain Is the Real Battleground AI 週報 — 2026/04/10–2026/04/17 模型封鎖潮來了,但工具鏈才是真戰場 Maybe this is how Open-Source apps are born... 🚀 Fine-Tune LLMs with LoRA and QLoRA: 2026 Guide tRPC v11 + Next.js App Router: End-to-End Type Safety Without the Boilerplate ShadCN UI in 2026: Why I Stopped Installing Component Libraries and Started Owning My Components SaaS Billing in React Server Components: Stripe + Supabase Without a Single `useEffect` Join our DEV Weekend Challenge — $1,000 in Prizes Across TEN winners! Submissions Due April 20 at 6:59 AM UTC. Implementing FSRS Spaced Repetition in Flutter + Supabase — Adding Memory Science to an AI Learning App "I Texted My Localhost From the Train — Claude Code Fixed the Bug Before I Got Home" I Built a Sales Prep AI and It Went Deeper Than Expected Design to Code #2: One JSON, Eleven Outputs Solving the 100M-Row Problem: A Summary Table Pattern for High-Volume Push Notification Logs Flutter Web With Wasm: What Actually Changes For Developers I Built 50 Royalty-Free Soundtracks for My Side Project in a Weekend Using AI Music Generation The Vibe Coding Security Checklist: 7 Things to Check Before You Ship Stop Letting Googlebot Guess Fix Your React App's SEO Right Desconstruindo o Streaming do LinkedIn: Como Criar um Engine de Extração de Vídeo de Alta Performance com HLS e FFmpeg (EDA Part-1) EDA (Exploratory Data Analysis) Explained With Real Life — Why Looking at Your Data Is the Most Important Step in Machine Learning Brand Relationship Management at Scale: Our 4-Touch Outreach System for 200+ Brands Why String.fromEnvironment() Might Return an Empty String in Dart JGuardrails 1.0.0 — Hardening Java LLM Apps Against Jailbreaks, Toxicity, and Prompt Injection Plan and Schedule a Full Week of Threads Content From One Claude Conversation Coding Cat Oran Ep3, Five Tables Changed Everything Updated: BFF Pattern I'm done watching freelancers get buried by 200 proposals. So I'm building the alternative. This is my first post BFS Algorithm in Java Step by Step Tutorial with Examples Tracking LLM Pricing Monthly: An Open Dataset for 22 AI Models How We Measure Content ROI on a Comparison Site: Revenue Attribution Without Perfect Data Introducing Nova AI Ops: The AI-Native Operating System for SRE Teams I built a free desktop video downloader for Windows — Grabbit How Talkie OCR Helps Vision-Impaired & Dyslexic Users Read the World Around Them VRCFaceTracking安装和iPhone面捕配置教程,有bug Even CrowdStrike Can't See Your Agents The Automation Gold Rush: What n8n Workflows and Claude Are Opening Up for Developers Right Now
Flaky Tests You Can't Fix With Better Selectors
Dmitry · 2026-06-03 · via DEV Community

You've fixed your locators. You've switched to web-first assertions. Your tests still fail intermittently — but now the failures look different. Duplicate records in the database. Tests that pass alone but fail in parallel. Mocks that say everything is fine while production is broken.

This is the next layer of flakiness. It lives in your API calls, your test doubles, and your database. Better selectors won't help here.

Code examples are simplified for clarity — focus on the idea, not the boilerplate.


TL;DR

  1. Use idempotency keys on POST requests — one network glitch shouldn't create two orders
  2. page.route mocks the browser, not your server — know the difference
  3. Use WireMock for server-to-server integrations you can't control
  4. Contract tests catch API drift before it reaches production
  5. Never rely on afterEach for database cleanup — use TTL or a cleanup queue instead

The Problem: Flakiness That Looks Like Application Bugs

When a selector fails, the error is obvious. When a test creates a duplicate order because a network request was retried, the error looks like a business logic bug. You spend an hour investigating something that has nothing to do with your application code.

Three categories cause this:

API flakiness — a request succeeds on the server but the response never arrives. Playwright retries. Now you have two orders.

Lying mocks — your mocks say the API returns { order_id: "123" }. The backend deployed last week and now returns { orderId: "123" }. Tests are green. Production is broken.

Data pollution — tests create users, orders, and transactions but don't clean up reliably. After a week, the test database is a graveyard that slows down queries and causes unique constraint violations.


Rule #1: Idempotency Keys — One Request, One Result

Networks are unreliable. A POST request can reach the server, create a record, and then the response gets lost in transit. Playwright sees a timeout and retries. The server sees a new request and creates another record.

The fix is an idempotency key — a unique header that tells the server "if you've seen this request before, return the same result instead of processing it again."

// api/infrastructure/idempotency.ts
import { createHash } from 'crypto';

export function generateIdempotencyKey(method: string, url: string, data: unknown): string {
  const payload = `${method}:${url}:${JSON.stringify(data)}`;
  return createHash('sha256').update(payload).digest('hex').slice(0, 16);
}

Enter fullscreen mode Exit fullscreen mode

// api/clients/BaseApiClient.ts
export abstract class BaseApiClient {
  protected async post(url: string, data?: unknown) {
    const key = generateIdempotencyKey('POST', url, data);
    return await this.request.post(url, {
      data,
      headers: { 'X-Idempotency-Key': key },
    });
  }
}

Enter fullscreen mode Exit fullscreen mode

The key is generated from the request method, URL, and body — so two identical requests get the same key, but two different requests (create user, then create order) get different keys. One network glitch no longer creates two records.

Important nuance: if your test legitimately creates two identical orders (same body, same URL), they'll get the same key — and the server will return the first result for both. This is intentional behaviour for retries, but it means this approach assumes each unique operation has unique data. If you need two genuinely identical records, add a unique field (like requestId) to the body.

Note: This only works if your backend handles the X-Idempotency-Key header. Check with your backend team — many order APIs support this out of the box. If your payment provider doesn't support it — that's their problem to solve, not yours. Look for their sandbox or test mode, or use WireMock to mock them entirely. If it's your own backend that's missing support — that's a tech-debt conversation with your backend team, not something to work around in tests.


Rule #2: Know What Your Mocks Actually Cover

page.route is Playwright's built-in way to intercept requests. It's great for testing UI behavior in isolation — how does the page look when the API returns an error?

// Good use of page.route — testing UI error state
await page.route('**/api/orders', (route) => {
  route.fulfill({
    status: 500,
    body: JSON.stringify({ error: 'Internal Server Error' }),
  });
});

await page.goto('/orders');
await expect(page.getByText('Something went wrong')).toBeVisible();

Enter fullscreen mode Exit fullscreen mode

The catch: page.route only intercepts requests made from inside the browser. If your test makes API calls directly through Playwright's request fixture — server-side, without a browser — page.route won't see them.

// This request bypasses page.route entirely
const response = await request.post('/api/orders', { data: orderData });

// But this goes through the browser context and IS intercepted by page.route
const response = await page.evaluate(() =>
  fetch('/api/orders', { method: 'POST' }).then((r) => r.json()),
);

Enter fullscreen mode Exit fullscreen mode

Why route.fulfill() instead of route.abort()? abort() causes the request to fail with a network error. Some apps handle this gracefully, but others enter an infinite retry loop waiting for a response that never comes. fulfill() returns a proper HTTP response (even a fake one) so the app moves on cleanly.

For direct API calls in tests, you need mocks at a different level — either a wrapper around request, or an infrastructure mock like WireMock.


Rule #3: WireMock for Integrations You Don't Control

Your backend calls Stripe for payments. It calls Twilio for SMS. It calls a shipping provider to get rates. In tests, you don't want any of that to actually happen.

page.route can't help here — these are server-to-server calls that never touch the browser. The solution is WireMock: a mock server that runs alongside your test environment and intercepts HTTP calls at the network level.

# docker-compose.yml
services:
  wiremock:
    image: wiremock/wiremock:3.3.1
    ports:
      - '8080:8080'
    volumes:
      - ./wiremock/mappings:/home/wiremock/mappings

Enter fullscreen mode Exit fullscreen mode

// wiremock/mappings/stripe-payment.json
{
  "request": {
    "method": "POST",
    "url": "/v1/payment_intents"
  },
  "response": {
    "status": 200,
    "jsonBody": {
      "id": "pi_test_123",
      "status": "succeeded"
    }
  }
}

Enter fullscreen mode Exit fullscreen mode

Now your backend hits localhost:8080 in tests instead of the real Stripe API. Tests are fast, isolated, and don't depend on external uptime.

Point your backend's base URLs to WireMock via environment variables in your test environment:

STRIPE_BASE_URL=http://localhost:8080
TWILIO_BASE_URL=http://localhost:8080

Enter fullscreen mode Exit fullscreen mode

One thing to be aware of: this requires your backend to use configurable base URLs for external services. In most well-structured backends this is already the case. If it's not — that's a conversation with the backend team, not a reason to skip WireMock.


Rule #4: Contract Tests — Stop Trusting Your Mocks

Here's the problem with all mocks: they can lie. Your WireMock returns { payment_id: "pay_123" }. The backend team renames the field to paymentId. Your tests stay green. Production breaks.

This is called a Lying Mock — a test double that no longer matches reality.

Contract testing fixes this. Instead of just mocking the response, you write a contract: "I expect this request to return this response." The backend then verifies that contract against its actual code.

// tests/contracts/payment.pact.spec.ts
import { PactV3, MatchersV3 } from '@pact-foundation/pact';

const provider = new PactV3({
  consumer: 'frontend-tests',
  provider: 'payment-service',
  dir: './pacts',
});

describe('Payment API contract', () => {
  it('returns payment confirmation', async () => {
    await provider
      .given('a valid payment intent exists')
      .uponReceiving('POST /v1/payment_intents')
      .withRequest({ method: 'POST', path: '/v1/payment_intents' })
      .willRespondWith({
        status: 200,
        body: {
          id: MatchersV3.string('pi_test_123'),
          status: MatchersV3.string('succeeded'),
        },
      })
      .executeTest(async (mockServer) => {
        const result = await createPayment(mockServer.url);
        expect(result.status).toBe('succeeded');
      });
  });
});

Enter fullscreen mode Exit fullscreen mode

After this test runs, it generates a JSON contract file in ./pacts. The backend team runs that contract against their actual API. If they rename id to paymentId — the contract verification fails in their pipeline, before the change is merged.

Where to start: You don't need to contract-test everything. Start with the APIs that change most often or have caused the most incidents. One contract on your payment flow is worth more than ten contracts on stable read-only endpoints.

One important caveat: contract testing requires the backend team to actually run the verification in their pipeline. This is an organizational commitment, not just a technical one. For small teams, storing contract JSON files in the backend repo and running verification manually is a simpler starting point than a full Pact Broker setup. Also note that contracts verify response structure — they don't catch business logic bugs or side effects.


Rule #5: Stop Relying on afterEach for Cleanup

The classic approach to test data cleanup:

// Unreliable
afterEach(async () => {
  await api.deleteUser(userId);
});

Enter fullscreen mode Exit fullscreen mode

This fails silently when a test crashes before setting userId. It doesn't run when the test runner itself crashes. After a CI failure mid-run, your database has orphaned records that affect the next run.

Three approaches that actually work:

TTL — let the database clean up automatically

Add an expires_at field to your test entities and set it when creating them:

// When creating test data
await api.createUser({
  email: `test_${Date.now()}@example.com`,
  expires_at: new Date(Date.now() + 24 * 60 * 60 * 1000).toISOString(),
});

Enter fullscreen mode Exit fullscreen mode

In PostgreSQL, a scheduled job handles cleanup:

-- Runs every hour via pg_cron
SELECT cron.schedule('cleanup-test-data', '0 * * * *', $$
  DELETE FROM users WHERE expires_at < NOW() AND is_test = true;
  DELETE FROM orders WHERE expires_at < NOW() AND is_test = true;
$$);

Enter fullscreen mode Exit fullscreen mode

In MongoDB and Redis, TTL indexes handle this natively — no cron job needed.

Cleanup queue — collect IDs, delete in bulk

Track everything your tests create, then clean it all up in a global teardown:

// In your base API client
protected async post(url: string, data?: unknown) {
  const response = await this.request.post(url, { data });
  const body = await response.json();

  if (body.id) {
    cleanupQueue.push({ url, id: body.id });
  }
  return response;
}

// global-teardown.ts
export default async function globalTeardown() {
  for (const item of cleanupQueue) {
    await api.delete(`${item.url}/${item.id}`);
  }
}

Enter fullscreen mode Exit fullscreen mode

Even if individual tests fail, the global teardown runs and cleans up the queue.

Which approach to use: TTL is the more reliable default — it works even if the test runner is killed with SIGKILL, because cleanup happens at the database level independently of your test process. Use TTL as your first line of defence. The cleanup queue is a good complement when you need guaranteed cleanup order or when your database doesn't support scheduled jobs — but it won't run if the process is hard-killed.


Putting It Together: The Data Reliability Cheat Sheet

Symptom Root cause Fix
Duplicate records after CI failure No idempotency on POST requests Add X-Idempotency-Key header
Tests green, production broken Mocks don't match real API Add contract tests for critical endpoints
page.route mock not working Request bypasses browser Use WireMock or request wrapper
Database full of test garbage afterEach cleanup unreliable TTL field + pg_cron or cleanup queue
External API causing flakiness Real network calls in tests WireMock for server-to-server calls

What's Next?

You now have three layers covered: test infrastructure, object lifecycle, and data reliability. The next layer is observability — how do you measure test health, identify patterns in flakiness, and prove to your manager that stability work has business value?

Want to go deeper on any of these topics? Check out the advanced version: Why Your Test Suite Lies to You at Scale


All patterns in this article are implemented in the Playwright BDR Template on GitHub.