惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

Microsoft Azure Blog
Microsoft Azure Blog
宝玉的分享
宝玉的分享
博客园 - 【当耐特】
有赞技术团队
有赞技术团队
G
Google Developers Blog
Microsoft Security Blog
Microsoft Security Blog
Apple Machine Learning Research
Apple Machine Learning Research
The Cloudflare Blog
Blog — PlanetScale
Blog — PlanetScale
博客园_首页
L
LangChain Blog
Stack Overflow Blog
Stack Overflow Blog
Last Week in AI
Last Week in AI
Y
Y Combinator Blog
罗磊的独立博客
T
Tailwind CSS Blog
博客园 - 叶小钗
T
The Blog of Author Tim Ferriss
Engineering at Meta
Engineering at Meta
博客园 - 聂微东
博客园 - Franky
B
Blog
freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More
F
Fortinet All Blogs

DEV Community

Authentication Security Deep Dive: From Brute Force to Salted Hashing (With Java Examples) Why AI Systems Don’t Fail — They Drift Spilling beans for how i learn for exam😁"Reinforcement Learning Cheat Sheet" I Replaced Chrome with Safari for AI Browser Automation. Here's What Broke (and What Finally Worked) How Python Borrows Other People's Work The $40 Architecture: Processing 1 Billion API Requests with 99.99% Uptime Vibe Coding: A Workflow Guide (From Zero to SaaS) Most webhook security guides protect the wrong side. The scary part is delivery. Headless CMS for TanStack Start: Build a Blog with Cosmic EU Age Verification App "Hacked in 2 Minutes" — What Actually Happened Comfy Cloud’s delete function does not actually remove files Running AI Models on GPU Cloud Servers: A Beginner Guide Event-driven media intelligence with AWS Step Functions and Bedrock I scored 500 AI prompts across 8 quality dimensions — here's what broke How to Call Google Gemini API from Next.js (Free Tier, No Backend Needed) The Portal Protocol: Reclaiming Human Connection in the Age of AI How to Fix Your Team's Scattered Knowledge Problem With a Self-Hosted Forum Intro to tc Cloud Functors: A Graph-First Mental Model for the Modern Cloud Designing Multi-Tenant Backends With Both Ownership and Team Access I Built a Neumorphic CSS Library with 77+ Components — Here's What I Learned PostgreSQL Performance Optimization: Why Connection Pooling Is Critical at Scale Cómo construí un SaaS multi-rubro para gestionar expensas en Argentina con FastAPI + Vue 3 🚀 I Built an Ethical Hacking Scanner Tool – Open Source Project I Replaced /usage and /context in Claude Code With a Single Statusline A Pythonic Way to Handle Emails (IMAP/SMTP) with Auto-Discovery and AI-Ready Design I Collected 8.9 Million Polymarket Price Points — Here's What I Found About How Markets Really Move EcoTrack AI — Carbon Footprint Tracker & Dashboard Everyone's Using AI. No One Agrees How. 5 self-hosted ebook managers worth trying in 2026 Building Your First AI Agent with LangChain: From Chatbot to Autonomous Assistant
Building the future for benefits tracking
Dylen Belanger · 2026-06-23 · via DEV Community

Most people have no idea what's left in their benefits plan. Their employer pays for dental, vision, massage, and more, and at the end of the year whatever you didn't touch just disappears. I never knew what I had left, and when I did I'd forget to use it before it reset, so I built something to fix that.

BenefitTrack emails you every month with exactly how much you have left to spend across all your benefits, so you stop losing money to coverage you already paid for.

The core problem

BenefitTrack runs a monthly browser automation against insurance carrier portals that change without warning and have no public API. What's difficult is that we don't control when those portals change. A portal can restructure a page, change how it signals a successful login, or drop a new modal on the home screen, and we find out only when something breaks at 8 AM on the first of the month.

Why Stagehand over selectors

The obvious approach is a traditional scraper: find the login form, click the button by its CSS class, wait for a redirect, and parse the balances. That works right up until the portal changes its markup, at which point the scrape either crashes or, worse, succeeds against a page that no longer means what you think it does.

To mitigate this we chose to build with Stagehand, a browser automation framework from Browserbase that lets you describe what you want in plain language instead of hardcoding selectors.

// selector-based: breaks if the class name or DOM structure changes
await page.click('.btn-primary.submit-login');

// Stagehand: works as long as there's a sign in button on the page
await stagehand.act('click the sign in button');

The catch is that every LLM call costs time and money. A full scrape takes a few minutes and is obviously more expensive than a selector-based one. On a portal that can change anytime without notice, it pays for itself the first time it survives a portal change you didn't see coming.

The stack

We're building BenefitTrack on Next.js, NestJS, Supabase, and Vercel. It lets us move fast and keep costs near zero while we're still getting off the ground.

The one genuinely interesting part is how we handle long-running scrapes. A full portal login with MFA can run several minutes, well past a normal serverless timeout. Vercel's waitUntil gets around it by letting the request return right away while the work keeps going in the background.

@Post()
@HttpCode(202)
run(@Body() body: ScrapeRequestDto) {
  waitUntil(
    this.scrapeService.run(body).catch(() => {
      // errors are already logged + persisted to scrape_tasks by the service
    }),
  );

  return { accepted: true };
}

The orchestrator fires a POST to /api/scrape for each employee, gets a 202 back, and moves on. Each scrape runs on its own and writes its own result to the database. However, if a function dies mid-scrape there's no recovery, and there's a hard 800-second ceiling per invocation. Both are fine for now, but at real scale a durable job queue would be ideal.

Handling credentials

Scraping a carrier portal means holding a user's login for that portal, which is about as sensitive as data gets. Credentials are encrypted at rest and only decrypted in memory for the length of a single scrape, and they never get written to logs or error traces.

Concurrency and reliability

Once a few dozen scrapes are running at once, there's a lot more that can go wrong than in a single run.

First is Browserbase itself: 25 concurrent sessions and 25 new sessions per minute on our current plan. These limits are enforced with a Bottleneck token bucket.

const limiter = new Bottleneck({
  maxConcurrent: 25,
  reservoir: 25,
  reservoirRefreshAmount: 25,
  reservoirRefreshInterval: 60_000, // refill 25 tokens every minute
});

await Promise.all(
  employeeCarriers.map((ec) =>
    limiter.schedule(async () => {
      await fetch(`${baseUrl}/api/scrape`, { method: 'POST', body: JSON.stringify(payload) });
    }),
  ),
);

Each scheduled task spends a token, and the bucket tops back up to 25 every minute. Cross either line and Browserbase answers with a 429.

Then there's not running the same scrape twice. Re-triggering the job is essential when something fails halfway, so the system has to know who already succeeded and leave them alone. Before dispatching, it pulls any scrape_task marked status: 'success' in the current calendar month and only queues the rest.

The email send has its own check, and that separation is deliberate. Before sending, it looks for a notification already sent that month for that employee. The two have separate checks because they fail independently: a scrape can succeed while the email fails, or an email can go out before a scrape ever finishes. A single combined check would let a failure on one side quietly block the other.

Sessions hang too. Browserbase times out, a portal stalls loading a page, and the task just sits in running. An hourly cron job retries anything stuck in running or queued for more than 30 minutes, marks it failed, and sends it back out. A loop guard cuts that off after two sweep-marked failures in a month, since otherwise a genuinely broken scraper would keep retrying itself until the month ran out.

And scrapes can lie. The session finishes, the LLM hands back a result, and the data is still wrong: empty benefit types, negative amounts, amountUsed > totalAmount. So before anything is saved, the service validates every field and throws on the first bad one. That matters because of idempotency: a scrape_task marked success blocks the retry path. If bad data gets written as a success, the employee gets a wrong email and nothing ever retries it. Failing the task and running again is better than sending something wrong every time.

The bug that taught me the most

The one that stuck with me was an auth-detection loop. A user's session was perfectly valid, but my check kept deciding they were logged out and kicking off a full re-login. One symptom, and underneath it three separate wrong assumptions, all of which I dug out in a single day.

First, I was reading the URL to decide whether login had worked, and the post-login URL wasn't consistent. So I switched to looking for the member's name and a "Sign Out" button as proof they were in, except both were tucked inside a menu on the logged-in home page and neither was actually visible. And under all of that, the check was firing before the page had finished loading at all, since the portal sat behind a bot-protection layer that could take a minute or more to paint.

Three causes, one symptom. The real takeaway was that the thing I'd chosen to mean "logged in" was never a reliable signal to begin with.

Where it stands today

Every carrier integration is built by hand, and right now only one portal is fully working. Adding a carrier means writing a new scraper from scratch, and a large enough layout change can still break extraction even with Stagehand soaking up the smaller stuff.

There's also no automated retry for real code bugs. The stuck-task sweep handles transient failures like a hung session, but it can't save a scraper that navigates to the wrong page or pulls the wrong field. That kind of thing takes a code change and a manual re-run.

What building it taught me

Almost none of the hard parts were things I prepared for. The trouble came from things I couldn't have guessed: a portal that took ninety seconds to load, a button hidden behind a menu, redirect chains that shifted from one session to the next. I couldn't have planned for these, but I was flexible enough to change my approach the moment they showed up.

The other lesson: when you're scraping a portal you don't own, don't trust URLs or specific elements to sit where you left them. Anchor on the content a real person would use to know they've landed in the right place.

Idempotency turned out to matter far more than it sounds like it should. If we did not include this wrong, someone could get multiple emails per month, or none at all. For a product whose entire job is one correct email a month, that's not a small thing.


BenefitTrack is available today for a limited number of carriers, with more being added over the coming weeks. If staying on top of your benefits sounds useful, take a look at benefittrack.ca.