惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

量子位
小众软件
小众软件
S
SegmentFault 最新的问题
人人都是产品经理
人人都是产品经理
博客园 - 【当耐特】
博客园 - 三生石上(FineUI控件)
C
Check Point Blog
S
Schneier on Security
Microsoft Azure Blog
Microsoft Azure Blog
N
Netflix TechBlog - Medium
Engineering at Meta
Engineering at Meta
GbyAI
GbyAI
罗磊的独立博客
有赞技术团队
有赞技术团队
V
V2EX
Y
Y Combinator Blog
博客园 - 叶小钗
让小产品的独立变现更简单 - ezindie.com
让小产品的独立变现更简单 - ezindie.com
freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More
Cyber Security Advisories - MS-ISAC
Cyber Security Advisories - MS-ISAC
F
Fortinet All Blogs
W
WeLiveSecurity
OSCHINA 社区最新新闻
OSCHINA 社区最新新闻
Stack Overflow Blog
Stack Overflow Blog
The Cloudflare Blog
S
Security @ Cisco Blogs
TaoSecurity Blog
TaoSecurity Blog
MyScale Blog
MyScale Blog
Hugging Face - Blog
Hugging Face - Blog
cs.AI updates on arXiv.org
cs.AI updates on arXiv.org
www.infosecurity-magazine.com
www.infosecurity-magazine.com
PCI Perspectives
PCI Perspectives
H
Heimdal Security Blog
Schneier on Security
Schneier on Security
Security Latest
Security Latest
AWS News Blog
AWS News Blog
月光博客
月光博客
Security Archives - TechRepublic
Security Archives - TechRepublic
Recent Announcements
Recent Announcements
Google DeepMind News
Google DeepMind News
博客园 - Franky
Cisco Talos Blog
Cisco Talos Blog
T
Threat Research - Cisco Blogs
M
MIT News - Artificial intelligence
T
Troy Hunt's Blog
N
News and Events Feed by Topic
Cloudbric
Cloudbric
Scott Helme
Scott Helme
云风的 BLOG
云风的 BLOG
Attack and Defense Labs
Attack and Defense Labs

DEV Community

Authentication Security Deep Dive: From Brute Force to Salted Hashing (With Java Examples) Why AI Systems Don’t Fail — They Drift Spilling beans for how i learn for exam😁"Reinforcement Learning Cheat Sheet" I Replaced Chrome with Safari for AI Browser Automation. Here's What Broke (and What Finally Worked) How Python Borrows Other People's Work The $40 Architecture: Processing 1 Billion API Requests with 99.99% Uptime Vibe Coding: A Workflow Guide (From Zero to SaaS) Most webhook security guides protect the wrong side. The scary part is delivery. Headless CMS for TanStack Start: Build a Blog with Cosmic EU Age Verification App "Hacked in 2 Minutes" — What Actually Happened Comfy Cloud’s delete function does not actually remove files Running AI Models on GPU Cloud Servers: A Beginner Guide Event-driven media intelligence with AWS Step Functions and Bedrock I scored 500 AI prompts across 8 quality dimensions — here's what broke How to Call Google Gemini API from Next.js (Free Tier, No Backend Needed) The Portal Protocol: Reclaiming Human Connection in the Age of AI How to Fix Your Team's Scattered Knowledge Problem With a Self-Hosted Forum Intro to tc Cloud Functors: A Graph-First Mental Model for the Modern Cloud Designing Multi-Tenant Backends With Both Ownership and Team Access I Built a Neumorphic CSS Library with 77+ Components — Here's What I Learned PostgreSQL Performance Optimization: Why Connection Pooling Is Critical at Scale Cómo construí un SaaS multi-rubro para gestionar expensas en Argentina con FastAPI + Vue 3 🚀 I Built an Ethical Hacking Scanner Tool – Open Source Project I Replaced /usage and /context in Claude Code With a Single Statusline A Pythonic Way to Handle Emails (IMAP/SMTP) with Auto-Discovery and AI-Ready Design I Collected 8.9 Million Polymarket Price Points — Here's What I Found About How Markets Really Move EcoTrack AI — Carbon Footprint Tracker & Dashboard Everyone's Using AI. No One Agrees How. 5 self-hosted ebook managers worth trying in 2026 Building Your First AI Agent with LangChain: From Chatbot to Autonomous Assistant Common SOC 2 Failures (Real World) Stop Vibe-Checking Your AI App: A Practical Guide to Evals How to Use SonarQube and SonarScanner Locally to Level Up Your Code Quality Your Next To-Do App Is Dead — I Replaced Mine with an OpenClaw AI Sign a Nostr event in 60 lines of Python using coincurve — no nostr-sdk, no nbxplorer, no rust toolchain ITGC Audit Explained Like You’re in Big 4 Patch Tuesday abril 2026: Microsoft parcha 163 vulnerabilidades y un zero-day en SharePoint Stop scraping everything: a better way to track competitor price changes Listing on MCPize + the Official MCP Registry while routing payments OUTSIDE the marketplace — how I kept 100% of my x402 revenue Building an AI-Powered Risk Intelligence System Using Serverless Architecture Why We Ripped Function Overloading Out of Our AI Toolchain Testing AI-Generated Code: How to Actually Know If It Works SaaS Churn Is Killing Your Business. Here Is What to Do About It (Without a Support Team) The Speed of AI Is No Longer Linear - And Self-Improving Models Are Why How to Implement RBAC for MCP Tools: A Practical Guide for Engineering Teams From Standard Quote to Persuasive Proposal: AI Automation for Arborists I built a CLI that scaffolds complete multi-tenant SaaS apps Axios CVE-2025–62718: The Silent SSRF Bug That Could Be Hiding in Your Node.js App Right Now The dashboard that ended our friendship Data Pipelines Explained Simply (and How to Build Them with Python) The Hidden Cost of AI Systems Nobody Talks About. undefined vs undeclared, and how typeof behaves Switching from file-based jobs to NATS/Kafka in Rust without changing code io_uring Adventures: Rust Servers That Love Syscalls Why Agentic AI is Killing the Traditional Database The POUR principles of web accessibility for developers and designers Quantum Neural Network 3D — A Deep Dive into Interactive WebGL Visualization How To Install Caveman In Codex On macOS And Windows Automation Pipeline Reliability: Why Your Workflow Breaks When Nobody Is Watching I Built an 'Open World' AI Coding Agent — It Works From ANY Folder From Freelancing to Product: A Tech Service Company's SaaS Transformation China's AI Giants: Adding Tencent Hunyuan & ByteDance Doubao to AI University (74 Providers) On the Vibe Coders and Their Lies clerk: Auto-Summarize Your Claude Code Sessions AI Weekly — 2026/04/10–04/17 | The Model Lockdown Is Here, but the Toolchain Is the Real Battleground AI 週報 — 2026/04/10–2026/04/17 模型封鎖潮來了,但工具鏈才是真戰場 Maybe this is how Open-Source apps are born... 🚀 Fine-Tune LLMs with LoRA and QLoRA: 2026 Guide tRPC v11 + Next.js App Router: End-to-End Type Safety Without the Boilerplate ShadCN UI in 2026: Why I Stopped Installing Component Libraries and Started Owning My Components SaaS Billing in React Server Components: Stripe + Supabase Without a Single `useEffect` Join our DEV Weekend Challenge — $1,000 in Prizes Across TEN winners! Submissions Due April 20 at 6:59 AM UTC. Implementing FSRS Spaced Repetition in Flutter + Supabase — Adding Memory Science to an AI Learning App "I Texted My Localhost From the Train — Claude Code Fixed the Bug Before I Got Home" I Built a Sales Prep AI and It Went Deeper Than Expected Design to Code #2: One JSON, Eleven Outputs Solving the 100M-Row Problem: A Summary Table Pattern for High-Volume Push Notification Logs Flutter Web With Wasm: What Actually Changes For Developers I Built 50 Royalty-Free Soundtracks for My Side Project in a Weekend Using AI Music Generation The Vibe Coding Security Checklist: 7 Things to Check Before You Ship Stop Letting Googlebot Guess Fix Your React App's SEO Right Desconstruindo o Streaming do LinkedIn: Como Criar um Engine de Extração de Vídeo de Alta Performance com HLS e FFmpeg (EDA Part-1) EDA (Exploratory Data Analysis) Explained With Real Life — Why Looking at Your Data Is the Most Important Step in Machine Learning Brand Relationship Management at Scale: Our 4-Touch Outreach System for 200+ Brands Why String.fromEnvironment() Might Return an Empty String in Dart JGuardrails 1.0.0 — Hardening Java LLM Apps Against Jailbreaks, Toxicity, and Prompt Injection Plan and Schedule a Full Week of Threads Content From One Claude Conversation Coding Cat Oran Ep3, Five Tables Changed Everything Updated: BFF Pattern I'm done watching freelancers get buried by 200 proposals. So I'm building the alternative. This is my first post BFS Algorithm in Java Step by Step Tutorial with Examples Tracking LLM Pricing Monthly: An Open Dataset for 22 AI Models How We Measure Content ROI on a Comparison Site: Revenue Attribution Without Perfect Data Introducing Nova AI Ops: The AI-Native Operating System for SRE Teams I built a free desktop video downloader for Windows — Grabbit How Talkie OCR Helps Vision-Impaired & Dyslexic Users Read the World Around Them VRCFaceTracking安装和iPhone面捕配置教程,有bug Even CrowdStrike Can't See Your Agents The Automation Gold Rush: What n8n Workflows and Claude Are Opening Up for Developers Right Now
Real-Time Fraud Detection in Java with Kafka Streams and Vector Similarity
Ricardo Mell · 2026-04-29 · via DEV Community

This content is based on a talk that my colleague Tim Kelly and I presented at DevNexus 2026 in Atlanta, one of the largest Java conferences in the world.

Imagine you’re at a supermarket or shopping online. You tap your card or click “Buy Now”. In that exact moment, your bank has to make a decision: is this transaction legitimate, or is it fraud? And it has to do that in milliseconds, and get it right. If it blocks a real transaction, it frustrates the customer. If it approves a fraudulent one, it leads to financial loss. So the question is: how do banks actually do this? What’s happening behind the scenes? In this article, we’ll explore a real-world approach to solving this problem using Java, Apache Kafka, Kafka Streams, and vector similarity with MongoDB.

What We’re Building

At a high level, a payment system works like this: a transaction is created, it goes through a validation step, and then a final decision is made to approve or decline:

High Level Payment flow

Now, let’s zoom in on that validation step, because this is where fraud detection actually happens:

Validation Process

In our approach, this validation is not a single check but a pipeline composed of multiple stages working together:

  1. Guardrails: Rule-based checks that catch obvious and well-known fraud patterns immediately, like impossible travel or unusual transaction velocity.
  2. Vector Scoring: AI-powered similarity matching that compares the transaction against known patterns to detect more subtle or behavioral fraud.
  3. Decision: The final verdict is based on the combined output of both layers: route the transaction to the approved_transactions collection or to the suspicious_transactions collection for further review or definitive blocking.

The decision flow is pretty straightforward.

If a transaction is flagged at any point, either by the guardrails or by the vector scoring stage, we treat it as suspicious and store it in the suspicious_transactions collection. From there, it can be reviewed, enriched with more context, or even blocked depending on the scenario.

On the other hand, if it passes both layers without any issues, we consider it safe and store it in the approved_transactions collection.

The Challenge: Millisecond Decisions with Resilience Under Failure

In fraud detection systems, especially in financial applications, speed is not just important, it is critical. Every transaction must be analyzed in real time, often under heavy load, with thousands of events happening every second. At the same time, the system cannot afford to break if a single component fails.

This leads to two non-negotiable constraints:

  • Decisions must be made in milliseconds
  • The system must remain resilient even when individual components fail

Before we get into how each layer works, it is important to understand how these constraints shape the entire architecture.

A natural first instinct is to build this using a simple synchronous architecture.

Synchronous Flow

The payment service calls the fraud service, which then queries the database, and the flow continues to the notification service. This works fine at the beginning. It’s simple and easy to understand.

But the problem is that everything is tightly connected. One service depends on the next.

Now imagine the database takes a bit longer to respond.

That delay starts to affect the whole flow. The fraud service slows down, the payment service has to wait, and the notification service also gets delayed.

At scale, this becomes a real issue. One slow component can impact everything else.

So how do we break this dependency and make each service more independent?

This is where event-driven architecture comes in.

The Strategy: Decoupling the System with Event-Driven Architecture

Instead of having services call each other directly, we introduce an event log as the backbone of communication.

Apache Kafka allows the payment service to publish a transaction event to a topic, without knowing who will consume it. The fraud service, the notification service, and any other interested component can subscribe to that topic independently:

Kafka Architecture

This fundamentally changes how the system behaves.

  • Services are no longer tightly coupled
  • Each service can fail or restart without affecting the others
  • The system becomes more resilient and easier to scale

At this point, we have removed the direct dependency between services and solved the resilience problem.

But we still have another challenge.

Stateful Checks Without a Database Round-Trip

Many fraud detection techniques depend on understanding recent activity.

For example, the system may need to evaluate how a card has been used within a short time window before deciding whether a new transaction is suspicious.

In a traditional architecture, this usually requires querying a database for every transaction.

At high throughput, those round-trips add latency that quickly becomes a bottleneck. In a system where decisions must be made in milliseconds, this approach does not scale.

So the question becomes: How do we perform these kinds of checks without hitting the database every time?

Solving It with Kafka Streams

Kafka Streams is a lightweight Java library that runs inside your application, processing events as they flow through Kafka topics. More importantly, it allows us to maintain state locally.

Instead of querying a remote database for every transaction, we keep the necessary data inside the same process that handles the events, using an embedded store backed by RocksDB.

Kafka Streams Flow

In our example, we want to analyze transaction activity per card within a short time window. To do that, we group events by card number. Each card number becomes a key, and for each key, the system maintains a running view of recent activity within a defined time window. This means we eliminate network calls and database round-trips from the critical path, relying only on local lookups. For our use case, this makes a big difference.

We avoid unnecessary latency and keep everything fast enough for real-time decisions, without depending on external systems. You can see where this happens directly in the code:

return stream
    .filter((key, tx) -> tx != null)
    .selectKey((key, tx) -> tx.cardNumber())
    .groupByKey(Grouped.with(Serdes.String(), transactionSerde))
    .aggregate(
        FraudDetectionState::empty,
        (cardNumber, newTransaction, currentState) -> { countTransactionsInWindow() },
        Materialized.as("fraud-detection-store")...
    )
    .toStream()

Enter fullscreen mode Exit fullscreen mode

This line is key:

Materialized.as("fraud-detection-store")

Enter fullscreen mode Exit fullscreen mode

It tells Kafka Streams to keep the state locally using an embedded store.

As new transactions arrive, the state for each card is updated right there in the application, so the system can keep track of recent activity in real time.

At this point, we can evaluate transaction activity in real time without relying on database calls.

But that raises an important question: What exactly are we measuring?

Which patterns or signals should trigger a fraud decision?

This is where guardrails come in.

Stage 1: Guardrails, Fast Rule-Based Validation

Guardrails are the first layer in our flow.

They’re just simple checks based on rules, used to catch the most obvious fraud cases as early as possible.

In practice, guardrails can be any business or risk rule that makes sense for your domain. They are fully customizable and evolve over time as new fraud patterns emerge.

In our case, we will implement a couple of common examples to illustrate the approach.

Impossible Travel Check

This rule checks if the same card shows up in places that don’t make sense given the time between transactions.

For example, if a card is used in New York and then shows up in São Paulo less than an hour later, that’s not realistic. Something is clearly wrong, and we treat that as a strong fraud signal.

public boolean isImpossibleTravelFraud(Transaction previous, Transaction current) {
    double distanceKm = calculateDistanceKm(
        previous.latitude(), previous.longitude(),
        current.latitude(), current.longitude()
    );
    Duration timeBetween = Duration.between(
        previous.transactionTime(),
        current.transactionTime()
    ).abs();
    // block if physically impossible
}

Enter fullscreen mode Exit fullscreen mode

Velocity Check

The velocity check looks at how many times a card is used in a short period of time. A very common pattern in fraud is when someone tests a stolen card by making several small transactions before trying something bigger.

In our case, if the same card is used more than three times within one minute, we flag it and block the transaction.

public boolean isVelocityFraud(long txCountInWindow, long maxAllowedPerWindow) {
    return txCountInWindow > maxAllowedPerWindow;
}

Enter fullscreen mode Exit fullscreen mode

These are just examples. In a real system, guardrails can include a wide range of checks depending on your use case, risk tolerance, and business rules.

Where Rule-Based Detection Falls Short

Guardrails are fast and effective, but they have blind spots. Consider a transaction like this:

  • Amount: $2.00
  • City: same as the cardholder’s usual city
  • Card: same as always
  • Activity: nothing unusual on the surface

This transaction would pass every guardrail check.

But what if $2 micro-charges at gas stations are a known fraud pattern?

This is a classic card-testing technique used before a larger fraudulent purchase.

Rules alone cannot catch this, because the system would need to know the pattern in advance.

This is where a different approach becomes necessary.

Stage 2: Vector Scoring, Behavioral Fraud Detection

Vector Scoring Layer

At this point, the question changes. Instead of asking: “Does this transaction violate a rule?”

We ask: “Does this transaction behave like something we have seen before?”

To answer that, we need a way to compare transactions based on behavior, rather than exact values.

For example:

  • A $12 purchase at a coffee shop
  • A $15 purchase at a bakery

These are not identical, but behaviorally they are very similar.

On the other hand:

  • A $10,000 crypto transaction at 3 AM

This is clearly very different.

So how do we represent this idea in a way that a system can understand?

From transactions to Vectors

To compare behavior, we need a consistent way to represent a transaction.

Instead of working with raw fields such as amount, merchant, or location separately, we transform the entire transaction into a single structured representation. One simple way to think about it is this:

We take a transaction like:

{ "amount": 15.90, "merchant": "Gas Station", "city": "New York" }

Enter fullscreen mode Exit fullscreen mode

And turn it into a descriptive text:

String textToEmbed = "merchant=%s, city=%s, amount=%s";

Enter fullscreen mode Exit fullscreen mode

This text is then passed to an embedding model, which produces a vector like:

[0.0392, 0.9323, -0.0323, ...]

Enter fullscreen mode Exit fullscreen mode

Embedding generator

In our case, we are using the voyage-finance-2 model, which is specialized for financial data from Voyage AI.

This model generates a vector with 1024 dimensions, where each value contributes to capturing a different aspect of the transaction’s behavior.

In the application, this embedding generation step can be implemented with a simple HTTP client in Spring. For example:

@HttpExchange(
url = "/v1/embeddings",
contentType = MediaType.APPLICATION_JSON_VALUE,
accept = MediaType.APPLICATION_JSON_VALUE
)
public interface VoyageEmbeddingsClient {
@PostExchange
EmbeddingsResponse embed(@RequestBody EmbeddingsRequest body);
}

Enter fullscreen mode Exit fullscreen mode

The role of this client is to send the transaction text to the embedding API and receive back the generated vector.

The model maps that description into a vector space where similar behaviors are placed closer together. This means that transactions that behave similarly will generate vectors that are close to each other.

To make this more intuitive, you can imagine a simplified 2D space.

2D representation

In this space:

  • Everyday purchases like coffee or groceries tend to cluster together
  • Similar types of transactions stay close
  • Unusual patterns, like a late-night crypto transaction, appear far apart

In reality, this space is not 2D, but has hundreds or thousands of dimensions.

That is what allows the model to capture subtle patterns that are not obvious when looking at raw transaction data.

The result is a numerical representation of the transaction that preserves its behavior, making it possible to compare transactions efficiently using vector similarity.

Building the Baseline: Seeding the Fraud Pattern Collection

Now that we can convert transactions into vectors, we need something to compare them against. Vector similarity only works if we have a reference dataset.

This means we need a collection of known patterns that represent both normal and fraudulent behavior. To do that, we create a fraud_patterns collection in MongoDB. Each document in this collection contains:

  • The original transaction data
  • A fraud label (true or false)
  • The corresponding embedding

A normal pattern might look like this:

{ 
"fraud": false, 
"transaction": {
"amount": 13.45, 
"merchant": "Coffee Shop" 
},
"embedding": [ ... ]
}

Enter fullscreen mode Exit fullscreen mode

A fraudulent pattern might look like this:

{ 
"fraud": true, 
"transaction": {
"amount": 87422.45, 
"merchant": "Crypto Exchange" 
},
"embedding": [ ... ]
}

Enter fullscreen mode Exit fullscreen mode

This dataset acts as the baseline for comparison.

Instead of asking “Is this transaction fraudulent?”, we ask: “Does this transaction look like something we already know?”

How This Collection Evolves

This collection does not need to be perfect from the start. It typically begins with a small, curated set of known patterns. As the system runs in production:

  • Confirmed fraud cases can be added
  • New patterns can be introduced immediately
  • The system improves over time

One important advantage here is that we do not need to retrain a model. We simply add new examples to the dataset.

Running Vector Search with MongoDB

To run a vector search, we need to create a vector index. This index allows MongoDB to efficiently search for similar vectors.

{
"fields": [
  {
"type": "vector",
"path": "embedding",
"numDimensions": 1024,
"similarity": "cosine"
   }
]
}

Enter fullscreen mode Exit fullscreen mode


java

Each field here matters.

  • type: "vector" tells MongoDB that this field contains embeddings
  • path: "embedding" defines where the vector is stored in the document
  • numDimensions: 1024 must match the size of the vectors produced by the model
  • similarity: "cosine" defines how similarity between vectors is calculated

If the number of dimensions does not match the embedding model, the index will not work correctly.

Performing the Search

Once the index is in place, MongoDB can efficiently find the nearest vectors. From Spring Data MongoDB, this becomes very straightforward:

@Repository
public interface FraudPatternRepository extends MongoRepository<FraudPattern, String> {

@VectorSearch(
indexName = "fraud_patterns_vector_index",
limit = "10",
numCandidates = "200"
)
SearchResults<FraudPattern> searchTopFraudPatternsByEmbeddingNear(
Vector vector,
Score score
);
}

Enter fullscreen mode Exit fullscreen mode

This method takes the transaction embedding as input and returns the most similar patterns stored in the collection.

Not All Matches Are Equal: Introducing the Threshold

After running the search, MongoDB returns the closest matches along with similarity scores. For example:

  • Result 1 = score: 0.94
  • Result 2 = score: 0.91
  • Result 3 = score: 0.72
  • Result 4 = score: 0.60

Each score tells us how similar a stored pattern is to the transaction we are analyzing. In practice, vector search returns the closest matches available, even when the similarity is relatively low.

That is why we introduce a threshold.

A threshold defines the minimum similarity score required for a result to be considered relevant.

Any result below this value is ignored. Choosing the right threshold is critical:

  • Too low → many false positives
  • Too high → missed fraud cases

The ideal value depends on your domain and risk tolerance.

The Full Architecture

Putting it all together, the flow looks like this:

Full Architecture

  • Client App produces a transaction event to a Kafka topic.
  • **FraudDetectorProcessor **consumes the event via Kafka Streams.
  • Guardrails run first: velocity check using local state store and impossible travel check. If triggered, the transaction is immediately inserted into the suspicious_transactions collection.
  • If it passes guardrails, it proceeds to Vector Scoring.
  • The transaction is converted into an embedding and compared against fraud patterns in MongoDB using cosine similarity.
  • If the score exceeds the threshold against a fraud-labeled pattern, the transaction is flagged as suspicious. Otherwise, it is approved.
  • Results flow to downstream services via Kafka.

Beyond Rules and Similarity

What we built here is just one way to approach fraud detection.

We combined simple rule-based checks with vector similarity to detect patterns in real time. Guardrails work really well for obvious cases, while vector search helps catch things that don’t follow clear rules.

But fraud detection is not the same for everyone.

Something that looks suspicious for one user might be totally normal for another. For example, a late-night crypto transaction might be unusual for most people, but not for someone who trades crypto regularly.

That’s where things can evolve.

One next step would be to include some form of user profiling, where the system learns how each user behaves over time and uses that as part of the decision.

Instead of only comparing against global patterns, we could also compare against the user’s own history. That adds more context and helps reduce false positives.

At the end of the day, it’s about combining different approaches: simple rules for quick decisions, vector similarity for behavior, and more context to better understand each user.

Source Code

The full project is available on GitHub. You can check out the code, run it locally, and see how everything fits together, from Kafka Streams and guardrails to vector scoring and MongoDB.

You’ll also find all the setup instructions in the README, including how to start Kafka, configure the embedding generator, and run simulations to test different scenarios.