惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

F
Full Disclosure
博客园 - 聂微东
博客园_首页
人人都是产品经理
人人都是产品经理
N
News | PayPal Newsroom
云风的 BLOG
云风的 BLOG
U
Unit 42
T
Tailwind CSS Blog
Recent Announcements
Recent Announcements
Security Archives - TechRepublic
Security Archives - TechRepublic
T
The Blog of Author Tim Ferriss
Stack Overflow Blog
Stack Overflow Blog
The Register - Security
The Register - Security
The Hacker News
The Hacker News
博客园 - Franky
Engineering at Meta
Engineering at Meta
Jina AI
Jina AI
月光博客
月光博客
freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More
F
Fortinet All Blogs
Threat Intelligence Blog | Flashpoint
Threat Intelligence Blog | Flashpoint
C
Check Point Blog
C
Cyber Attacks, Cyber Crime and Cyber Security
有赞技术团队
有赞技术团队
TaoSecurity Blog
TaoSecurity Blog
博客园 - 司徒正美
GbyAI
GbyAI
G
Google Developers Blog
B
Blog
G
GRAHAM CLULEY
Y
Y Combinator Blog
雷峰网
雷峰网
爱范儿
爱范儿
酷 壳 – CoolShell
酷 壳 – CoolShell
D
Darknet – Hacking Tools, Hacker News & Cyber Security
Microsoft Azure Blog
Microsoft Azure Blog
WordPress大学
WordPress大学
V
V2EX
罗磊的独立博客
Know Your Adversary
Know Your Adversary
AWS News Blog
AWS News Blog
T
Troy Hunt's Blog
S
SegmentFault 最新的问题
P
Privacy & Cybersecurity Law Blog
T
Threat Research - Cisco Blogs
H
Help Net Security
N
Netflix TechBlog - Medium
Help Net Security
Help Net Security
L
LangChain Blog
D
Docker

Databricks

Why Talent Transformation Is the Missing Focus of Enterprise AI Public Health Intelligence Shouldn't Require a Data Scientist Mean Time to Detect Is a Data Access Problem First-party audience data is the ad sales relationship now Rethinking Distributed Systems for Serverless Performance and Reliability The AI Scaling Gap Hiding in Digital Native Companies 10 trillion samples a day: Scaling beyond traditional monitoring infra at Databricks AI success starts with clean data, not just better models How nOps Rebuilt Their Cloud Optimization Platform on Databricks Lakebase, and Why Other ISVs Should Too Peril Predicts: Precision Payouts for a Volatile World The foundation of AI scalability: one team, one platform, one operating model The Federal Data Paradox: Rich in Data, Poor in Access Driving Budapest Forward: How BKK Uses Databricks to Transform City Mobility LLM Vs AI: A Practical Guide to Differences, Use Cases, and Tools Model Risk Governance Is Not the Same as Risk Intelligence Generative AI for Business: A Complete Strategy and Implementation Guide Data Science vs Data Engineering: Choosing Analysis or Infrastructure AI Applications: Tools, Use Cases, and Platforms MLOps vs DevOps: A Practical Guide for Data Scientists and IT Teams Top Data Warehouse Tools For Modern Data Analytics Unlocking SAP Business Context in Databricks with Semantic Metadata Delta Sharing The marketing activation gap has a fix: Databricks and Stitch partner to turn data infrastructure into marketing performance Alert Fatigue Is a Business Risk Backstage with Lakebase Shipping Faster isn’t Learning Faster Why Your OEE Dashboard Is Lying to You The Turbine That Tried to Tell You It Was Failing Predicting Readmissions Isn't Enough. Acting in Time Is. Clinical Trials Run Longer Than They Have To. That's a Patient Problem Network Quality Is a Revenue Problem, Not a Technical One Shelf Availability Starts with Better Demand Visibility When Predicting the Next Hit Requires More Than Intuition Approximate Answers, Exact Decisions: New Sketch Functions for Analytics Companies Winning with AI Built the Data Layer First Rethinking SQL ETL for modern data platforms Stripe data now available on Databricks via Databricks Marketplace Databricks and Stripe Projects: Infrastructure Built for Agents Agents are ready but your architecture probably isn't Interoperability Between Unity Catalog and Google BigQuery via Catalog Federation Built In, Not Bolted On: What AI-Native Actually Means in Cybersecurity Operationalizing AI for public sector fraud prevention From months to minutes: Building real-time clinical data pipelines with natural language Agentic Data Engineering with Genie Code and Lakeflow Securely send first-party conversion signals with Snapchat Conversions API on Databricks Marketplace How leading tech companies are killing the builder’s tax with Lakebase Inside one of the first production deployments of Lakebase: LangGuard's agentic workflow governance engine The next generation of Databricks Genie Model Risk Management in 2026: A Banker’s Guide to the Revised Interagency Guidance OpenAI GPT-5.5 now available on Databricks, fully-governed through Unity AI Gateway Operational databases: How they work and when to use them Databricks partners with OpenAI on GPT-5.5 Announcing the Public Preview of Lakeflow Designer Are LLM agents good at join order optimization? How conversational analytics removes the BI bottleneck How to transform document activation workflows with Genie and Agent Bricks Beyond the spreadsheet: how Databricks is delivering the modern CFO in Financial Services AI App Development: Guide To Building AI-Powered Apps IoT in Manufacturing: Strategy, Components, Use Cases, and Challenges Stop Hand-Coding Change Data Capture Pipelines Multimodal Data Integration: Production Architectures for Healthcare AI Personalization Strategies for Media Companies A Modern AI Risk Management Framework Introducing the Databricks Excel Add-in for Business Users Real-Time Decisioning for AI Agents: Why you Need a Customer Context Layer First A Practical Guide to LLM Fine Tuning AI Data Transformation Guide for Data Engineers and Data Scientists Concurrency Control in DBMS: How Locking, MVCC and Optimistic Strategies Keep Data Consistent Bridging data science and marketing: Databricks unveils Delta Sharing integration for Adobe Experience Platform and agentic marketing workflows Take Control: Customer-Managed Keys for Lakebase Postgres Get hands on with agents, vibe coding and more at Data+ AI Summit Mercedes-Benz Builds a Cross-Cloud Data Mesh with Delta Sharing and Intelligent Replication, Cutting Costs by 66% What Is a Transactional Database? Introducing Genie Agent Mode Governing coding agent sprawl with Unity AI Gateway Governing Coding Agent Sprawl with Unity AI Gateway What is pgvector? Banks Don’t Have an AI Problem – They Have a Data Platform Problem Open Platform, Unified Pipelines: Why dbt on Databricks is Accelerating Why Your Agents Can’t Read Enterprise Documents — and How to Fix It Building with Databricks Document Intelligence and Lakeflow Databricks on Google Cloud: Innovate Faster. Smarter. Together. Introducing the Databricks Connector for Google Sheets: Real-Time, Governed Lakehouse Data in the Sheets Users Love Unity AI Gateway: How to connect agents to external MCPs securely Expanding agent governance with Unity AI Gateway Agentic reasoning in practice: Making sense of structured and unstructured data Agent Bricks: The Governed Enterprise Agent Platform 8 AI and data trends shaping financial services in 2026 Building real-time product search on Databricks Lovable + Databricks: Build Data-Driven Apps at the Speed of Thought Memory scaling for AI agents Powering clinical research innovation: How TriNetX uses Databricks to accelerate drug development Database Branching in Postgres: Git-Style Workflows with Databricks Lakebase How Zalando built a unified data foundation for AI and analytics on Databricks The next era of the open lakehouse: Apache Iceberg™ v3 in Public Preview on Databricks How FSIs eliminate silos between clients, operations, and finance How MakeMyTrip achieved millisecond personalization at scale with Databricks A multi-agent approach to audience intelligence AiChemy: Next-generation agent with MCP, skills and custom data for drug discovery Accelerate business insights with Lakeflow Connect, now with a Free Tier Unlocking Next-Gen Customer Experiences with Data Intelligence for Marketing
Announcing Lakebase Search: agent-native retrieval built into Lakebase Postgres
Pranav Aurora · 2026-06-16 · via Databricks

Today, we're introducing Lakebase Search: hybrid vector and full-text retrieval built into Lakebase, available now in beta on AWS and Azure. Powered by two native Postgres extensions, lakebase_vector and lakebase_text, it allows your entire agent loop to rely on a single data backend, a lakebase. 

This brings next-level scale, next-level economics, and agent-first ergonomics. 

Agents transform search into an operational workflow: they retrieve context, reason, act, and remember. This deeply connects the read path (retrieval) with the write path (memory), making instant retrieval essential to access freshly generated insights in real time.

Until now, that loop had no Postgres-native home built for the scale and economics that search at scale demands.

For agents, search is actually an operational workload

Agents now operate 4x more databases on Lakebase than human users do, and their primary requirement is entirely different from a human's. Traditional search engines assume a read-only snapshot of stale data. Agents, however, treat search like a live operational database.

Look at a typical agent schema: chunked documents and embeddings live directly alongside an active conversational memory log. This creates a continuous read/write loop. Agents write new learnings to memory on one turn, and need that exact data fully indexed and searchable on the next. They don't just need fast retrieval; they need instant search on the absolute latest writes.

Search is a strange workload

Search is a unique workload with two defining properties. 

First, you store vastly more data than you actually query, leaving the majority of it cold. 

Second, vector search causes severe data bloat. A 1 KB text file expands when vectorized. This is because the document is split into multiple chunks, with each chunk generating a distinct high-dimensional embedding—even before accounting for index overhead.

When multiplied across thousands of mostly idle tenants, traditional search architectures break down. Industry-standard vector indexes like HNSW are fundamentally memory-bound. Because fast graph traversal relies heavily on the index remaining resident in RAM, hosting cold multi-tenant data is expensive.

Search needs a lakebase

Last year, we introduced Lakebase: a serverless Postgres OLTP architecture where data lives in cheap cloud object storage, but a tiered cache (RAM, local NVMe, pageserver) ensures hot pages read at local-disk latency.

We realized this is the exact architecture modern search needs. But there was a catch: to actually unlock these economics without destroying query speed, you need an index layout explicitly designed to live in a tiered storage hierarchy. Lakebase didn't have one. So, we built it.

By pairing a tiered architecture with a purpose-built tiered index, we achieve:

  • Next-level scale without the speed penalty: By intelligently fetching only the required pages from object storage into a local cache , smaller Postgres instances achieve the same recall and latency without requiring massive compute resources.
  • Next-level economics: The cold tail of vectors sits in nearly-free object storage, while the hot working set lives on NVMe. You pay for what you query, not what you store.

The economics are easiest to see as a table. Per terabyte per month, at cloud list prices:

Where the data lives

Cost

RAM

~$3,000 / TB / month

Local NVMe (cache)

~$100 / TB / month

Object storage

~$20 / TB / month

Our indexing method  lets Lakebase keep only the active working set in RAM. The cold majority rests in object storage, making the system two orders of magnitude cheaper—while delivering the high-performance search your application actually requires.

Bringing lake-native search indexes to Postgres.

When building Lakebase Search, we centered on two non-negotiable properties. 

When building Lakebase Search, we had two strict requirements: it had to be 100% Postgres-native (reusing standard pgvector/tsvector types and ecosystem tools), and the indexing had to be built from the ground up for tiered cloud object storage.

To achieve this, we are launching two new Postgres extensions in Beta today. Both share the same goal: deliver state-of-the-art search relevance without forcing you to over-provision RAM.

  1. lakebase_vector: 32x compression and 1B+ scale.

We retained standard pgvector data types and operators but changed the underlying index type. Because the data remains in native pgvector format, it maintains compatibility and can be exported to other systems. By clustering and compressing vectors using RaBitQ (Randomized Binary Quantization), we shrink the index footprint 32x while maintaining high recall. A 100-million-vector index that previously required 300GB of RAM fits into under 10GB. This reduced memory footprint allows a single index to scale to over 1 billion vectors. The active working set is cached on local NVMe, while the cold tail resides in object storage. 

  1. lakebase_text: True BM25 without the GIN memory bloat.

Postgres handles exact keyword matching via GIN indexes, which must remain resident in RAM to maintain performance. This architecture causes memory costs to scale linearly with dataset size.  

lakebase_text replaces GIN with an index optimized for sequential reads from cloud object storage. It introduces native BM25 relevance ranking to Postgres without the associated RAM footprint. 

Because both extensions execute within the same engine, hybrid search runs in a single SQL query. Vector similarity and keyword relevance are combined via reciprocal rank fusion (RRF), allowing results to be joined and filtered against operational tables.

Postgres is ready for large-scale, serious search workloads

We benchmarked Lakebase Search on LAION-100M—100 million 768-dimensional vectors, top-10 retrieval, on a single instance.  Query performance with a warm cache and a single connection delivers exact nearest neighbor recall with zero bloat:

Recall@10

P99 latency

QPS

0.955

30 ms

51

0.942

18 ms

104

0.926

14 ms

142

Achieving this scale traditionally requires a memory-bound architecture. A standard pgvector HNSW index requires the neighbor graph and its target heap pages to remain resident in RAM for performant traversal. At 100 million vectors: 

  1. pgvector: Requires a 512 GB (64 CPU) instance. Index build takes ~40 hours. Because graph traversal relies on un-localized random access, cold restarts cause heavy disk-read latencies, making the first query take minutes. 
  2. lakebase_vector: Runs on a 192 GB (96 CU / 24 CPU) instance. Index build takes 1.5 hours. Although traversal is still random access, the index layout clusters data so that random lookups are localized within a hot working set on NVMe cache, leaving the cold tail in object storage. The instance scales to zero when idle; the first cold-cache query takes 1.13 seconds. 

This architecture changes how to approach total cost of ownership. Legacy search requires a fixed baseline cost regardless of query volume, while Lakebase tracks actual usage: 

Workload Type

Traditional Architecture (Memory-Bound)

Lakebase Search Architecture

Large Knowledge Bases (Mostly idle)

Fixed baseline costs to keep idle datasets resident in RAM.

Scales compute to zero. You pay only for object storage.

Agent Memory & Chat (Bursty)

Over-provisioned RAM and compute to handle traffic spikes.

Dynamically scales compute for spikes, then scales down to zero.

Search Bars (Sustained)

Massive instances sized to fit the entire dataset in RAM.

Smaller, cheaper instances because the dataset bypasses RAM residency

Lakebase Search enables agent-first ergonomics 

A single backend for memory and context: 

Agents shouldn't have to stitch together a vector database for context and a transactional database for memory. By pushing your retrieval logic directly into the database, your entire agent loop runs on one backend. Because Lakebase Search is Postgres—fully reusing standard pgvector and tsvector types—it plugs natively into your existing MCPs, standard drivers, and connectors. More importantly, because search lives right next to your operational data, you can execute a hybrid search, join against your application's tables, and safely filter by tenant, all in a single SQL query.

Continuous search experimentation

Optimizing chunking strategies or hybrid weights requires trial and error. Instead of exporting data to external batch systems for reprocessing, Lakebase Search connects with the Lakehouse to create a tight feedback loop. You can branch multi-terabyte datasets instantly at zero cost, build indexes out-of-band using parallel compute, and route agent feedback back to the Lakehouse for offline evaluation. 

A dedicated retrieval engine per agent

Traditional architectures require sharing a single search cluster across all agents. Because idle indexes in Lakebase incur near-zero storage costs, you can provision thousands of isolated corpora dedicated to specific agents, users, or sessions. This shifts search from a stale snapshot into an operational read/write loop; data an agent writes on one turn is committed and retrievable on the next with full transactional guarantees. 

A single foundation for the agent loop

Lakebase eliminates the need to wire together separate vector stores, search clusters, and transactional databases. By consolidating the entire lifecycle inside a single Postgres system, it delivers the scale and low cost of tiered cloud object storage alongside the real-time read/write ergonomics required for agentic workflows. 

Lakebase Search is available today in Beta on AWS and Azure. What will your agents build?