惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

博客园_首页
爱范儿
爱范儿
罗磊的独立博客
V
V2EX
量子位
Last Week in AI
Last Week in AI
Hugging Face - Blog
Hugging Face - Blog
博客园 - 司徒正美
Jina AI
Jina AI
博客园 - 叶小钗
小众软件
小众软件
博客园 - 【当耐特】
Y
Y Combinator Blog
让小产品的独立变现更简单 - ezindie.com
让小产品的独立变现更简单 - ezindie.com
T
Tailwind CSS Blog
博客园 - 聂微东
Microsoft Security Blog
Microsoft Security Blog
美团技术团队
P
Proofpoint News Feed
Cyber Security Advisories - MS-ISAC
Cyber Security Advisories - MS-ISAC
有赞技术团队
有赞技术团队
MongoDB | Blog
MongoDB | Blog
Recent Announcements
Recent Announcements
酷 壳 – CoolShell
酷 壳 – CoolShell

Databricks

Why Talent Transformation Is the Missing Focus of Enterprise AI Public Health Intelligence Shouldn't Require a Data Scientist Mean Time to Detect Is a Data Access Problem First-party audience data is the ad sales relationship now Rethinking Distributed Systems for Serverless Performance and Reliability The AI Scaling Gap Hiding in Digital Native Companies 10 trillion samples a day: Scaling beyond traditional monitoring infra at Databricks AI success starts with clean data, not just better models How nOps Rebuilt Their Cloud Optimization Platform on Databricks Lakebase, and Why Other ISVs Should Too Peril Predicts: Precision Payouts for a Volatile World The foundation of AI scalability: one team, one platform, one operating model The Federal Data Paradox: Rich in Data, Poor in Access Driving Budapest Forward: How BKK Uses Databricks to Transform City Mobility LLM Vs AI: A Practical Guide to Differences, Use Cases, and Tools Model Risk Governance Is Not the Same as Risk Intelligence Generative AI for Business: A Complete Strategy and Implementation Guide Data Science vs Data Engineering: Choosing Analysis or Infrastructure AI Applications: Tools, Use Cases, and Platforms MLOps vs DevOps: A Practical Guide for Data Scientists and IT Teams Top Data Warehouse Tools For Modern Data Analytics Unlocking SAP Business Context in Databricks with Semantic Metadata Delta Sharing The marketing activation gap has a fix: Databricks and Stitch partner to turn data infrastructure into marketing performance Alert Fatigue Is a Business Risk Backstage with Lakebase Shipping Faster isn’t Learning Faster Why Your OEE Dashboard Is Lying to You The Turbine That Tried to Tell You It Was Failing Predicting Readmissions Isn't Enough. Acting in Time Is. Clinical Trials Run Longer Than They Have To. That's a Patient Problem Network Quality Is a Revenue Problem, Not a Technical One
Announcing Lakebase Search: agent-native retrieval built ...
Pranav Aurora · 2026-06-16 · via Databricks

Today, we're introducing Lakebase Search: hybrid vector and full-text retrieval built into Lakebase, available now in beta on AWS and Azure. Powered by two native Postgres extensions, lakebase_vector and lakebase_text, it allows your entire agent loop to rely on a single data backend, a lakebase. 

This brings next-level scale, next-level economics, and agent-first ergonomics. 

Agents transform search into an operational workflow: they retrieve context, reason, act, and remember. This deeply connects the read path (retrieval) with the write path (memory), making instant retrieval essential to access freshly generated insights in real time.

Until now, that loop had no Postgres-native home built for the scale and economics that search at scale demands.

For agents, search is actually an operational workload

Agents now operate 4x more databases on Lakebase than human users do, and their primary requirement is entirely different from a human's. Traditional search engines assume a read-only snapshot of stale data. Agents, however, treat search like a live operational database.

Look at a typical agent schema: chunked documents and embeddings live directly alongside an active conversational memory log. This creates a continuous read/write loop. Agents write new learnings to memory on one turn, and need that exact data fully indexed and searchable on the next. They don't just need fast retrieval; they need instant search on the absolute latest writes.

Search is a strange workload

Search is a unique workload with two defining properties. 

First, you store vastly more data than you actually query, leaving the majority of it cold. 

Second, vector search causes severe data bloat. A 1 KB text file expands when vectorized. This is because the document is split into multiple chunks, with each chunk generating a distinct high-dimensional embedding—even before accounting for index overhead.

When multiplied across thousands of mostly idle tenants, traditional search architectures break down. Industry-standard vector indexes like HNSW are fundamentally memory-bound. Because fast graph traversal relies heavily on the index remaining resident in RAM, hosting cold multi-tenant data is expensive.

Search needs a lakebase

Last year, we introduced Lakebase: a serverless Postgres OLTP architecture where data lives in cheap cloud object storage, but a tiered cache (RAM, local NVMe, pageserver) ensures hot pages read at local-disk latency.

We realized this is the exact architecture modern search needs. But there was a catch: to actually unlock these economics without destroying query speed, you need an index layout explicitly designed to live in a tiered storage hierarchy. Lakebase didn't have one. So, we built it.

By pairing a tiered architecture with a purpose-built tiered index, we achieve:

  • Next-level scale without the speed penalty: By intelligently fetching only the required pages from object storage into a local cache , smaller Postgres instances achieve the same recall and latency without requiring massive compute resources.
  • Next-level economics: The cold tail of vectors sits in nearly-free object storage, while the hot working set lives on NVMe. You pay for what you query, not what you store.

The economics are easiest to see as a table. Per terabyte per month, at cloud list prices:

Where the data lives

Cost

RAM

~$3,000 / TB / month

Local NVMe (cache)

~$100 / TB / month

Object storage

~$20 / TB / month

Our indexing method  lets Lakebase keep only the active working set in RAM. The cold majority rests in object storage, making the system two orders of magnitude cheaper—while delivering the high-performance search your application actually requires.

Bringing lake-native search indexes to Postgres.

When building Lakebase Search, we centered on two non-negotiable properties. 

When building Lakebase Search, we had two strict requirements: it had to be 100% Postgres-native (reusing standard pgvector/tsvector types and ecosystem tools), and the indexing had to be built from the ground up for tiered cloud object storage.

To achieve this, we are launching two new Postgres extensions in Beta today. Both share the same goal: deliver state-of-the-art search relevance without forcing you to over-provision RAM.

  1. lakebase_vector: 32x compression and 1B+ scale.

We retained standard pgvector data types and operators but changed the underlying index type. Because the data remains in native pgvector format, it maintains compatibility and can be exported to other systems. By clustering and compressing vectors using RaBitQ (Randomized Binary Quantization), we shrink the index footprint 32x while maintaining high recall. A 100-million-vector index that previously required 300GB of RAM fits into under 10GB. This reduced memory footprint allows a single index to scale to over 1 billion vectors. The active working set is cached on local NVMe, while the cold tail resides in object storage. 

  1. lakebase_text: True BM25 without the GIN memory bloat.

Postgres handles exact keyword matching via GIN indexes, which must remain resident in RAM to maintain performance. This architecture causes memory costs to scale linearly with dataset size.  

lakebase_text replaces GIN with an index optimized for sequential reads from cloud object storage. It introduces native BM25 relevance ranking to Postgres without the associated RAM footprint. 

Because both extensions execute within the same engine, hybrid search runs in a single SQL query. Vector similarity and keyword relevance are combined via reciprocal rank fusion (RRF), allowing results to be joined and filtered against operational tables.

Postgres is ready for large-scale, serious search workloads

We benchmarked Lakebase Search on LAION-100M—100 million 768-dimensional vectors, top-10 retrieval, on a single instance.  Query performance with a warm cache and a single connection delivers exact nearest neighbor recall with zero bloat:

Recall@10

P99 latency

QPS

0.955

30 ms

51

0.942

18 ms

104

0.926

14 ms

142

Achieving this scale traditionally requires a memory-bound architecture. A standard pgvector HNSW index requires the neighbor graph and its target heap pages to remain resident in RAM for performant traversal. At 100 million vectors: 

  1. pgvector: Requires a 512 GB (64 CPU) instance. Index build takes ~40 hours. Because graph traversal relies on un-localized random access, cold restarts cause heavy disk-read latencies, making the first query take minutes. 
  2. lakebase_vector: Runs on a 192 GB (96 CU / 24 CPU) instance. Index build takes 1.5 hours. Although traversal is still random access, the index layout clusters data so that random lookups are localized within a hot working set on NVMe cache, leaving the cold tail in object storage. The instance scales to zero when idle; the first cold-cache query takes 1.13 seconds. 

This architecture changes how to approach total cost of ownership. Legacy search requires a fixed baseline cost regardless of query volume, while Lakebase tracks actual usage: 

Workload Type

Traditional Architecture (Memory-Bound)

Lakebase Search Architecture

Large Knowledge Bases (Mostly idle)

Fixed baseline costs to keep idle datasets resident in RAM.

Scales compute to zero. You pay only for object storage.

Agent Memory & Chat (Bursty)

Over-provisioned RAM and compute to handle traffic spikes.

Dynamically scales compute for spikes, then scales down to zero.

Search Bars (Sustained)

Massive instances sized to fit the entire dataset in RAM.

Smaller, cheaper instances because the dataset bypasses RAM residency

Lakebase Search enables agent-first ergonomics 

A single backend for memory and context: 

Agents shouldn't have to stitch together a vector database for context and a transactional database for memory. By pushing your retrieval logic directly into the database, your entire agent loop runs on one backend. Because Lakebase Search is Postgres—fully reusing standard pgvector and tsvector types—it plugs natively into your existing MCPs, standard drivers, and connectors. More importantly, because search lives right next to your operational data, you can execute a hybrid search, join against your application's tables, and safely filter by tenant, all in a single SQL query.

Continuous search experimentation

Optimizing chunking strategies or hybrid weights requires trial and error. Instead of exporting data to external batch systems for reprocessing, Lakebase Search connects with the Lakehouse to create a tight feedback loop. You can branch multi-terabyte datasets instantly at zero cost, build indexes out-of-band using parallel compute, and route agent feedback back to the Lakehouse for offline evaluation. 

A dedicated retrieval engine per agent

Traditional architectures require sharing a single search cluster across all agents. Because idle indexes in Lakebase incur near-zero storage costs, you can provision thousands of isolated corpora dedicated to specific agents, users, or sessions. This shifts search from a stale snapshot into an operational read/write loop; data an agent writes on one turn is committed and retrievable on the next with full transactional guarantees. 

A single foundation for the agent loop

Lakebase eliminates the need to wire together separate vector stores, search clusters, and transactional databases. By consolidating the entire lifecycle inside a single Postgres system, it delivers the scale and low cost of tiered cloud object storage alongside the real-time read/write ergonomics required for agentic workflows. 

Lakebase Search is available today in Beta on AWS and Azure. What will your agents build?