惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

V
Vulnerabilities – Threatpost
V
V2EX
GbyAI
GbyAI
Recent Announcements
Recent Announcements
Microsoft Security Blog
Microsoft Security Blog
阮一峰的网络日志
阮一峰的网络日志
Hugging Face - Blog
Hugging Face - Blog
T
Tailwind CSS Blog
Y
Y Combinator Blog
C
Check Point Blog
爱范儿
爱范儿
CTFtime.org: upcoming CTF events
CTFtime.org: upcoming CTF events
美团技术团队
雷峰网
雷峰网
IT之家
IT之家
WordPress大学
WordPress大学
V
Visual Studio Blog
Microsoft Azure Blog
Microsoft Azure Blog
MyScale Blog
MyScale Blog
N
News and Events Feed by Topic
罗磊的独立博客
S
SegmentFault 最新的问题
S
Security Affairs
aimingoo的专栏
aimingoo的专栏
F
Fortinet All Blogs
K
KPMG report finds enterprise disconnect between AI and its ROI | CIO
Exploit-DB.com RSS Feed
Exploit-DB.com RSS Feed
H
Hacker News: Front Page
Google DeepMind News
Google DeepMind News
B
Blog
O
OpenAI News
C
Cisco Blogs
Simon Willison's Weblog
Simon Willison's Weblog
The Last Watchdog
The Last Watchdog
Hacker News: Ask HN
Hacker News: Ask HN
博客园_首页
人人都是产品经理
人人都是产品经理
C
Cybersecurity and Infrastructure Security Agency CISA
Recent Commits to openclaw:main
Recent Commits to openclaw:main
Help Net Security
Help Net Security
月光博客
月光博客
J
Java Code Geeks
L
LangChain Blog
博客园 - 司徒正美
Stack Overflow Blog
Stack Overflow Blog
Security Archives - TechRepublic
Security Archives - TechRepublic
Apple Machine Learning Research
Apple Machine Learning Research
T
The Exploit Database - CXSecurity.com
N
News and Events Feed by Topic
freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More

博客园 - -银光-

从 305 GB 到 7.4 GB:大模型 KVCache 架构演进全景 大模型推理引擎中的 Beam Search:工程挑战、主流实现与 SGLang 深度优化 从词向量到大模型:NLP 技术演进浅记 基于 mini-sglang 学习大模型推理关键功能 【短文】大模型推理加速:从面向对象到面向数据设计 大模型推理加速:Overlap Scheduling 的深入剖析与性能权衡艺术 vLLM 权重加载机制全解析:从挑战到理想架构 基于 nano-vLLM 学习大模型推理关键功能 SGLang 的 DP Attention 模式浅析 SGLang 的 PP 模式浅析 SGLang 的 TP 模式浅析 SGLang 分布式集群模式概览 了解英伟达和黄仁勋——基于《英伟达之道》和《英伟达之芯》 C++ lambda 引用捕获临时对象引发 coredump 的案例 GCC8 编译优化 BUG 导致的内存泄漏 C++小练习:字符串分割的高性能实现 B+树的Copy-on-Write设计 so库链接和运行时选择哪个路径下的库? Xapian索引-文档检索过程分析之匹配百分比 Xapian索引-文档检索过程分析 Xapian的内存索引-添加文档 Xapian使用入门
Xapian的内存索引
-银光- · 2019-02-22 · via 博客园 - -银光-

  关键字:xapian、内存索引

  xapian除了提供用于生产环境的磁盘索引,也提供了内存索引(InMemoryDatabase)。内存索引。我们可以通过观察内存索引的设计,来了解xapian的设计思路。

1 用途

  官方文档说法:

  “inmemory, This type is a database held entirely in memory. It was originally written for testing purposes only, but may prove useful for building up temporary small databases.”

  早期版本的源码说明:

  “This backend stores a database entirely in memory.  When the database is closed these indexed contents are lost.This is useful for searching through relatively small amounts of data (such as a single large file) which hasn't previously been indexed.”

  早期版本的源码注释:

  “This is a prototype database, mainly used for debugging and testing”

  总的来说,这是一个原型DB,最初只用来做测试和debug用,没有持久化,关闭DB数据就丢失,可以用来处理小量数据的搜索,并且这部分数据可以在内存中实时建索引。

2 使用内存索引

/***************************************************************************
*
* @file ram_xapian.cpp
* @author cswuyg
* @date 2019/02/21
* @brief
*
**************************************************************************/
// inmemory index use deprecated class, disalbe the compile error.
#pragma warning(disable : 4996)
#include <iostream>
#include "xapian.h"

#pragma comment(lib, "libxapian.a")
#pragma comment(lib, "zdll.lib")

const char* const K_DB_PATH = "index_data";
const char* const K_DOC_UNIQUE_ID = "007";

Xapian::WritableDatabase createIndex() {
    std::cout << "--index start--" << std::endl;
    Xapian::WritableDatabase db = Xapian::InMemory::open();

    Xapian::Document doc;
    doc.add_posting("T世界", 1);
    doc.add_posting("T体育", 1);
    doc.add_posting("T比赛", 1);
    doc.set_data("世界体育比赛");
    doc.add_boolean_term(K_DOC_UNIQUE_ID);

    Xapian::docid innerId = db.replace_document(K_DOC_UNIQUE_ID, doc);

    std::cout << "add doc innerId=" << innerId << std::endl;

    db.commit();

    std::cout << "--index finish--" << std::endl;
    return db;
}

void queryIndex(Xapian::WritableDatabase db) {
    std::cout << "--search start--" << std::endl;
    Xapian::Query termOne = Xapian::Query("T世界");
    Xapian::Query termTwo = Xapian::Query("T比赛");
    Xapian::Query termThree = Xapian::Query("T体育");
    auto query = Xapian::Query(Xapian::Query::OP_OR, Xapian::Query(Xapian::Query::OP_OR, termOne, termTwo), termThree);
    std::cout << "query=" << query.get_description() << std::endl;

    Xapian::Enquire enquire(db);
    enquire.set_query(query);
    Xapian::MSet result = enquire.get_mset(0, 10);
    std::cout << "find results count=" << result.get_matches_estimated() << std::endl;

    for (auto it = result.begin(); it != result.end(); ++it) {
        Xapian::Document doc = it.get_document();
        std::string data = doc.get_data();
        Xapian::weight docScoreWeight = it.get_weight();
        Xapian::percent docScorePercent = it.get_percent();

        std::cout << "doc=" << data << ",weight=" << docScoreWeight << ",percent=" << docScorePercent << std::endl;
    }

    std::cout << "--search finish--" << std::endl;
}

int main() {
    auto db = createIndex();
    queryIndex(db);
    return 0;
}

github: https://github.com/cswuyg/xapian_exercise/tree/master/ram_xapian

3 数据结构

内存索引包含一系列数据结构,通过这些数据结构,可以一窥xapian的索引设计思路。

内存索引数据结构如下图所示:

几个主要的操作类封装

InMemoryPostList:内存中的postlist,单个term,操作的就是倒排链表;    

InMemoryAllDocsPostList:内存中的postlist,整个DB,操作的实际上是termlist表(doc表);

InMemoryTermList: 某个doc的term列表;

InMemoryDatabase: 内存DB;

InMemoryAllTermsList: 内存中的termlist,实际上是整个DB的postlists;

InMemoryDocument:单个doc的操作封装  ;

InMemoryPositionList:内存中的position列表操作封装