惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

AWS News Blog
AWS News Blog
T
Tenable Blog
Project Zero
Project Zero
T
The Exploit Database - CXSecurity.com
L
LINUX DO - 热门话题
T
Threat Research - Cisco Blogs
T
Threatpost
Security Latest
Security Latest
C
Cisco Blogs
L
Lohrmann on Cybersecurity
S
Security @ Cisco Blogs
Google Online Security Blog
Google Online Security Blog
NISL@THU
NISL@THU
AI
AI
V
Vulnerabilities – Threatpost
Google DeepMind News
Google DeepMind News
C
Cyber Attacks, Cyber Crime and Cyber Security
C
CXSECURITY Database RSS Feed - CXSecurity.com
The Last Watchdog
The Last Watchdog
G
GRAHAM CLULEY
Cloudbric
Cloudbric
H
Hackread – Cybersecurity News, Data Breaches, AI and More
H
Hacker News: Front Page
U
Unit 42
A
Arctic Wolf
CTFtime.org: upcoming CTF events
CTFtime.org: upcoming CTF events
MyScale Blog
MyScale Blog
O
OpenAI News
Scott Helme
Scott Helme
V2EX - 技术
V2EX - 技术
P
Proofpoint News Feed
博客园 - 叶小钗
Hugging Face - Blog
Hugging Face - Blog
云风的 BLOG
云风的 BLOG
V
Visual Studio Blog
Application and Cybersecurity Blog
Application and Cybersecurity Blog
Cyberwarzone
Cyberwarzone
博客园 - 【当耐特】
H
Heimdal Security Blog
S
Schneier on Security
阮一峰的网络日志
阮一峰的网络日志
Help Net Security
Help Net Security
D
DataBreaches.Net
Y
Y Combinator Blog
Hacker News - Newest:
Hacker News - Newest: "LLM"
TaoSecurity Blog
TaoSecurity Blog
K
Kaspersky official blog
N
News and Events Feed by Topic
WordPress大学
WordPress大学
P
Palo Alto Networks Blog

博客园 - searchDM

Elasticsearch 集成 Elasticsearch 环境 Elasticsearch 优化 Elasticsearch入门 Elasticsearch概述 吴恩达自然语言处理第一课:分类与向量空间 Linux下C语言字符串操作之字符串转数值型 lucene 3.4 contrib/facet 切面搜索 solr3.4 高亮(highlight),拼写检查(spellCheck),匹配相似(moreLikeThis) 应用实践 堆与堆排序 Trie树|字典树的简介及实现 快速排序 归并排序的实现 Lucene.net搜索结果排序(单条件和多条件) 直接选择排序及交换二个数据的实现 冒泡排序 直接插入排序的三种实现 希尔排序的实现 几乎所有食物的英文翻译
在ubuntu上安装全文搜索中文分词Coreseek/sphinx及和Rails集成
searchDM · 2011-10-17 · via 博客园 - searchDM

Sphinx(狮身人面像) 想必大家都比较了解,就不作介绍了,不了解的童鞋可以自己Google

原生的Sphinx只支持中文
所以这里重点介绍支持中文分词的 Coreseek。

注意:Coreseek 3.2 后,只有安装 Coreseek 就可以了,它对LibMMSeg和sphinx做了整合,不用再安装原生Sphinx。(3.2前是要安装原生Sphinx,还要装补丁,非常繁琐)

安装coreseek

下面以coreseek-3.2.14为例,它基于Sphinx 0.99(不用安装Sphinx 0.99)

详细官方手册:http://www.coreseek.cn/products-install/install_on_bsd_linux/

ubuntu-10.04 安装 coreseek安装需要预装的软件

sudo apt-get install make gcc g++ automake libtool mysql-client libmysqlclient15-dev   libxml2-dev libexpat1-dev
#下载 coreseek-3.2.14,里面已经包含 mmsegcd /tmpwget http://www.coreseek.cn/uploads/csft/3.2/coreseek-3.2.14.tar.gztar xzvf coreseek-3.2.14.tar.gzcd coreseek-3.2.14# 先安装mmsegcd mmseg-3.2.14./bootstrap    #输出的warning信息可以忽略,如果出现error则需要解决./configure --prefix=/opt/mmseg  [color=red]#下面安装coreseek 需要此路径[/color]sudo make && sudo make install#安装coreseekcd ..cd csft-3.2.14sh buildconf.sh    #输出的warning信息可以忽略,如果出现error则需要解决./configure --prefix=/opt/coreseek --without-unixodbc --with-mmseg --with-mmseg-includes=/opt/mmseg/include/mmseg/ --with-mmseg-libs=/opt/mmseg/lib/ --with-mysql    ##如果提示mysql问题,可以查看MySQL数据源安装说明sudo make && sudo make installcd ..#然后建立命令快捷方式,方便使用sudo ln -s /opt/coreseek/bin/indexer   /usr/local/bin/indexersudo ln -s /opt/coreseek/bin/indextool  /usr/local/bin/indextoolsudo ln -s /opt/coreseek/bin/search  /usr/local/bin/searchsudo ln -s /opt/coreseek/bin/searchd  /usr/local/bin/searchdsudo ln -s /opt/coreseek/bin/spelldump /usr/local/bin/spelldump

集成到rails项目

这里用的是gem thinking-sphinx

如果没有新项目自己创建Rails项目,这里不做说明。
可以参考Railscasts: http://railscasts.com/episodes/120-thinking-sphinx?autoplay=true

安装 thinking-sphinx
官方手册:http://freelancing-god.github.com/ts/en/installing_thinking_sphinx.html

这里只说明下 rails 3.0

在Gemfile中加入

gem 'thinking-sphinx'

然后

bundle install

可以参考快速实现:http://freelancing-god.github.com/ts/en/quickstart.html

在model中定义索引

class Post < ActiveRecord::Basedefine_index doindexes :title, :body#声明使用实时索引set_property :delta => true  #注意这句一定要加,否则添加了记录不会自动索引endend

记得添加实时索引字段 delta

ruby script/generate migration add_delta_to_posts delta:boolean
class AddDeltaToPosts < ActiveRecord::Migrationdef self.upadd_column :posts, :delta,:boolean, :default => true, :null => falseenddef self.downremove_column :posts, :deltaendend

rake db:migrate

在controller中加search代码,controler大家自己建啦。

class FullTextSearchController < ApplicationControllerdef searchper_page = params[:limit].to_ipage = (params[:start].to_i / per_page) + 1total_count = ThinkingSphinx.count(params[:query], :classes => [Post], :page => page, :per_page => per_page)@results = ThinkingSphinx.search(params[:query], :classes => [Post], :page => page, :per_page => per_page)respond_to do |format|format.json { render(:json => {:total => total_count, :success => true,:items => @results.map{ |i| {:id  => i.id, :title => i.title, :body => i.body} unless i.blank? },}.to_json)}endendend
#创建rails项目全文搜索数据目录cd 你的rails目录mkdir fulltext_search_data#从安装包中复制字典到Rails项目cp /tmp/coreseek-3.2.14/mmseg-3.2.14/data/*.* 你的rails目录/fulltext_search_data#新建配置文件:vi config/sphinx.yml#内容如下:test:bin_path: /opt/coreseek/binmem_limit: 128Mconfig_file: config/test.sphinx.confcharset_type: zh_cn.utf-8charset_dictpath: <%=::Rails.root.to_s + "/fulltext_search_data "%>pid_file: "/tmp/searchd.test.pid"ngram_len: 0development:bin_path: /opt/coreseek/binmem_limit: 128Mconfig_file: config/development.sphinx.confcharset_type: zh_cn.utf-8charset_dictpath: <%=::Rails.root.to_s + "/fulltext_search_data "%>pid_file: "/tmp/searchd.development.pid"ngram_len: 0production:bin_path: /opt/coreseek/binmem_limit: 128Mconfig_file: config/production.sphinx.confcharset_type: zh_cn.utf-8charset_dictpath: <%=::Rails.root.to_s + "/fulltext_search_data "%>pid_file: "/tmp/searchd.production.pid"ngram_len: 0#建立配置文件rake thinking_sphinx:configure#建立索引rake thinking_sphinx:index#开启服务rake thinking_sphinx:start#现在可以先加些数据,在浏览器中访问你的controller测试搜索#也可以用如下命令来测试全文搜索引擎search '关键字' -c 你的Rsils项目/config/development.sphinx.conf

最后再补几点注意事项
1.定义索引时 "set_property :delta => true", 没有这句新加的记录不会索引。
2.coreseek 安装包中的字典文件一定要复制到Rails项目数据目录,否则中文无法支持。

参考资料

详见:https://github.com/freelancing-god/thinking-sphinx
官方手册:http://freelancing-god.github.com/ts/en/installing_thinking_sphinx.html
快速实现:http://freelancing-god.github.com/ts/en/quickstart.html