惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

雷峰网
雷峰网
S
Security @ Cisco Blogs
V
Visual Studio Blog
WordPress大学
WordPress大学
Hugging Face - Blog
Hugging Face - Blog
The Cloudflare Blog
云风的 BLOG
云风的 BLOG
小众软件
小众软件
钛媒体:引领未来商业与生活新知
钛媒体:引领未来商业与生活新知
爱范儿
爱范儿
B
Blog RSS Feed
Recorded Future
Recorded Future
The GitHub Blog
The GitHub Blog
宝玉的分享
宝玉的分享
博客园 - 司徒正美
U
Unit 42
G
Google Developers Blog
博客园 - Franky
阮一峰的网络日志
阮一峰的网络日志
H
Hackread – Cybersecurity News, Data Breaches, AI and More
I
InfoQ
T
The Blog of Author Tim Ferriss
F
Fortinet All Blogs
博客园 - 【当耐特】
腾讯CDC
博客园 - 聂微东
IT之家
IT之家
Martin Fowler
Martin Fowler
cs.CV updates on arXiv.org
cs.CV updates on arXiv.org
奇客Solidot–传递最新科技情报
奇客Solidot–传递最新科技情报
Recent Announcements
Recent Announcements
The Register - Security
The Register - Security
PCI Perspectives
PCI Perspectives
C
Check Point Blog
AI
AI
Webroot Blog
Webroot Blog
I
Intezer
博客园 - 三生石上(FineUI控件)
Google DeepMind News
Google DeepMind News
L
LangChain Blog
C
Cybersecurity and Infrastructure Security Agency CISA
S
Schneier on Security
C
CERT Recently Published Vulnerability Notes
Forbes - Security
Forbes - Security
S
Secure Thoughts
Y
Y Combinator Blog
Google Online Security Blog
Google Online Security Blog
H
Heimdal Security Blog
有赞技术团队
有赞技术团队
L
Lohrmann on Cybersecurity

博客园 - Donal

docker命令 NLP | 自然语言处理 - 语言模型(Language Modeling) windows: Python安装scipy,scikit-image时提示"no lapack/blas resources found"的解决方法 Sense2vec with spaCy and Gensim nohup command > myout.file 2>&1 & NLTK vs SKLearn vs Gensim vs TextBlob vs spaCy Gensim进阶教程:训练word2vec与doc2vec模型 Gensim入门教程 使用pdb调试python git只clone仓库中指定子目录 转:深度学习与自然语言处理之五:从RNN到LSTM 转:如何构建爬虫代理服务? RHEL7下安装使用TensorFlow和kcws RHEL7 -- Linux搭建FTP虚拟用户 解决windows10搜索不到内容的问题 forward和redirect 的区别 RHEL7磁盘分区挂载和格式化 Spring注解 100 open source Big Data architecture papers for data professionals
python 去停用词
Donal · 2017-05-25 · via 博客园 - Donal

Try caching the stopwords object, as shown below. Constructing this each time you call the function seems to be the bottleneck.

    from nltk.corpus import stopwords

    cachedStopWords = stopwords.words("english")

    def testFuncOld():
        text = 'hello bye the the hi'
        text = ' '.join([word for word in text.split() if word not in stopwords.words("english")])

    def testFuncNew():
        text = 'hello bye the the hi'
        text = ' '.join([word for word in text.split() if word not in cachedStopWords])

    if __name__ == "__main__":
        for i in xrange(10000):
            testFuncOld()
            testFuncNew()

I ran this through the profiler: python -m cProfile -s cumulative test.py. The relevant lines are posted below.

nCalls Cumulative Time

10000 7.723 words.py:7(testFuncOld)

10000 0.140 words.py:11(testFuncNew)

So, caching the stopwords instance gives a ~70x speedup.