惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

L
LangChain Blog
博客园 - 司徒正美
美团技术团队
WordPress大学
WordPress大学
奇客Solidot–传递最新科技情报
奇客Solidot–传递最新科技情报
人人都是产品经理
人人都是产品经理
让小产品的独立变现更简单 - ezindie.com
让小产品的独立变现更简单 - ezindie.com
T
Troy Hunt's Blog
S
Schneier on Security
T
The Exploit Database - CXSecurity.com
P
Proofpoint News Feed
云风的 BLOG
云风的 BLOG
Engineering at Meta
Engineering at Meta
Cisco Talos Blog
Cisco Talos Blog
T
Tor Project blog
B
Blog
NISL@THU
NISL@THU
月光博客
月光博客
博客园 - 【当耐特】
AWS News Blog
AWS News Blog
cs.CL updates on arXiv.org
cs.CL updates on arXiv.org
腾讯CDC
L
Lohrmann on Cybersecurity
The Cloudflare Blog
L
LINUX DO - 最新话题
S
Security @ Cisco Blogs
S
Secure Thoughts
Spread Privacy
Spread Privacy
有赞技术团队
有赞技术团队
The Last Watchdog
The Last Watchdog
Project Zero
Project Zero
cs.AI updates on arXiv.org
cs.AI updates on arXiv.org
Vercel News
Vercel News
H
Hacker News: Front Page
S
SegmentFault 最新的问题
Schneier on Security
Schneier on Security
aimingoo的专栏
aimingoo的专栏
P
Privacy & Cybersecurity Law Blog
博客园 - 三生石上(FineUI控件)
Forbes - Security
Forbes - Security
C
CXSECURITY Database RSS Feed - CXSecurity.com
I
InfoQ
T
Tailwind CSS Blog
Application and Cybersecurity Blog
Application and Cybersecurity Blog
G
GRAHAM CLULEY
W
WeLiveSecurity
小众软件
小众软件
Recorded Future
Recorded Future
Cyberwarzone
Cyberwarzone
cs.CV updates on arXiv.org
cs.CV updates on arXiv.org

博客园 - Donal

docker命令 NLP | 自然语言处理 - 语言模型(Language Modeling) windows: Python安装scipy,scikit-image时提示"no lapack/blas resources found"的解决方法 Sense2vec with spaCy and Gensim nohup command > myout.file 2>&1 & NLTK vs SKLearn vs Gensim vs TextBlob vs spaCy Gensim进阶教程:训练word2vec与doc2vec模型 Gensim入门教程 使用pdb调试python git只clone仓库中指定子目录 转:深度学习与自然语言处理之五:从RNN到LSTM 转:如何构建爬虫代理服务? RHEL7下安装使用TensorFlow和kcws RHEL7 -- Linux搭建FTP虚拟用户 解决windows10搜索不到内容的问题 forward和redirect 的区别 RHEL7磁盘分区挂载和格式化 Spring注解 100 open source Big Data architecture papers for data professionals
python 去停用词
Donal · 2017-05-25 · via 博客园 - Donal

Try caching the stopwords object, as shown below. Constructing this each time you call the function seems to be the bottleneck.

    from nltk.corpus import stopwords

    cachedStopWords = stopwords.words("english")

    def testFuncOld():
        text = 'hello bye the the hi'
        text = ' '.join([word for word in text.split() if word not in stopwords.words("english")])

    def testFuncNew():
        text = 'hello bye the the hi'
        text = ' '.join([word for word in text.split() if word not in cachedStopWords])

    if __name__ == "__main__":
        for i in xrange(10000):
            testFuncOld()
            testFuncNew()

I ran this through the profiler: python -m cProfile -s cumulative test.py. The relevant lines are posted below.

nCalls Cumulative Time

10000 7.723 words.py:7(testFuncOld)

10000 0.140 words.py:11(testFuncNew)

So, caching the stopwords instance gives a ~70x speedup.