惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

MongoDB | Blog
MongoDB | Blog
大猫的无限游戏
大猫的无限游戏
博客园 - 聂微东
Blog — PlanetScale
Blog — PlanetScale
F
Full Disclosure
雷峰网
雷峰网
T
Tailwind CSS Blog
Hugging Face - Blog
Hugging Face - Blog
D
DataBreaches.Net
量子位
Microsoft Azure Blog
Microsoft Azure Blog
cs.AI updates on arXiv.org
cs.AI updates on arXiv.org
V
Vulnerabilities – Threatpost
人人都是产品经理
人人都是产品经理
Security Archives - TechRepublic
Security Archives - TechRepublic
P
Proofpoint News Feed
Help Net Security
Help Net Security
腾讯CDC
I
InfoQ
Know Your Adversary
Know Your Adversary
CTFtime.org: upcoming CTF events
CTFtime.org: upcoming CTF events
C
Check Point Blog
W
WeLiveSecurity
NISL@THU
NISL@THU
Cyber Security Advisories - MS-ISAC
Cyber Security Advisories - MS-ISAC
N
Netflix TechBlog - Medium
月光博客
月光博客
Schneier on Security
Schneier on Security
Forbes - Security
Forbes - Security
Project Zero
Project Zero
宝玉的分享
宝玉的分享
P
Palo Alto Networks Blog
P
Privacy & Cybersecurity Law Blog
罗磊的独立博客
Latest news
Latest news
Apple Machine Learning Research
Apple Machine Learning Research
V
V2EX
Stack Overflow Blog
Stack Overflow Blog
S
Security Affairs
I
Intezer
爱范儿
爱范儿
Microsoft Security Blog
Microsoft Security Blog
A
Arctic Wolf
有赞技术团队
有赞技术团队
云风的 BLOG
云风的 BLOG
T
Tenable Blog
Hacker News: Ask HN
Hacker News: Ask HN
L
LINUX DO - 最新话题
奇客Solidot–传递最新科技情报
奇客Solidot–传递最新科技情报
freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More

博客园 - Donal

docker命令 NLP | 自然语言处理 - 语言模型(Language Modeling) windows: Python安装scipy,scikit-image时提示"no lapack/blas resources found"的解决方法 Sense2vec with spaCy and Gensim nohup command > myout.file 2>&1 & NLTK vs SKLearn vs Gensim vs TextBlob vs spaCy Gensim进阶教程:训练word2vec与doc2vec模型 Gensim入门教程 使用pdb调试python git只clone仓库中指定子目录 转:深度学习与自然语言处理之五:从RNN到LSTM 转:如何构建爬虫代理服务? RHEL7下安装使用TensorFlow和kcws RHEL7 -- Linux搭建FTP虚拟用户 解决windows10搜索不到内容的问题 forward和redirect 的区别 RHEL7磁盘分区挂载和格式化 Spring注解 100 open source Big Data architecture papers for data professionals
python 去停用词
Donal · 2017-05-25 · via 博客园 - Donal

Try caching the stopwords object, as shown below. Constructing this each time you call the function seems to be the bottleneck.

    from nltk.corpus import stopwords

    cachedStopWords = stopwords.words("english")

    def testFuncOld():
        text = 'hello bye the the hi'
        text = ' '.join([word for word in text.split() if word not in stopwords.words("english")])

    def testFuncNew():
        text = 'hello bye the the hi'
        text = ' '.join([word for word in text.split() if word not in cachedStopWords])

    if __name__ == "__main__":
        for i in xrange(10000):
            testFuncOld()
            testFuncNew()

I ran this through the profiler: python -m cProfile -s cumulative test.py. The relevant lines are posted below.

nCalls Cumulative Time

10000 7.723 words.py:7(testFuncOld)

10000 0.140 words.py:11(testFuncNew)

So, caching the stopwords instance gives a ~70x speedup.