惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

Recent Commits to openclaw:main
Recent Commits to openclaw:main
P
Palo Alto Networks Blog
C
Cybersecurity and Infrastructure Security Agency CISA
C
Cisco Blogs
Cyberwarzone
Cyberwarzone
S
Schneier on Security
T
Threatpost
cs.CL updates on arXiv.org
cs.CL updates on arXiv.org
Scott Helme
Scott Helme
T
Tor Project blog
P
Privacy & Cybersecurity Law Blog
V
Vulnerabilities – Threatpost
L
Lohrmann on Cybersecurity
T
The Exploit Database - CXSecurity.com
Know Your Adversary
Know Your Adversary
The Hacker News
The Hacker News
Security Latest
Security Latest
A
Arctic Wolf
P
Proofpoint News Feed
Google DeepMind News
Google DeepMind News
K
KPMG report finds enterprise disconnect between AI and its ROI | CIO
CTFtime.org: upcoming CTF events
CTFtime.org: upcoming CTF events
Vercel News
Vercel News
WordPress大学
WordPress大学
美团技术团队
C
Cyber Attacks, Cyber Crime and Cyber Security
罗磊的独立博客
Microsoft Security Blog
Microsoft Security Blog
量子位
H
Help Net Security
Webroot Blog
Webroot Blog
月光博客
月光博客
S
SegmentFault 最新的问题
cs.CV updates on arXiv.org
cs.CV updates on arXiv.org
Schneier on Security
Schneier on Security
freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More
IT之家
IT之家
H
Hacker News: Front Page
Exploit-DB.com RSS Feed
Exploit-DB.com RSS Feed
Jina AI
Jina AI
云风的 BLOG
云风的 BLOG
J
Java Code Geeks
www.infosecurity-magazine.com
www.infosecurity-magazine.com
Last Week in AI
Last Week in AI
C
CXSECURITY Database RSS Feed - CXSecurity.com
Spread Privacy
Spread Privacy
S
Security @ Cisco Blogs
博客园 - Franky
奇客Solidot–传递最新科技情报
奇客Solidot–传递最新科技情报
D
Darknet – Hacking Tools, Hacker News & Cyber Security

博客园 - Donal

docker命令 NLP | 自然语言处理 - 语言模型(Language Modeling) windows: Python安装scipy,scikit-image时提示"no lapack/blas resources found"的解决方法 Sense2vec with spaCy and Gensim nohup command > myout.file 2>&1 & NLTK vs SKLearn vs Gensim vs TextBlob vs spaCy Gensim进阶教程:训练word2vec与doc2vec模型 Gensim入门教程 使用pdb调试python git只clone仓库中指定子目录 转:深度学习与自然语言处理之五:从RNN到LSTM 转:如何构建爬虫代理服务? RHEL7下安装使用TensorFlow和kcws RHEL7 -- Linux搭建FTP虚拟用户 解决windows10搜索不到内容的问题 forward和redirect 的区别 RHEL7磁盘分区挂载和格式化 Spring注解 100 open source Big Data architecture papers for data professionals
python 去停用词
Donal · 2017-05-25 · via 博客园 - Donal

Try caching the stopwords object, as shown below. Constructing this each time you call the function seems to be the bottleneck.

    from nltk.corpus import stopwords

    cachedStopWords = stopwords.words("english")

    def testFuncOld():
        text = 'hello bye the the hi'
        text = ' '.join([word for word in text.split() if word not in stopwords.words("english")])

    def testFuncNew():
        text = 'hello bye the the hi'
        text = ' '.join([word for word in text.split() if word not in cachedStopWords])

    if __name__ == "__main__":
        for i in xrange(10000):
            testFuncOld()
            testFuncNew()

I ran this through the profiler: python -m cProfile -s cumulative test.py. The relevant lines are posted below.

nCalls Cumulative Time

10000 7.723 words.py:7(testFuncOld)

10000 0.140 words.py:11(testFuncNew)

So, caching the stopwords instance gives a ~70x speedup.