惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

SecWiki News
SecWiki News
OSCHINA 社区最新新闻
OSCHINA 社区最新新闻
V
Visual Studio Blog
博客园 - 叶小钗
S
SegmentFault 最新的问题
IT之家
IT之家
大猫的无限游戏
大猫的无限游戏
博客园_首页
Apple Machine Learning Research
Apple Machine Learning Research
钛媒体:引领未来商业与生活新知
钛媒体:引领未来商业与生活新知
月光博客
月光博客
酷 壳 – CoolShell
酷 壳 – CoolShell
腾讯CDC
D
Darknet – Hacking Tools, Hacker News & Cyber Security
V
V2EX
阮一峰的网络日志
阮一峰的网络日志
L
Lohrmann on Cybersecurity
量子位
C
Cyber Attacks, Cyber Crime and Cyber Security
T
Tor Project blog
J
Java Code Geeks
cs.CL updates on arXiv.org
cs.CL updates on arXiv.org
博客园 - 三生石上(FineUI控件)
Attack and Defense Labs
Attack and Defense Labs
AI
AI
The Cloudflare Blog
T
Tailwind CSS Blog
S
Schneier on Security
爱范儿
爱范儿
PCI Perspectives
PCI Perspectives
Stack Overflow Blog
Stack Overflow Blog
S
Secure Thoughts
Exploit-DB.com RSS Feed
Exploit-DB.com RSS Feed
T
The Exploit Database - CXSecurity.com
博客园 - 【当耐特】
V2EX - 技术
V2EX - 技术
S
Securelist
P
Proofpoint News Feed
T
Threat Research - Cisco Blogs
Help Net Security
Help Net Security
C
Cisco Blogs
N
News and Events Feed by Topic
人人都是产品经理
人人都是产品经理
B
Blog RSS Feed
K
Kaspersky official blog
T
The Blog of Author Tim Ferriss
G
Google Developers Blog
S
Security Affairs
freeCodeCamp Programming Tutorials: Python, JavaScript, Git & More
Simon Willison's Weblog
Simon Willison's Weblog

博客园 - 川川籽

GO面试题:new 和 map 的区别 minikube dashboard ImagePullBackOff 失败问题的解决方法 hashicorp/raft模块实现的raft集群存在节点跨集群身份冲突问题 macos 13安装openvpn - 川川籽 [转发] Go pprof内存指标含义备忘录 自己搭建一个https的dns,让不同的浏览器使用不同的DNS,使用相同的域名访问到不同的主机上 记一次docker buildx build 推送到本地私有仓库出现 connection refused 的问题 Linux C 获取本机IPV4和IPV6地址列表 Mac M1 安装python3.6.x golang map 和 interface 的一些记录 MacOS M1 安装python3.5 使用php的openssl_encrypt和python的pycrypt进行跨语言的对称加密和解密问题 golang random string 【转载】coroutine 与 goroutine 区别 python简单的time ticker 'invalid flag in #cgo LDFLAGS: -w' 问题解决 记录一次python的mysqlclient依赖库报错问题 airflow当触发具有多层subDAG的任务的时候,出现[Duplicate entry ‘xxxx’ for key dag_id]的错误的问题处理 python hash 每次调用结果不一样
Python3并发写文件
川川籽 · 2019-11-20 · via 博客园 - 川川籽

使用python2在进行并发写的时候,发现文件会乱掉,就是某一行中间会插入其他行的内容。
但是在使用python3进行并发写的时候,无论是多进程,还是多线程,都没有出现这个问题,难道是python3的特性吗?

import time
import os
import multiprocessing
from multiprocessing.dummy import Pool as ThreadPool


def write(val, file):
    w = open(file, "a")
    for i in range(100):
        w.write("%s\n" % val)
        time.sleep(0.001)

def thread_write(file):
    res, pools = [], ThreadPool(10)
    for i in range(10):
        val = str(i) * 1000
        res.append(pools.apply_async(func=write, args=(val, file, )))

    while res:
        for ret in res:
            if ret.ready():
                res.remove(ret)
        time.sleep(0.01)

def mutil_write(file):
    pools = multiprocessing.Pool(processes=10)
    res = []
    for i in range(100):
        res.append(pools.apply_async(thread_write, args=(file, )))

    while res:
        for ret in res:
            if ret.ready():
                res.remove(ret)
        time.sleep(0.01)

if __name__ == '__main__':
    file = "./write_test"
    mutil_write(file)

    with open(file) as fb:
        lines = 0
        line_len = []
        for line in fb:
            lines += 1
            line = line.strip()
            line_len.append(len(line))
            if len(line) != 1000:
                raise(Exception("error line: %s, len: %d" % (line, len(line))))

        print("lines:%d, max len:%d, min:%d, avg:%.2f" % (lines, max(line_len), min(line_len), sum(line_len)/len(line_len)))
    os.remove(file)

上面代码,多进程并发写结束后,校验每一行的长度是否是设置好的长度。用python3反复运行,均通过测试没有异常。

$ python3 --version
Python 3.7.4

$ python3 t.py
lines:10000, max len:1000, min:1000, avg:1000.00

如果用python2,则会出现异常:

$ python2 --version
Python 2.7.15

$ python2 t.py
Traceback (most recent call last):
  File "t.py", line 49, in <module>
    raise(Exception("error line: %s, len: %d" % (line, len(line))))
Exception: error line: 333333333333333333333333333333333333333333333333333333333333333333333333333333333333333333330000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, len: 1092