Mysql 数据库数据导入,如果数据量大的话会很耗时,可以通过 LOAD DATA LOCAL INFILE 实现高性能数据导入。
核心原理
LOAD DATA INFILE 是MySQL原生支持的批量数据加载方式,其性能优势体现在:
绕过SQL解析器直接操作存储引擎
减少事务日志写入次数
自动禁用索引后再重建(建议大表先手动禁用索引)
基于二进制协议传输数据
环境配置
服务端配置
修改 my.cnf 配置文件(路径参考:/etc/mysql/my.cnf)
1 2 [mysqld] local_infile = on # 启用本地文件加载功能
重启MySQL服务生效:
1 2 sudo system ctl restart mysql # Ubuntu/Debian sudo system ctl restart mysqld # CentOS/RHEL
客户端配置
Python连接器需要启用本地文件加载权限,allow_local_infile 为 True:
1 2 3 4 5 6 7 8 9 import mysql.connectorconn = mysql.connector.connect( host="localhost" , user="your_username" , password="your_password" , database="your_database" , allow_local_infile=True # 关键参数 )
代码示例
基础版(CSV文件导入)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 pythonCopy Codedef bulk_load_csv(csv_path): """高性能CSV数据导入""" conn = mysql.connector.connect( host="localhost" , user="your_username" , password="your_password" , database="your_database" , allow_local_infile=True ) cursor = conn.cursor() load_sql = f""" LOAD DATA LOCAL INFILE '{csv_path} ' INTO TABLE users FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\\n' IGNORE 1 LINES (id, name, age) """
try : cursor.execute(load_sql) conn.commit() print (f"成功导入 {cursor.rowcount} 条数据" ) except mysql.connector.Error as err: print (f"导入失败: {err} " ) finally : cursor.close() conn.close()
# 执行导入(测试10万行CSV) bulk_load_csv('data.csv' )
性能优化技巧
1 2 3 4 5 6 7 8 sqlCopy Code/* 执行前禁用索引 */ ALTER TABLE users DISABLE KEYS;/* 执行后重建索引 */ ALTER TABLE users ENABLE KEYS;
/* 调整缓冲区大小(按需设置) */ SET GLOBAL bulk_insert_buffer_size = 1024 * 1024 * 256 ; -- 256MB
文章链接:
https://www.zywvvd.com/notes/coding/dataset/mysql/mysql-infile/mysql-infile/