惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

WordPress大学
WordPress大学
Engineering at Meta
Engineering at Meta
cs.AI updates on arXiv.org
cs.AI updates on arXiv.org
Martin Fowler
Martin Fowler
雷峰网
雷峰网
Recent Announcements
Recent Announcements
博客园 - 叶小钗
F
Full Disclosure
C
Check Point Blog
A
About on SuperTechFans
L
LangChain Blog
Vercel News
Vercel News
T
The Blog of Author Tim Ferriss
博客园 - 司徒正美
C
Cybersecurity and Infrastructure Security Agency CISA
C
CXSECURITY Database RSS Feed - CXSecurity.com
Cisco Talos Blog
Cisco Talos Blog
Hugging Face - Blog
Hugging Face - Blog
Recorded Future
Recorded Future
MongoDB | Blog
MongoDB | Blog
Project Zero
Project Zero
让小产品的独立变现更简单 - ezindie.com
让小产品的独立变现更简单 - ezindie.com
D
Darknet – Hacking Tools, Hacker News & Cyber Security
Know Your Adversary
Know Your Adversary
D
Docker
U
Unit 42
酷 壳 – CoolShell
酷 壳 – CoolShell
T
The Exploit Database - CXSecurity.com
L
Lohrmann on Cybersecurity
奇客Solidot–传递最新科技情报
奇客Solidot–传递最新科技情报
Latest news
Latest news
V
Visual Studio Blog
钛媒体:引领未来商业与生活新知
钛媒体:引领未来商业与生活新知
L
LINUX DO - 最新话题
Google DeepMind News
Google DeepMind News
小众软件
小众软件
NISL@THU
NISL@THU
GbyAI
GbyAI
H
Heimdal Security Blog
Jina AI
Jina AI
I
InfoQ
PCI Perspectives
PCI Perspectives
P
Privacy International News Feed
P
Proofpoint News Feed
N
News and Events Feed by Topic
C
CERT Recently Published Vulnerability Notes
aimingoo的专栏
aimingoo的专栏
SecWiki News
SecWiki News
B
Blog RSS Feed
阮一峰的网络日志
阮一峰的网络日志

祈雨的笔记

安全多方计算MPC spark原理解析 kueue执行源码分析 spark on k8s执行源码分析 spark-operator源码解析 系统压测遇到的缓存击穿问题 我的世界PC与安卓联机 蚂蚁金服流量投放平台的AIG改造 G1大对象致Old区占用率高 日志打印导致接口响应率下跌分析 Groovy加载类导致OOM分析 ERROR日志打印导致CPU满载 记OceanBase死锁超时 应用发版期间服务响应超时 Ark Serverless初探 系统优化复盘一二三 The user specified as a definer does not exist Kong网关初探 API网关选型调研 CPU火焰图常用工具 配置中心选型调研 root操作Nginx导致用户组错误 基于Proxifier使用代理 FastJSON字段智能匹配踩坑 Nacos初探 记一次Nginx服务器CPU满荷载故障 基于券系统分库分表的思考 limit不参与SQL成本计算致索引失效 Linux常用性能监控命令 golang低版本http2偶现400 hostname in certificate didn't match 常见对称加密原理以及应用 tcp_tw_recycle引起的TCP握手失败 记一次mysql执行DDL导致锁表 mysql磁盘占用查看 mysql对text字段update致磁盘增长 elasticsearch报错index read-only TIME_WAIT与Http的Keep-Alive 记一次TIME_WAIT导致连接数报警 记一次生产事故OOM问题排查 redis分布式锁RedissonLock的实现细节 webservice复杂加密签名(2)java调用 webservice复杂加密签名(1)SoapUI mysql延时关联 利用中间人拦截实现APP内H5窜改 MySQL表字符集不同导致关联查询索引失效 通过SSH隧道远程办公 数据落盘方案 BeanDefinitionRegistryPostProcessor扩展 mysql空间索引 HTTPS攻击 spring循环依赖过程解析 elasticsearch性能优化 mysql IS NULL 使用索引 常用加密算法 xml与javaBean转换 初探InnoDB MVCC源码实现 mysql索引原理 redis之list源码分析 redis之key过期源码分析 redis之string源码分析 redis之hash源码分析 线程池之ThreadPoolExecutor mysql数据页结构 Using temporary与Using filesort mysql回表致索引失效 springboot(28)HTTP连接池 定时任务之ScheduledThreadPoolExecutor elasticsearch常用script聚合 elasticsearch实现like查询 elasticsearch实现乐观锁 elasticsearch准实时原理 springboot(27)自定义缓存读写机制CachingConfigurerSupport optimizer tracing arthas常用命令 HTTP和HTTPS详解 redis集群选举机制 kafka消息重试 一点压力测试的经验 kafka架构概念 explain分析sql语句字段的解释 JVM问题分析处理手册 logstash过滤器(2)date logstash过滤器(3)dissect logstash编码器(1)json logstash编码器(2)multiline logstash表达式 logstash输入(1)通用选项 logstash输入(3)file logstash过滤器(1)通用选项 logstash输入(2)stdin logstash安装 记一次前端vConsole导致JSON序列化错误排查 解决多个异步操作嵌套问题 fastjson反序列化失败autoType is not support RTMP串流服务 POI自动调整列宽错误 Nginx+Lua实现动态黑名单 使用curl命令模拟POST和GET请求 ResponseEntity下载文件
mysql字符集utf8mb4失效踩坑
祈雨的笔记 · 2019-08-16 · via 祈雨的笔记

现象

mysql数据表的字符集已经设置成了utf8mb4,但是通过JDBC向数据库写入4字节的emoji表情时报错,但是通过直接使用命令行插入该4字节的emoji表情时却成功了。示例如下:

1
2
3
4
5
6
CREATE TABLE `user_info` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(11) NOT NULL,
`age` int(4) DEFAULT NULL
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

JDBC写入失败

1
2
3
4
User user = new User();
user.setName("\uD83D\uDC8B");
user.setAge(18);
userMapper.insertUser(user);
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
org.springframework.jdbc.UncategorizedSQLException: 
### Error updating database. Cause: java.sql.SQLException: Incorrect string value: '\xF0\x9F\x92\x8B' for column 'name' at row 1
### The error may involve com.wakzz.database.persistence.UserMapper.insertUser-Inline
### The error occurred while setting parameters
### SQL: insert into user_info (name, age) values (?, ? )
### Cause: java.sql.SQLException: Incorrect string value: '\xF0\x9F\x92\x8B' for column 'name' at row 1
; uncategorized SQLException; SQL state [HY000]; error code [1366]; Incorrect string value: '\xF0\x9F\x92\x8B' for column 'name' at row 1; nested exception is java.sql.SQLException: Incorrect string value: '\xF0\x9F\x92\x8B' for column 'name' at row 1

at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:89)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
at org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:73)
at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:446)
at com.sun.proxy.$Proxy81.insert(Unknown Source)
at org.mybatis.spring.SqlSessionTemplate.insert(SqlSessionTemplate.java:278)
at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:58)
at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:59)
at com.sun.proxy.$Proxy92.insertUser(Unknown Source)

命令行写入成功

1
2
mysql> insert into user_info (name,age) values ('💋',18);
Query OK, 1 row affected

解决

修改mysql配置文件,添加以下配置:

1
character_set_server = utf8mb4

修改前字符集

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
mysql> show variables like "%char%";
+
| Variable_name | Value |
+
| character_set_client | utf8mb4 |
| character_set_connection | utf8mb4 |
| character_set_database | latin1 |
| character_set_filesystem | binary |
| character_set_results | utf8mb4 |
| character_set_server | utf8 |
| character_set_system | utf8 |
| character_sets_dir | /usr/soft/mysql-5.6.31/share/charsets/ |
+
8 rows in set

mysql> SHOW VARIABLES LIKE 'collation%';
+
| Variable_name | Value |
+
| collation_connection | utf8mb4_general_ci |
| collation_database | latin1_swedish_ci |
| collation_server | utf8_general_ci |
+
3 rows in set

修改后字符集

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
mysql> show variables like "%char%";
+
| Variable_name | Value |
+
| character_set_client | utf8mb4 |
| character_set_connection | utf8mb4 |
| character_set_database | latin1 |
| character_set_filesystem | binary |
| character_set_results | utf8mb4 |
| character_set_server | utf8mb4 |
| character_set_system | utf8 |
| character_sets_dir | /usr/soft/mysql-5.6.31/share/charsets/ |
+
8 rows in set

mysql> SHOW VARIABLES LIKE 'collation%';
+
| Variable_name | Value |
+
| collation_connection | utf8mb4_general_ci |
| collation_database | latin1_swedish_ci |
| collation_server | utf8mb4_general_ci |
+
3 rows in set

原理

通过抓包比较mysql配置character_set_server修改前后的tcp数据,发现每一个mysql的数据库连接建立成功后会JDBC会自动执行一次字符集设置语句SET NAMES xxx。当character_set_serverutf8时JDBC执行SET NAMES utf8,而当character_set_serverutf8mb4时JDBC执行SET NAMES utf8mb4

image

image

在命令行中测试SET NAMES发现即使数据库表的字符集是utf8mb4时,若执行了SET NAMES utf8也会导致4字节字符写入mysql失败。成功复现了JDBC写入emoji写入异常的问题。

1
2
3
4
5
mysql> SET NAMES utf8;
Query OK, 0 rows affected

mysql> insert into user_info (name,age) values ('💋',18);
1366 - Incorrect string value: '\xF0\x9F\x92\x8B' for column 'name' at row 1
1
2
3
4
5
mysql> SET NAMES utf8mb4;
Query OK, 0 rows affected

mysql> insert into user_info (name,age) values ('💋',18);
Query OK, 1 row affected

在官方文档MySQL Connector/Java中,文档表明JDBC会通过character_set_server的值自动执行SET NAMES命令设置字符集编码。其目的是自动检测服务端字符集编码配置而减少JDBC客户端的字符集编码配置。

Functionality Added or Changed

  • Connector/J did not support utf8mb4 for servers 5.5.2 and newer.
  • Connector/J now auto-detects servers configured with character_set_server=utf8mb4 or treats the Java encoding utf-8 passed using characterEncoding=… as utf8mb4 in the SET NAMES= calls it makes when establishing the connection. (Bug #54175)

继续抓包以及查看JDBC源码发现,JDBC成功与mysql建立连接后,mysql会返回其版本号和一些附带信息例如字符集编码。如下图,0x00000167位表示当前mysql的默认字符集编码。图一的字符集编码值为0x21即十进制33,图二的字符集编码值为0x2D即十进制45

通过翻阅JDBC源码发现,33表示utf845表示utf8mb4,即该值就是mysql的character_set_server值。

image

image

mysql-connector-java-5.1.46-sources.jar!/com/mysql/jdbc/CharsetMapping.java:258

1
2
collation[33] = new Collation(33, "utf8_general_ci", 1, MYSQL_CHARSET_NAME_utf8);
collation[45] = new Collation(45, "utf8mb4_general_ci", 1, MYSQL_CHARSET_NAME_utf8mb4);

在获取mysql的服务器参数后,解析字符集编码:

  • character_set_serverutf8时,执行SET NAMES utf8
  • character_set_serverutf8mb4时,执行SET NAMES utf8mb4

mysql-connector-java-5.1.46-sources.jar!/com/mysql/jdbc/ConnectionImpl.java:1697

1
2
3
4
5
6
7
8
9
10
11
12
13
boolean utf8mb4Supported = versionMeetsMinimum(5, 5, 2);

boolean useutf8mb4 = utf8mb4Supported && (CharsetMapping.UTF8MB4_INDEXES.contains(this.io.serverCharsetIndex));

if (!getUseOldUTF8Behavior()) {
if (dontCheckServerMatch || !characterSetNamesMatches("utf8") || (utf8mb4Supported && !characterSetNamesMatches("utf8mb4"))) {

execSQL(null, "SET NAMES " + (useutf8mb4 ? "utf8mb4" : "utf8"), -1, null, DEFAULT_RESULT_SET_TYPE,
DEFAULT_RESULT_SET_CONCURRENCY, false, this.database, null, false);
this.serverVariables.put("character_set_client", useutf8mb4 ? "utf8mb4" : "utf8");
this.serverVariables.put("character_set_connection", useutf8mb4 ? "utf8mb4" : "utf8");
}
}