惯性聚合 高效追踪和阅读你感兴趣的博客、新闻、科技资讯
阅读原文 在惯性聚合中打开

推荐订阅源

WordPress大学
WordPress大学
Security Latest
Security Latest
C
Cisco Blogs
P
Palo Alto Networks Blog
Know Your Adversary
Know Your Adversary
Project Zero
Project Zero
C
Cyber Attacks, Cyber Crime and Cyber Security
NISL@THU
NISL@THU
Exploit-DB.com RSS Feed
Exploit-DB.com RSS Feed
S
Secure Thoughts
P
Privacy International News Feed
V
Vulnerabilities – Threatpost
D
Docker
Google Online Security Blog
Google Online Security Blog
K
KPMG report finds enterprise disconnect between AI and its ROI | CIO
Recent Announcements
Recent Announcements
T
The Exploit Database - CXSecurity.com
G
Google Developers Blog
Schneier on Security
Schneier on Security
小众软件
小众软件
爱范儿
爱范儿
GbyAI
GbyAI
J
Java Code Geeks
T
Tailwind CSS Blog
Cisco Talos Blog
Cisco Talos Blog
The Hacker News
The Hacker News
D
DataBreaches.Net
Blog — PlanetScale
Blog — PlanetScale
TaoSecurity Blog
TaoSecurity Blog
MyScale Blog
MyScale Blog
B
Blog RSS Feed
Cyberwarzone
Cyberwarzone
有赞技术团队
有赞技术团队
Martin Fowler
Martin Fowler
C
CXSECURITY Database RSS Feed - CXSecurity.com
S
Securelist
L
Lohrmann on Cybersecurity
cs.CL updates on arXiv.org
cs.CL updates on arXiv.org
Y
Y Combinator Blog
S
Schneier on Security
Latest news
Latest news
Apple Machine Learning Research
Apple Machine Learning Research
博客园 - 叶小钗
F
Fortinet All Blogs
M
MIT News - Artificial intelligence
PCI Perspectives
PCI Perspectives
V
V2EX
V2EX - 技术
V2EX - 技术
O
OpenAI News
W
WeLiveSecurity

RSS feed of yihong0618's gitblog

12 月,姥姥,以及大模型 · Issue #344 · yihong0618/gitblog 不要自动化你的情感链接 · Issue #343 · yihong0618/gitblog 那些幸福的瞬间 · Issue #342 · yihong0618/gitblog 12 年过去了,你过得还好么? · Issue #341 · yihong0618/gitblog Be a human, not a claw. · Issue #339 · yihong0618/gitblog 新年快乐以及一些近况 · Issue #338 · yihong0618/gitblog 偶得的佳句与 AI · Issue #337 · yihong0618/gitblog 确定性的丧失 · Issue #336 · yihong0618/gitblog Something Boring or Interesting · Issue #335 · yihong0618/gitblog 2025 年我提了 500 个 Pull Requests · Issue #334 · yihong0618/gitblog 网友 · Issue #333 · yihong0618/gitblog 如何判断一个项目是个好的开源项目并且想贡献? · Issue #331 · yihong0618/gitblog Things I don’t like · Issue #329 · yihong0618/gitblog Notes on Pycon2025 · Issue #327 · yihong0618/gitblog 从错误中学习 · Issue #326 · yihong0618/gitblog 一次艰难的 cpython issue 排查过程,以及我学到了什么 · Issue #325 · yihong0618/gitblog 八月 · Issue #324 · yihong0618/gitblog 从给 Greptime 贡献代码的过程中,我都学到了什么 · Issue #320 · yihong0618/gitblog Notes on May · Issue #318 · yihong0618/gitblog Things I like · Issue #311 · yihong0618/gitblog 我的 35 岁 · Issue #309 · yihong0618/gitblog 连续跑了 365 天 · Issue #306 · yihong0618/gitblog 最近给开源项目贡献的记录 · Issue #304 · yihong0618/gitblog 2020-2025 最喜欢的 12 篇文章和一张图 · Issue #302 · yihong0618/gitblog 断章 · Issue #301 · yihong0618/gitblog 这两个月 · Issue #299 · yihong0618/gitblog 从给 RisingWave 提 PR 说起,聊聊怎么快速切入一个复杂的开源项目 · Issue #296 · yihong0618/gitblog 能几日,又中秋 · Issue #294 · yihong0618/gitblog 第一次 AI 尝试,完全用 AI(MarsCode) 做个需求 · Issue #293 · yihong0618/gitblog 一件趣事儿 · Issue #292 · yihong0618/gitblog 这玩意不是就__ · Issue #291 · yihong0618/gitblog 六月 · Issue #290 · yihong0618/gitblog 跑了 10000km 花了 13 年 · Issue #289 · yihong0618/gitblog 抉择 · Issue #287 · yihong0618/gitblog 三月 · Issue #283 · yihong0618/gitblog About Me · Issue #282 · yihong0618/gitblog 用小爱加手指机器人实现传统开关语音关灯 · Issue #281 · yihong0618/gitblog 记忆碎片 · Issue #280 · yihong0618/gitblog 开源值得么? · Issue #279 · yihong0618/gitblog 聊聊读书 · Issue #275 · yihong0618/gitblog 旧文,移动端最好的游戏,《Dream Quest》以及其它 · Issue #274 · yihong0618/gitblog 微信好友 · Issue #272 · yihong0618/gitblog 如何写一个 PostgreSQL Extension · Issue #270 · yihong0618/gitblog 五月病 · Issue #268 · yihong0618/gitblog 杂感 · Issue #262 · yihong0618/gitblog 开源四年的一些记录 · Issue #259 · yihong0618/gitblog 不用 root 使用小爱同学和 ChatGPT 交互折腾记 · Issue #258 · yihong0618/gitblog 用 eBPF trace PostgreSQL 的几种姿势 · Issue #257 · yihong0618/gitblog 2022 失落的一年 · Issue #255 · yihong0618/gitblog 写诗吧 · Issue #254 · yihong0618/gitblog 利用 GitHubPoster 和 GitHub Actions 备份任意用户推特 · Issue #252 · yihong0618/gitblog 一杯敬月光 · Issue #251 · yihong0618/gitblog 如何判断一个数字帐号来自中国大陆 · Issue #248 · yihong0618/gitblog Random Thoughts · Issue #242 · yihong0618/gitblog 反思 · Issue #241 · yihong0618/gitblog 今天的跑步电台 · Issue #239 · yihong0618/gitblog LeetCode 是如何运行的--测试篇 · Issue #237 · yihong0618/gitblog 新冠纪元 · Issue #235 · yihong0618/gitblog Postgres 需要学的东西 · Issue #234 · yihong0618/gitblog Postgres 中 GUC 的一些记录 · Issue #233 · yihong0618/gitblog 运交华盖欲何求 · Issue #231 · yihong0618/gitblog 年终岁尾 · Issue #230 · yihong0618/gitblog 二零二二的三个小目标 · Issue #229 · yihong0618/gitblog 经历了人生体验最棒的一次面试 · Issue #228 · yihong0618/gitblog 参加了《开源面对面》《ByteTalk》《枫言枫语》播客的一些记录 · Issue #223 · yihong0618/gitblog 有趣与无趣 · Issue #221 · yihong0618/gitblog running_page 开源一周年的总结 · Issue #220 · yihong0618/gitblog 接下来要写的东西 · Issue #219 · yihong0618/gitblog 一点思考 · Issue #218 · yihong0618/gitblog 友情链接 · Issue #217 · yihong0618/gitblog 朋友圈 · Issue #216 · yihong0618/gitblog 张小龙的饭否 · Issue #215 · yihong0618/gitblog 一件小事 · Issue #214 · yihong0618/gitblog 从 Rich 作者的一个问题说起 · Issue #212 · yihong0618/gitblog 如何用一个仓库记录自己的一年 · Issue #209 · yihong0618/gitblog 逃离 · Issue #208 · yihong0618/gitblog 近况 · Issue #207 · yihong0618/gitblog 年过完了 · Issue #206 · yihong0618/gitblog 力扣的程序是如何运行的 · Issue #205 · yihong0618/gitblog 生存守则 · Issue #204 · yihong0618/gitblog 沮丧 · Issue #203 · yihong0618/gitblog 今年的三个小目标 · Issue #202 · yihong0618/gitblog 新年碎碎念 · Issue #201 · yihong0618/gitblog 二零二一开始 · Issue #200 · yihong0618/gitblog 记住这一刻 · Issue #199 · yihong0618/gitblog 巧妙利用 iOS 的快捷指令配合 GitHub Actions 实现自动化 · Issue #198 · yihong0618/gitblog 如何获取悦跑圈数据 · Issue #197 · yihong0618/gitblog 生的对立面 · Issue #196 · yihong0618/gitblog 推荐一些我一直关注的觉得不错的和 Python 相关的独立博客 · Issue #195 · yihong0618/gitblog 2020 年 10 月总结 11 月计划 · Issue #194 · yihong0618/gitblog 写在 0x20 岁之前 · Issue #193 · yihong0618/gitblog 如何获取 keep 跑步数据 · Issue #191 · yihong0618/gitblog 2020 年 9 月总结 10 月计划 · Issue #190 · yihong0618/gitblog 决定把gitblog和kb结合起来 · Issue #189 · yihong0618/gitblog 感冒了,真是难受 · Issue #188 · yihong0618/gitblog 把running page 变得通用必须提上日程了 · Issue #187 · yihong0618/gitblog 我转行程序员的经历 · Issue #186 · yihong0618/gitblog 普通人和天才之间的巨大鸿沟 · Issue #185 · yihong0618/gitblog 月跑 250 km计划 · Issue #184 · yihong0618/gitblog 2020年八月总结九月月展望 · Issue #182 · yihong0618/gitblog
vm.overcommit_memory 学习笔记 · Issue #261 · yihong0618/gitblog
yihong0618 · 2023-04-12 · via RSS feed of yihong0618's gitblog

关于 overcommit_memory 学习的一些记录,其中一些为网上前辈文章的整理记录

什么是 vm.overcommit

但在实际情况下,进程请求的内存可能比实际需要的内存少很多,而且很多进程请求的内存并不会全部使用。在这种情况下,如果系统采用严格的内存申请机制,将会导致系统的内存利用率低下,一些能够使用的内存被浪费。
因此,为了提高系统的内存利用率和灵活性,Linux 提供了 vm.overcommit_memory 机制。当这个参数的值为 0 或 1 时,内核允许进程请求超过系统实际可用内存的内存量。这种行为被称为 "overcommit"。

vm.overcommit 的三个值

#define OVERCOMMIT_GUESS        0 // default
#define OVERCOMMIT_ALWAYS       1  // redis
#define OVERCOMMIT_NEVER        2 // postgres
The Linux kernel supports the following overcommit handling modes

0	-	Heuristic overcommit handling. Obvious overcommits of
		address space are refused. Used for a typical system. It
		ensures a seriously wild allocation fails while allowing
		overcommit to reduce swap usage.  root is allowed to 
		allocate slightly more memory in this mode. This is the 
		default.

1	-	Always overcommit. Appropriate for some scientific
		applications. Classic example is code using sparse arrays
		and just relying on the virtual memory consisting almost
		entirely of zero pages.

2	-	Don't overcommit. The total address space commit
		for the system is not permitted to exceed swap + a
		configurable amount (default is 50%) of physical RAM.
		Depending on the amount you use, in most situations
		this means a process will not be killed while accessing
		pages but will receive errors on memory allocation as
		appropriate.

		Useful for applications that want to guarantee their
		memory allocations will be available in the future
		without having to initialize every page.

https://github.com/torvalds/linux/blob/v4.6/mm/util.c#L481

grep -i commit /proc/meminfo

image

/proc/meminfo 中的 Committed_AS 表示所有进程已经申请的内存总大小,(注意是已经申请的,不是已经分配的),如果 Committed_AS 超过 CommitLimit 就表示发生了 overcommit,超出越多表示 overcommit 越严重 -> 触发 OOM

那么这个 CommitLimit 是怎么算出来的呢?

int __vm_enough_memory(struct mm_struct *mm, long pages, int cap_sys_admin)
{
	long free, allowed, reserve;

	VM_WARN_ONCE(percpu_counter_read(&vm_committed_as) <
			-(s64)vm_committed_as_batch * num_online_cpus(),
			"memory commitment underflow");

	vm_acct_memory(pages);

	/*
	 * Sometimes we want to use more memory than we have
	 */
	if (sysctl_overcommit_memory == OVERCOMMIT_ALWAYS)
		return 0;

	if (sysctl_overcommit_memory == OVERCOMMIT_GUESS) {
		free = global_page_state(NR_FREE_PAGES);
		free += global_page_state(NR_FILE_PAGES);

		/*
		 * shmem pages shouldn't be counted as free in this
		 * case, they can't be purged, only swapped out, and
		 * that won't affect the overall amount of available
		 * memory in the system.
		 */
		free -= global_page_state(NR_SHMEM);

		free += get_nr_swap_pages();

		/*
		 * Any slabs which are created with the
		 * SLAB_RECLAIM_ACCOUNT flag claim to have contents
		 * which are reclaimable, under pressure.  The dentry
		 * cache and most inode caches should fall into this
		 */
		free += global_page_state(NR_SLAB_RECLAIMABLE);

		/*
		 * Leave reserved pages. The pages are not for anonymous pages.
		 */
		if (free <= totalreserve_pages)
			goto error;
		else
			free -= totalreserve_pages;

		/*
		 * Reserve some for root
		 */
		if (!cap_sys_admin)
			free -= sysctl_admin_reserve_kbytes >> (PAGE_SHIFT - 10);

		if (free > pages)
			return 0;

		goto error;
	}

	allowed = vm_commit_limit();
	/*
	 * Reserve some for root
	 */
	if (!cap_sys_admin)
		allowed -= sysctl_admin_reserve_kbytes >> (PAGE_SHIFT - 10);

	/*
	 * Don't let a single process grow so big a user can't recover
	 */
	if (mm) {
		reserve = sysctl_user_reserve_kbytes >> (PAGE_SHIFT - 10);
		allowed -= min_t(long, mm->total_vm / 32, reserve);
	}

	if (percpu_counter_read_positive(&vm_committed_as) < allowed)
		return 0;
error:
	vm_unacct_memory(pages);

	return -ENOMEM;
}

红帽犯过这个错误 :https://access.redhat.com/solutions/665023

image

OOM-killer

OMM killer 机制:linux 会为每个进程算一个 score,当内存不足时候它会根据 score kill
score 越大表示越容易杀死

#!/bin/bash

while read -r pid comm
do
    printf '%d\t%d\t%s\n' "$pid" "$(cat /proc/$pid/oom_score)" "$comm"
done < <(ps -e -o pid= -o comm=) | sort -k2 -n

/proc/${pid}/oom_score_adj, 取值范围为-1000到1000, 如果将该值设置为-1000,则进程永远不会被杀死 --> pg
/proc/${pid}/oom_adj, 取值是-17到+15,取值越高,越容易被干掉。如果是-17,则表示不能被kill
/proc/${pid}/oom_score, 是系统综合进程的内存消耗量、CPU 时间(utime + stime)、存活时间(uptime - start time)和 oom_adj 计算出的,消耗内存越多分越高。

https://elixir.bootlin.com/linux/v5.4.58/source/mm/oom_kill.c

数据库如何设置,如何避免被 oom-killer

  • short answer for redis -> echo 1 > /proc/sys/vm/overcommit_memory :)
  • for PostgresSQL or Greenplum -> echo 2 > /proc/sys/vm/overcommit_memory :)

why

redis:

The Redis background saving schema relies on the copy-on-write semantic of the fork system call in modern operating systems: Redis forks (creates a child process) that is an exact copy of the parent. The child process dumps the DB on disk and finally exits. In theory the child should use as much memory as the parent being a copy, but actually thanks to the copy-on-write semantic implemented by most modern operating systems the parent and child process will share the common memory pages. A page will be duplicated only when it changes in the child or in the parent. Since in theory all the pages may change while the child process is saving, Linux can't tell in advance how much memory the child will take, so if the overcommit_memory setting is set to zero the fork will fail unless there is as much free RAM as required to really duplicate all the parent memory pages. If you have a Redis dataset of 3 GB and just 2 GB of free memory it will fail.
image

Setting overcommit_memory to 1 tells Linux to relax and perform the fork in a more optimistic allocation fashion, and this is indeed what you want for Redis.

PostgresSQL or Greenplum:

Why Use Strategy 2?
The issue with the default overcommits strategy, or using strategy 1 is the unpredictable nature of the failure. In either case we are not guaranteed that the memory is available at time of allocation and that results in unpredictable termination of processes. In particular, the nature of the failure with strategy 1 can result in corruption of either datafiles or transaction logs as a memory failure can occur mid-transaction resulting in the immediate shutdown of the database process without any cleanup.

那如果 redis 和 pg 跑在一台机器上有什么好的建议么?

  • 采用策略 1 保证 redis 在内存不足时候可用,设置 pg master 的 oom_score 让它避免被 oom-kill
  • 采用策略 2 尽量避免造成 redis 会 oom 的情况
  • 分两台机器

当设置 overcommit = 2 时有什么需要注意的

  1. 如前文说的如果服务器有 redis 要保证内存充足
  2. JVM 启动时 Xms Xmx 设置成相同的值,避免向系统申请内存