前些天有位朋友在微信上找到我，说他们的电力系统出现了内存暴涨的问题，让我帮忙看下怎么回事？由于不是训练营里的学员，所以这个分析能力还是有点门外汉，让这位朋友将dump丢给我，然后我们上windbg分析。
要想知道为什么会涨，很简单，用老套路观察内存分布就行了，首先上 !address -summary 即可。
0:000> !address -summary
--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free 206 7ffe`f5a2e000 ( 127.996 TB) 100.00%
<unknown> 252 0`f9541000 ( 3.896 GB) 93.61% 0.00%
Image 1070 0`0c047000 ( 192.277 MB) 4.51% 0.00%
Stack 195 0`03f80000 ( 63.500 MB) 1.49% 0.00%
Heap 50 0`00e58000 ( 14.344 MB) 0.34% 0.00%
Other 13 0`001df000 ( 1.871 MB) 0.04% 0.00%
TEB 65 0`00082000 ( 520.000 kB) 0.01% 0.00%
PEB 1 0`00001000 ( 4.000 kB) 0.00% 0.00%
--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE 522 0`f9fd3000 ( 3.906 GB) 93.85% 0.00%
MEM_IMAGE 1075 0`0c06c000 ( 192.422 MB) 4.52% 0.00%
MEM_MAPPED 49 0`04583000 ( 69.512 MB) 1.63% 0.00%
--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE 206 7ffe`f5a2e000 ( 127.996 TB) 100.00%
MEM_COMMIT 1360 0`f1915000 ( 3.774 GB) 90.69% 0.00%
MEM_RESERVE 286 0`18cad000 ( 396.676 MB) 9.31% 0.00%
接下来使用 !eeheap -gc 观察托管堆，这样就可以一分为二的则出问题。
0:000> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x000001d1803e6ee8
generation 1 starts at 0x000001d18038fc98
generation 2 starts at 0x000001d080001000
ephemeral segment allocation context: none
segment begin allocated size
000001d080000000 000001d080001000 000001d08fffffa0 0xfffefa0(268431264)
000001d098000000 000001d098001000 000001d0a7ffff58 0xfffef58(268431192)
000001d0a8000000 000001d0a8001000 000001d0b7ffff20 0xfffef20(268431136)
000001d0c0000000 000001d0c0001000 000001d0cffffd80 0xfffed80(268430720)
000001d0d8000000 000001d0d8001000 000001d0e7ffff28 0xfffef28(268431144)
000001d0fb7f0000 000001d0fb7f1000 000001d10b7effb8 0xfffefb8(268431288)
000001d10db60000 000001d10db61000 000001d11db5fe20 0xfffee20(268430880)
000001d11db60000 000001d11db61000 000001d12db5fcf0 0xfffecf0(268430576)
000001d12db60000 000001d12db61000 000001d13db5ff30 0xfffef30(268431152)
000001d13db60000 000001d13db61000 000001d14db5ffd8 0xfffefd8(268431320)
000001d14db60000 000001d14db61000 000001d15db5ffe8 0xfffefe8(268431336)
000001d15db60000 000001d15db61000 000001d16db5d040 0xfffc040(268419136)
000001d16db60000 000001d16db61000 000001d17db5ada0 0xfff9da0(268410272)
000001d17db60000 000001d17db61000 000001d1808f97b0 0x2d987b0(47810480)
Large object heap starts at 0x000001d090001000
segment begin allocated size
000001d090000000 000001d090001000 000001d0952f8028 0x52f7028(86994984)
000001d0d0000000 000001d0d0001000 000001d0d6958340 0x6957340(110457664)
Total Size: Size: 0xde9d0d70 (3734834544) bytes.
------------------------------
GC Heap Size: Size: 0xde9d0d70 (3734834544) bytes.
从卦中可以轻松的看出，obviously，当前是托管堆内存暴涨，接下来观察托管堆。
0:000> !dumpheap -stat
Statistics:
MT Count TotalSize Class Name
00007ffa6e7a1ec0 1 24 System.Collections.Generic.GenericEqualityComparer`1[[System.Int64, mscorlib]]
00007ffa6dc5e1e0 1 24 System.Collections.Generic.GenericEqualityComparer`1[[System.String, mscorlib]]
00007ffa6dc5e110 1 24 System.Collections.Generic.GenericEqualityComparer`1[[System.Int32, mscorlib]]
00007ffa6dc58a98 1 24 System.CLSCompliantAttribute
...
00007ffa1132a1e8 482984 88869056 xxx.C_Sys_UserAppTokenModel
00007ffa112f2488 602175 178243800 xxx.C_Sys_UserModel
00007ffa6dc41ac8 229351 238910824 System.String[]
00007ffa6dc59808 30618357 1297263358 System.String
000001d0ebefbff0 4003171 1418932312 Free
Total 41839190 objects
Fragmented blocks larger than 0.5 MB:
Addr Size Followed by
000001d18060cca8 1.0MB 000001d180702c58 System.Byte[]
000001d180702ce8 0.5MB 000001d18078a340 System.Net.SocketAddress
000001d18078a3c0 1.4MB 000001d1808f9568 System.Byte[]
从卦中 bottom-up 来看，当前托管堆中貌似是 Free 导致的内存碎片化，接下来就是往内存碎片化这边 investigation。
由于 free 的个数高达 400w +，直接观察free的前后是什么，这个有一点的实操难度，at this point 我们发现C_Sys_UserAppTokenModel 和 C_Sys_UserModel 也高达48w和60w，虽不是很高，但也给我们留了是不是实体失控的模糊影子？
接下来祭出perfview观察这两哥们是不是来自于某个集合变量，截图如下：
从卦中可以清晰的看到，托管堆上有三个 static 变量，分别吃了1G，712M 和 282M，尤其是1G的 s_DeviceManager 集合，经过展开 perfview 是一个 ConcurrentDictionary<Int32, List<String>> 集合，能吃1G的ConcurrentDictionary，要么当缓存使用，要么就是失控。
有了这些概念之后，我们再次回望托管堆上的free 的前后数据，输出如下：
0:000> !eeheap -gc
Number of GC Heaps: 1
generation 0 starts at 0x000001d1803e6ee8
generation 1 starts at 0x000001d18038fc98
generation 2 starts at 0x000001d080001000
ephemeral segment allocation context: none
segment begin allocated size
000001d080000000 000001d080001000 000001d08fffffa0 0xfffefa0(268431264)
000001d098000000 000001d098001000 000001d0a7ffff58 0xfffef58(268431192)
000001d0a8000000 000001d0a8001000 000001d0b7ffff20 0xfffef20(268431136)
000001d0c0000000 000001d0c0001000 000001d0cffffd80 0xfffed80(268430720)
000001d0d8000000 000001d0d8001000 000001d0e7ffff28 0xfffef28(268431144)
000001d0fb7f0000 000001d0fb7f1000 000001d10b7effb8 0xfffefb8(268431288)
000001d10db60000 000001d10db61000 000001d11db5fe20 0xfffee20(268430880)
000001d11db60000 000001d11db61000 000001d12db5fcf0 0xfffecf0(268430576)
000001d12db60000 000001d12db61000 000001d13db5ff30 0xfffef30(268431152)
000001d13db60000 000001d13db61000 000001d14db5ffd8 0xfffefd8(268431320)
000001d14db60000 000001d14db61000 000001d15db5ffe8 0xfffefe8(268431336)
000001d15db60000 000001d15db61000 000001d16db5d040 0xfffc040(268419136)
000001d16db60000 000001d16db61000 000001d17db5ada0 0xfff9da0(268410272)
000001d17db60000 000001d17db61000 000001d1808f97b0 0x2d987b0(47810480)
Large object heap starts at 0x000001d090001000
segment begin allocated size
000001d090000000 000001d090001000 000001d0952f8028 0x52f7028(86994984)
000001d0d0000000 000001d0d0001000 000001d0d6958340 0x6957340(110457664)
Total Size: Size: 0xde9d0d70 (3734834544) bytes.
------------------------------
GC Heap Size: Size: 0xde9d0d70 (3734834544) bytes.
0:000> !dumpheap 000001d10db61000 000001d11db5fe20
...
000001d10db62408 00007ffa11436458 272
000001d10db62518 00007ffa114365d8 48
000001d10db62548 000001d0ebefbff0 422 Free
000001d10db626f0 00007ffa6d54ac78 40
000001d10db62718 00007ffa6dc59808 34
000001d10db62740 00007ffa6dc59808 34
000001d10db62768 000001d0ebefbff0 110 Free
000001d10db627d8 00007ffa6d54ac78 40
000001d10db62800 00007ffa6dc59808 34
000001d10db62828 00007ffa6dc59808 34
...
0:000> !gcroot 000001d10db62740
Thread 2aa0:
000000a1905be300 00007ffa1112fead xxx.C_DeviceManager.<xxxDicTODB>b__30_0()
rbp+10: 000000a1905be3b0
-> 000001d0805121a8 xxx.C_DeviceManager
-> 000001d080512200 System.Collections.Concurrent.ConcurrentDictionary`2[[System.Int32, mscorlib],[System.Collections.Generic.List`1[[System.String, mscorlib]], mscorlib]]
-> 000001d142690a08 System.Collections.Concurrent.ConcurrentDictionary`2+Tables[[System.Int32, mscorlib],[System.Collections.Generic.List`1[[System.String, mscorlib]], mscorlib]]
-> 000001d0912a9958 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Int32, mscorlib],[System.Collections.Generic.List`1[[System.String, mscorlib]], mscorlib]][]
-> 000001d14250af68 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Int32, mscorlib],[System.Collections.Generic.List`1[[System.String, mscorlib]], mscorlib]]
-> 000001d10db626f0 System.Collections.Generic.List`1[[System.String, mscorlib]]
-> 000001d116e68958 System.String[]
-> 000001d10db62740 System.String
最后我们到 <xxxDicTODB>b__30_0 中找相关的代码逻辑，由于隐私关系这里就不显示了，让朋友去看下 <xxxDicTODB>b__30_0 中为什么把 ConcurrentDictionary 搞这么大，最后朋友发现是 TODB 过程中的数据堆积所致，而这个堆积又引发了托管堆的碎片化，有点意思。
这次生产事故是因为业务逻辑问题导致内存static堆积，而堆积又引发了堆碎片化，所以这个dump在分析时有一定的误导性，很多人往往会被 free 给先入为主了。。。
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。