When out of memory occurs, the out_of_memory function chooses a process that the kernel considers guilty of allocating too much memory and kills the process.
There is a high probability of freeing up more free pages and then jumping back to retry the memory allocation operation.
Here we do not discuss out_of_memory function process selection at the expense of process strategy.
We will only discuss the meaning of the output information from the kernel when out of memory occurs.
1. OOM information
The following is a typical out of memory kernel output message:
- <4>[12345.342532] systemd-journal invoked oom-killer: gfp_mask=0x800d0, order=0, oom_score_adj=0
- <4>[12345.351216] CPU: 1 PID: 1371 Comm: systemd-journal Tainted: G O 3.14.31-00017-g40fab71 #1
- <4>[12345.360695] Backtrace:
- <4>[12345.363263] [<c0012fcc>] (dump_backtrace) from [<c00131a4>] (show_stack+0x20/0x24)
- <4>[12345.371192] r6:00000000 r5:ffffffff r4:00000000 r3:bd943631
- <4>[12345.377136] [<c0013184>] (show_stack) from [<c07bbe78>] (dump_stack+0x7c/0xc8)
- <4>[12345.384710] [<c07bbdfc>] (dump_stack) from [<c07ba7e4>] (dump_header.isra.14+0x74/0x188)
- <4>[12345.393184] r6:000800d0 r5:00000000 r4:e8088000 r3:00000002
- <4>[12345.399126] [<c07ba770>] (dump_header.isra.14) from [<c00f8a28>] (oom_kill_process+0x230/0x3e0)
- <4>[12345.408234] r10:00000000 r8:000800d0 r7:00000000 r6:c0b89aa8 r5:000800d0 r4:e9bb79c0
- <4>[12345.416462] [<c00f87f8>] (oom_kill_process) from [<c00f90c8>] (out_of_memory+0x2f4/0x354)
- <4>[12345.425024] r10:00000000 r9:00000000 r8:000800d0 r7:00000000 r6:c0b89aa8 r5:c0b89d08
- <4>[12345.433249] r4:c0b89aa8
- <4>[12345.435903] [<c00f8dd4>] (out_of_memory) from [<c00fd6c8>] (__alloc_pages_nodemask+0x93c/0x988)
- <4>[12345.445011] r10:00000000 r9:c0c38fc0 r8:c0b871d8 r7:e8088000 r6:c0c39bc0 r5:00000000
- <4>[12345.453234] r4:000800d0
- <4>[12345.455887] [<c00fcd8c>] (__alloc_pages_nodemask) from [<c00fd734>] (__get_free_pages+0x20/0x3c)
- <4>[12345.465087] r10:e97d36a8 r9:00000063 r8:e8089f6c r7:00000063 r6:b6f79f68 r5:e97d36a8
- <4>[12345.473311] r4:00000000
- <4>[12345.475965] [<c00fd714>] (__get_free_pages) from [<c0196878>] (proc_pid_readlink+0x68/0x110)
- <4>[12345.484808] [<c0196810>] (proc_pid_readlink) from [<c013dcb8>] (SyS_readlinkat+0xf0/0x104)
- <4>[12345.493461] r7:bea40520 r6:ffffff9c r5:00004000 r4:00000000
- <4>[12345.499402] [<c013dbc8>] (SyS_readlinkat) from [<c000eee0>] (ret_fast_syscall+0x0/0x34)
- <4>[12345.507785] r10:00000000 r9:e8088000 r8:c000f148 r7:0000014c r6:00000063 r5:b6f79f68
- <4>[12345.516011] r4:00000064
- <4>[12345.518663] Mem-info:
- <4>[12345.521049] Normal per-cpu:
- <4>[12345.523969] CPU 0: hi: 42, btch: 7 usd: 23
- <4>[12345.528979] CPU 1: hi: 42, btch: 7 usd: 25
- <4>[12345.534004] HighMem per-cpu:
- <4>[12345.537013] CPU 0: hi: 186, btch: 31 usd: 27
- <4>[12345.542199] CPU 1: hi: 186, btch: 31 usd: 29
- <4>[12345.547247] active_anon:21860 inactive_anon:14790 isolated_anon:0
- <4>[12345.547247] active_file:41585 inactive_file:10422 isolated_file:0
- <4>[12345.547247] unevictable:0 dirty:9 writeback:205 unstable:0
- <4>[12345.547247] free:285748 slab_reclaimable:2100 slab_unreclaimable:26286
- <4>[12345.547247] mapped:26079 shmem:14857 pagetables:687 bounce:0
- <4>[12345.547247] free_cma:57779
- <4>[12345.581839] Normal free:233460kB min:2488kB low:3108kB high:3732kB active_anon:17312kB
- inactive_anon:10824kB active_file:128kB inactive_file:4kB unevictable:0kB isolated(anon):0kB
- isolated(file):0kB present:774144kB managed:387568kB mlocked:0kB dirty:16kB writeback:76kB
- mapped:3296kB shmem:10840kB slab_reclaimable:8400kB slab_unreclaimable:105144kB kernel_stack:1168kB
- pagetables:2748kB unstable:0kB bounce:0kB free_cma:231116kB writeback_tmp:0kB pages_scanned:1648
- all_unreclaimable? yes
- <4>[12345.627014] lowmem_reserve[]: 0 10168 10168
- <4>[12345.631565] HighMem free:909036kB min:512kB low:2604kB high:4696kB active_anon:70632kB
- inactive_anon:48336kB active_file:166212kB inactive_file:41684kB unevictable:0kB isolated(anon):0kB
- isolated(file):0kB present:1301504kB managed:1301504kB mlocked:0kB dirty:20kB writeback:744kB
- mapped:101020kB shmem:48588kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB
- pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0
- all_unreclaimable? no
- <4>[12345.675614] lowmem_reserve[]: 0 0 0
- <4>[12345.679437] Normal: 1165*4kB (MRC) 1122*8kB (RC) 1119*16kB (RC) 1118*32kB (C) 1068*64kB (RC)
- 748*128kB (C) 0*256kB 0*512kB 0*1024kB 1*2048kB (R) 0*4096kB 0*8192kB = 233460kB
- <4>[12345.695797] HighMem: 99*4kB (M) 1148*8kB (UM) 1314*16kB (UM) 880*32kB (UM) 327*64kB (M)
- 87*128kB (M) 34*256kB (M) 38*512kB (M) 12*1024kB (M) 10*2048kB (M) 3*4096kB (M) 91*8192kB (UMR) = 909516kB
- <4>[12345.714293] 66770 total pagecache pages
- <4>[12345.718309] 0 pages in swap cache
- <4>[12345.724832] Swap cache stats: add 0, delete 0, find 0/0
- <4>[12345.730308] Free swap = 0kB
- <4>[12345.733412] Total swap = 0kB
- <4>[12345.747245] 520192 pages of RAM
- <4>[12345.750577] 286253 free pages
- <4>[12345.753778] 97924 reserved pages
- <4>[12345.757258] 28061 slab pages
- <4>[12345.760574] 115601 pages shared
- <4>[12345.764283] 0 pages swap cached
- <6>[12345.767572] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
- <6>[12345.775906] [ 1366] 0 1366 459 125 3 0 0 sh
- <6>[12345.785861] [ 1367] 0 1367 665 235 4 0 0 propertyd
- <6>[12345.794802] [ 1368] 0 1368 26553 8835 58 0 0 seed
- <6>[12345.803296] [ 1371] 0 1371 1648 772 5 0 0 systemd-journal
- <6>[12345.812792] [ 1375] 0 1375 750 300 4 0 -1000 systemd-udevd
- <6>[12345.822449] [ 2416] 1040 2416 3852 510 7 0 0 secd
- <6>[12345.831341] [ 2419] 0 2419 6678 923 9 0 0 storagemanagerd
- <6>[12345.840944] [ 2420] 0 2420 1267 497 5 0 0 connmand
- <6>[12345.849566] [ 2422] 0 2422 4484 687 8 0 0 uuid
- <6>[12345.857843] [ 2424] 0 2424 1161 358 5 0 0 connman-vpnd
- <6>[12345.867271] [ 2427] 1000 2427 1593 461 6 0 0 logboxd
- <6>[12345.875846] [ 2432] 0 2432 9483 1718 15 0 0 cmns
- <6>[12345.884104] [ 2451] 81 2451 1355 474 4 0 -900 dbus-daemon
- <6>[12345.893018] [ 2532] 0 2532 11794 246 10 0 0 adbd
- <6>[12345.901304] [ 2535] 0 2535 1502 347 5 0 0 wpa_supplicant
- <6>[12345.910473] [ 2536] 0 2536 12820 866 12 0 0 udisksd
- <6>[12345.919119] [ 2537] 0 2537 1898 527 6 0 0 tyid
- <6>[12345.927361] [ 2540] 0 2540 10076 2157 16 0 0 datamanagerd
- <6>[12345.936349] [ 2554] 0 2554 5983 574 7 0 0 connectivityser
- <6>[12345.945635] [ 2558] 0 2558 10604 5388 21 0 0 weston
- <6>[12345.964101] [ 2589] 0 2589 14597 1917 17 0 0 pagemanagerd
- <6>[12345.973272] [ 2590] 0 2590 3832 515 7 0 0 amt
- <6>[12345.981730] [ 2593] 0 2593 6176 1343 12 0 0 weston-desktop-
- <6>[12345.991046] [ 2599] 0 2599 7185 761 12 0 0 scim-launcher
- <6>[12346.098925] [ 5580] 0 5580 458 116 3 0 0 sh
- <6>[12346.107065] [ 5581] 0 5581 492 175 3 0 0 gzip
- <3>[12346.115335] Out of memory: Kill process 5575 thread_x score 481 or sacrifice child
- <3>[12346.124212] Killed process 5575 thread_x total-vm:106212kB, anon-rss:18036kB, file-rss:2704kB
2 OOM Information Analysis
2.1
Systemd-journal: The current process is systemd-journal, which triggers oom-killer when requesting page assignment
- <4>[12345.342532] systemd-journal invoked oom-killer: gfp_mask=0x800d0, order=0, oom_score_adj=0
gfp_mask=0x800d0: is the GFP flag of alloc_page. For the current scene, it represents u GFP_RECLAIMABLE | u GFP_HIGH | u GFP_IO | u GFP_FS.
order=0: This means that the order of alloc_page is 0, which means that only 1 ^ 0 = 1 page is requested.
oom_score_adj=0: Indicates the probability that the process will be killed. The value of oom_score_adj is 0(never kill)~1000(always kill)
- <4>[12345.351216] CPU: 1 PID: 1371 Comm: systemd-journal Tainted: G O 3.14.31-00017-g40fab71 #1
- <4>[12345.360695] Backtrace:
- <4>[12345.363263] [<c0012fcc>] (dump_backtrace) from [<c00131a4>] (show_stack+0x20/0x24)
- <4>[12345.371192] r6:00000000 r5:ffffffff r4:00000000 r3:bd943631
- <4>[12345.377136] [<c0013184>] (show_stack) from [<c07bbe78>] (dump_stack+0x7c/0xc8)
- <4>[12345.384710] [<c07bbdfc>] (dump_stack) from [<c07ba7e4>] (dump_header.isra.14+0x74/0x188)
- <4>[12345.393184] r6:000800d0 r5:00000000 r4:e8088000 r3:00000002
- <4>[12345.399126] [<c07ba770>] (dump_header.isra.14) from [<c00f8a28>] (oom_kill_process+0x230/0x3e0)
- <4>[12345.408234] r10:00000000 r8:000800d0 r7:00000000 r6:c0b89aa8 r5:000800d0 r4:e9bb79c0
- <4>[12345.416462] [<c00f87f8>] (oom_kill_process) from [<c00f90c8>] (out_of_memory+0x2f4/0x354)
- <4>[12345.425024] r10:00000000 r9:00000000 r8:000800d0 r7:00000000 r6:c0b89aa8 r5:c0b89d08
- <4>[12345.433249] r4:c0b89aa8
- <4>[12345.435903] [<c00f8dd4>] (out_of_memory) from [<c00fd6c8>] (__alloc_pages_nodemask+0x93c/0x988)
- <4>[12345.445011] r10:00000000 r9:c0c38fc0 r8:c0b871d8 r7:e8088000 r6:c0c39bc0 r5:00000000
- <4>[12345.453234] r4:000800d0
- <4>[12345.455887] [<c00fcd8c>] (__alloc_pages_nodemask) from [<c00fd734>] (__get_free_pages+0x20/0x3c)
- <4>[12345.465087] r10:e97d36a8 r9:00000063 r8:e8089f6c r7:00000063 r6:b6f79f68 r5:e97d36a8
- <4>[12345.473311] r4:00000000
- <4>[12345.475965] [<c00fd714>] (__get_free_pages) from [<c0196878>] (proc_pid_readlink+0x68/0x110)
- <4>[12345.484808] [<c0196810>] (proc_pid_readlink) from [<c013dcb8>] (SyS_readlinkat+0xf0/0x104)
- <4>[12345.493461] r7:bea40520 r6:ffffff9c r5:00004000 r4:00000000
- <4>[12345.499402] [<c013dbc8>] (SyS_readlinkat) from [<c000eee0>] (ret_fast_syscall+0x0/0x34)
- <4>[12345.507785] r10:00000000 r9:e8088000 r8:c000f148 r7:0000014c r6:00000063 r5:b6f79f68
- <4>[12345.516011] r4:00000064
The output of dump_header - > dump_stack triggers the call function stack of OOM, starting with ret_fast_syscall.
From this output, it can be inferred that a paging operation triggered by system D-JOURNAL calling readlink system call resulted in OOM.
2.2
Dump_header - > show_mem outputs the current system memory information.
- <4>[12345.518663] Mem-info:
- <4>[12345.521049] Normal per-cpu:
- <4>[12345.523969] CPU 0: hi: 42, btch: 7 usd: 23
- <4>[12345.528979] CPU 1: hi: 42, btch: 7 usd: 25
- <4>[12345.534004] HighMem per-cpu:
- <4>[12345.537013] CPU 0: hi: 186, btch: 31 usd: 27
- <4>[12345.542199] CPU 1: hi: 186, btch: 31 usd: 29
- <4>[12345.547247] active_anon:21860 inactive_anon:14790 isolated_anon:0
- <4>[12345.547247] active_file:41585 inactive_file:10422 isolated_file:0
- <4>[12345.547247] unevictable:0 dirty:9 writeback:205 unstable:0
- <4>[12345.547247] free:285748 slab_reclaimable:2100 slab_unreclaimable:26286
- <4>[12345.547247] mapped:26079 shmem:14857 pagetables:687 bounce:0
- <4>[12345.547247] free_cma:57779
- <4>[12345.581839] Normal free:233460kB min:2488kB low:3108kB high:3732kB active_anon:17312kB
- inactive_anon:10824kB active_file:128kB inactive_file:4kB unevictable:0kB isolated(anon):0kB
- isolated(file):0kB present:774144kB managed:387568kB mlocked:0kB dirty:16kB writeback:76kB
- mapped:3296kB shmem:10840kB slab_reclaimable:8400kB slab_unreclaimable:105144kB kernel_stack:1168kB
- pagetables:2748kB unstable:0kB bounce:0kB free_cma:231116kB writeback_tmp:0kB pages_scanned:1648
- all_unreclaimable? yes
- <4>[12345.627014] lowmem_reserve[]: 0 10168 10168
- <4>[12345.631565] HighMem free:909036kB min:512kB low:2604kB high:4696kB active_anon:70632kB
- inactive_anon:48336kB active_file:166212kB inactive_file:41684kB unevictable:0kB isolated(anon):0kB
- isolated(file):0kB present:1301504kB managed:1301504kB mlocked:0kB dirty:20kB writeback:744kB
- mapped:101020kB shmem:48588kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB
- pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
- <4>[12345.675614] lowmem_reserve[]: 0 0 0
- <4>[12345.679437] Normal: 1165*4kB (MRC) 1122*8kB (RC) 1119*16kB (RC) 1118*32kB (C) 1068*64kB (RC)
- 748*128kB (C) 0*256kB 0*512kB 0*1024kB 1*2048kB (R) 0*4096kB 0*8192kB = 233460kB
- <4>[12345.695797] HighMem: 99*4kB (M) 1148*8kB (UM) 1314*16kB (UM) 880*32kB (UM) 327*64kB (M)
- 87*128kB (M) 34*256kB (M) 38*512kB (M) 12*1024kB (M) 10*2048kB (M) 3*4096kB (M) 91*8192kB (UMR) = 909516kB
- <4>[12345.714293] 66770 total pagecache pages
- <4>[12345.718309] 0 pages in swap cache
- <4>[12345.724832] Swap cache stats: add 0, delete 0, find 0/0
- <4>[12345.730308] Free swap = 0kB
- <4>[12345.733412] Total swap = 0kB
- <4>[12345.747245] 520192 pages of RAM
- <4>[12345.750577] 286253 free pages
- <4>[12345.753778] 97924 reserved pages
- <4>[12345.757258] 28061 slab pages
- <4>[12345.760574] 115601 pages shared
- <4>[12345.764283] 0 pages swap cached
2.2.1
<4>[12345.521049] Normal per-cpu:
<4>[12345.523969] CPU 0: hi: 42, btch: 7 usd: 23
<4>[12345.528979] CPU 1: hi: 42, btch: 7 usd: 25
<4>[12345.534004] HighMem per-cpu:
<4>[12345.537013] CPU 0: hi: 186, btch: 31 usd: 27
<4>[12345.542199] CPU 1: hi: 186, btch: 31 usd: 29
Each memory management area defines a "per CPU" page frame cache. All "per CPU" caches contain pre-allocated page frames that are used to satisfy a single page memory request from a local CPU.
CPU 0: hi: 42, btch: 7 usd: 23
Represents CPU 0,
hi: 42 represents the upper limit. If this number is exceeded, the batch page frames are released into the buddy system.
btch: 7 represents the size of a page block when adding or deleting a page frame to a cache
Usd: Number of Page Boxes in 23 Page Box Cache
2.2.2
- <4>[12345.547247] active_anon:21860 inactive_anon:14790 isolated_anon:0
- <4>[12345.547247] active_file:41585 inactive_file:10422 isolated_file:0
- <4>[12345.547247] unevictable:0 dirty:9 writeback:205 unstable:0
- <4>[12345.547247] free:285748 slab_reclaimable:2100 slab_unreclaimable:26286
- <4>[12345.547247] mapped:26079 shmem:14857 pagetables:687 bounce:0
- <4>[12345.547247] free_cma:57779
active_anon: Anonymous mapping of activities, "activities" refers to recent visits, and "anonymity" refers to page mappings that are not related to any data source.
inactive_anon: Inactive Anonymous Mapping
isolated_anon: DON'T KNOW
active_file: Active file mapping, page mapping and disk file association
inactive_file: Inactive file mapping
isolated_file: DON'T KNOW
unevictable:
Dirty: dirty pages that represent the content of pages and the original content on fast devices.
Writteback: The current page is in a write-back state
unstable:
free: free pages
Slab_relaimable: Recyclable pages in slab cache
Slab_unreclaimable: Pages that cannot be recycled in slab cache
mapped: BH_MAPPED, which means that this page is used as a buffer map for fast devices. Note that this map is different from anon and file mappings.
shmem: Pages for shared memory mapping
pagetable: Pages occupied by page tables, that is, the number of pages occupied by PTE PTD
bounce:
Free_cma: The free page of the continuous memory allocator.
2.2.3
- <4>[12345.581839] Normal free:233460kB min:2488kB low:3108kB high:3732kB active_anon:17312kB inactive_anon:10824kB
- active_file:128kB inactive_file:4kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:774144kB
- managed:387568kB mlocked:0kB dirty:16kB writeback:76kB mapped:3296kB shmem:10840kB slab_reclaimable:8400kB
- slab_unreclaimable:105144kB kernel_stack:1168kB pagetables:2748kB unstable:0kB bounce:0kB free_cma:231116kB
- writeback_tmp:0kB pages_scanned:1648 all_unreclaimable? yes
- <4>[12345.627014] lowmem_reserve[]: 0 10168 10168
- <4>[12345.631565] HighMem free:909036kB min:512kB low:2604kB high:4696kB active_anon:70632kB inactive_anon:48336kB
- active_file:166212kB inactive_file:41684kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1301504kB
- managed:1301504kB mlocked:0kB dirty:20kB writeback:744kB mapped:101020kB shmem:48588kB slab_reclaimable:0kB
- slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB
- pages_scanned:0 all_unreclaimable? no
- <4>[12345.675614] lowmem_reserve[]: 0 0 0
Normal free: Free space in Normal zone
min, low, high are several watermarks for normal zone to perform page replacement
lowmem_reserve: Represents the number of allocatable pages reserved for other zones by this zone
present: Represents the physical memory size of a zone
Managed: The size of present ation memory managed by buddy system, managed = preset - reserved
The other values can be referred to in Section 2.2.2. They have similar meanings except that the values represent Normal zone s.
Note 1. Several items are unique to Normal, such as kernel_stack, pagetables, free_cma, slab_reclaimable, slab_unreclaimable, because the pages of normal zone are directly mapped and are used by the kernel.
For highmem, it is mainly used for anonymous mapping, file mapping, mapped, and shared memory.
2.2.4
buddy system information, order range 0-11
- <4>[12345.679437] Normal: 1165*4kB (MRC) 1122*8kB (RC) 1119*16kB (RC) 1118*32kB (C) 1068*64kB (RC)
- 748*128kB (C) 0*256kB 0*512kB 0*1024kB 1*2048kB (R) 0*4096kB 0*8192kB = 233460kB
- <4>[12345.695797] HighMem: 99*4kB (M) 1148*8kB (UM) 1314*16kB (UM) 880*32kB (UM) 327*64kB (M)
- 87*128kB (M) 34*256kB (M) 38*512kB (M) 12*1024kB (M) 10*2048kB (M) 3*4096kB (M) 91*8192kB (UMR) = 909516kB
M denotes moveable
R denotes Reserve
C means CMA.
U stands for unmovable
E denotes reclaimable
1. Only (C) means that this freelist can only be allocated to allocations with ALLOC_CMA flags.
2. Highmem has no C flag because continuous memory allocation occurs only in Normal zone
3. MRC, which means that this freelist has CMA memory, Reserve memory and Movable memory
3 Who triggered OOM
Several factors influence the occurrence of OOM
1. The size of the order allocated and the way the system treats the order
2. Which zone does the allocation occur?
3. Zone's Watermark Size
4. Degree of memory fragmentation
5. It is said that the constant allocation of address space will also lead to the occurrence of OOM (not yet verified).
For the OOM information above, we can see that there is a lot of free space in the system: 233460KB, but OOM still happens.
The first order allocated is 0, so fragmentation is irrelevant. gfp_mask=0x800d0 indicates that the allocation occurs in the Normal partition and the type is Reclaimable. Reclaimable also means that memory cannot be allocated from CMA.
Since it is not the order that is too large that causes the allocation failure, it is the free space less than the memory watermarking min that causes OOM killer.
The allocation type is Reclaimable, which causes free space to be subtracted from CMA free space by 233460KB - 223116kB = 2352KB, less than min watermarking. The system starts OOM killer.
- static bool __zone_watermark_ok(struct zone *z, unsigned int order,
- unsigned long mark, int classzone_idx, int alloc_flags,
- long free_pages)
- {
- ...
- #ifdef CONFIG_CMA
- /* If allocation can't use CMA areas don't use free CMA pages */
- if (!(alloc_flags & ALLOC_CMA))
- free_cma = zone_page_state(z, NR_FREE_CMA_PAGES);
- #endif
- if (free_pages - free_cma <= min + z->lowmem_reserve[classzone_idx])
- return false;
- ...
- }