calico被oom分析
作者：互联网
Oct 19 12:00:14 g1-med-online1-206 systemd: Removed slice User Slice of sa.
Oct 19 12:00:21 g1-med-online1-206 kubelet: E1019 12:00:21.906083 981242 pod_workers.go:191] Error syncing pod 5ff33e66-b4d8-4934-b036-6a757f72e9e5 ("budget-sys-go-serve-lsvxb-776bf76b7-rs5kv_bi(5ff33e66-b4d8-4934-b036-6a757f72e9e5)"),
skipping: failed to "StartContainer" for "go-serve" with CrashLoopBackOff: "back-off 5m0s restarting failed container=go-serve pod=budget-sys-go-serve-lsvxb-776bf76b7-rs5kv_bi(5ff33e66-b4d8-4934-b036-6a757f72e9e5)"
Oct 19 12:00:24 g1-med-online1-206 kubelet: I1019 12:00:24.164582 981242 setters.go:73] Using node IP: "10.16.25.206"
Oct 19 12:00:32 g1-med-online1-206 kubelet: E1019 12:00:32.907254 981242 pod_workers.go:191] Error syncing pod 5ff33e66-b4d8-4934-b036-6a757f72e9e5 ("budget-sys-go-serve-lsvxb-776bf76b7-rs5kv_bi(5ff33e66-b4d8-4934-b036-6a757f72e9e5)"),
skipping: failed to "StartContainer" for "go-serve" with CrashLoopBackOff: "back-off 5m0s restarting failed container=go-serve pod=budget-sys-go-serve-lsvxb-776bf76b7-rs5kv_bi(5ff33e66-b4d8-4934-b036-6a757f72e9e5)"
Oct 19 12:00:34 g1-med-online1-206 kubelet: I1019 12:00:34.211832 981242 setters.go:73] Using node IP: "10.16.25.206"
Oct 19 12:00:43 g1-med-online1-206 kernel: iptables-legacy invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=-998
Oct 19 12:00:43 g1-med-online1-206 kernel: iptables-legacy cpuset=1292d8682e805865484d004a38aecab6813f3b8c323f62e8b4ead030a0b8bf07 mems_allowed=0-1
Oct 19 12:00:43 g1-med-online1-206 kernel: CPU: 27 PID: 2580053 Comm: iptables-legacy Kdump: loaded Not tainted 4.19.2-1.el7.elrepo.x86_64 #1
Oct 19 12:00:43 g1-med-online1-206 kernel: Hardware name: Lenovo ThinkSystem SR590 -[7X99CTO1WW]-/-[7X99CTO1WW]-, BIOS -[TEE160N-2.71]- 09/18/2020
Oct 19 12:00:43 g1-med-online1-206 kernel: Call Trace:
Oct 19 12:00:43 g1-med-online1-206 kernel: dump_stack+0x63/0x88
Oct 19 12:00:43 g1-med-online1-206 kernel: dump_header+0x78/0x2a4
Oct 19 12:00:43 g1-med-online1-206 kernel: ? mem_cgroup_scan_tasks+0x9c/0xf0
Oct 19 12:00:43 g1-med-online1-206 kernel: oom_kill_process+0x262/0x290
Oct 19 12:00:43 g1-med-online1-206 kernel: out_of_memory+0x140/0x4b0
Oct 19 12:00:43 g1-med-online1-206 kernel: mem_cgroup_out_of_memory+0x4b/0x80
Oct 19 12:00:43 g1-med-online1-206 kernel: try_charge+0x667/0x6f0
Oct 19 12:00:43 g1-med-online1-206 kernel: mem_cgroup_try_charge+0x8c/0x1e0
Oct 19 12:00:43 g1-med-online1-206 kernel: __add_to_page_cache_locked+0x94/0x1f0
Oct 19 12:00:43 g1-med-online1-206 kernel: add_to_page_cache_lru+0x52/0x100
Oct 19 12:00:43 g1-med-online1-206 kernel: filemap_fault+0x41b/0x6f0
Oct 19 12:00:43 g1-med-online1-206 kernel: ? radix_tree_next_chunk+0x10c/0x2e0
Oct 19 12:00:43 g1-med-online1-206 kernel: ? filemap_map_pages+0x2b2/0x410
Oct 19 12:00:43 g1-med-online1-206 kernel: ext4_filemap_fault+0x31/0x50 [ext4]
Oct 19 12:00:43 g1-med-online1-206 kernel: __do_fault+0x24/0x80
Oct 19 12:00:43 g1-med-online1-206 kernel: __handle_mm_fault+0xc0f/0xef0
Oct 19 12:00:43 g1-med-online1-206 kernel: ? hugepage_vma_check+0xb0/0xd0
Oct 19 12:00:43 g1-med-online1-206 kernel: handle_mm_fault+0x102/0x220
Oct 19 12:00:43 g1-med-online1-206 kernel: __do_page_fault+0x212/0x4e0
Oct 19 12:00:43 g1-med-online1-206 kernel: do_page_fault+0x37/0x140
Oct 19 12:00:43 g1-med-online1-206 kernel: ? page_fault+0x8/0x30
Oct 19 12:00:43 g1-med-online1-206 kernel: page_fault+0x1e/0x30
Oct 19 12:00:43 g1-med-online1-206 kernel: RIP: 0033:0x7f715669a0db
Oct 19 12:00:43 g1-med-online1-206 kernel: Code: Bad RIP value.
Oct 19 12:00:43 g1-med-online1-206 kernel: RSP: 002b:00007ffc25020e30 EFLAGS: 00010207
Oct 19 12:00:43 g1-med-online1-206 kernel: RAX: 0000558163adffd0 RBX: 00007f71556b59c8 RCX: 00007f715578edf0
Oct 19 12:00:43 g1-med-online1-206 kernel: RDX: 0000000000000000 RSI: 00007f71556b5a68 RDI: 0000558163adfffa
Oct 19 12:00:43 g1-med-online1-206 kernel: RBP: 0000000001d3c990 R08: 0000000000000006 R09: 0000000000000003
Oct 19 12:00:43 g1-med-online1-206 kernel: R10: 0000558161d15010 R11: 00007f715586f070 R12: 0000000000022528
Oct 19 12:00:43 g1-med-online1-206 kernel: R13: 0000558163adfef0 R14: 0000558161d15260 R15: 00007ffc25020e58
Oct 19 12:00:43 g1-med-online1-206 kernel: memory: usage 250000kB, limit 250000kB, failcnt 0
Oct 19 12:00:43 g1-med-online1-206 kernel: memory+swap: usage 250000kB, limit 250000kB, failcnt 201174
Oct 19 12:00:43 g1-med-online1-206 kernel: kmem: usage 44272kB, limit 9007199254740988kB, failcnt 0
Oct 19 12:00:43 g1-med-online1-206 kernel: Memory cgroup stats for /kubepods/burstable/podd9e034d2-cf48-48e5-b524-f9915b8f3464/1292d8682e805865484d004a38aecab6813f3b8c323f62e8b4ead030a0b8bf07: cache:0KB rss:205744KB rss_huge:45056KB shmem:0KB mapped_file:2376KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:205700KB inactive_file:0KB active_file:0KB unevictable:0KB
Oct 19 12:00:43 g1-med-online1-206 kernel: Tasks state (memory values in pages):
Oct 19 12:00:43 g1-med-online1-206 kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Oct 19 12:00:43 g1-med-online1-206 kernel: [3360427] 0 3360427 1092 150 53248 0 -998 runsvdir
Oct 19 12:00:43 g1-med-online1-206 kernel: [3360618] 0 3360618 1054 105 53248 0 -998 runsv
Oct 19 12:00:43 g1-med-online1-206 kernel: [3360619] 0 3360619 1054 133 53248 0 -998 runsv
Oct 19 12:00:43 g1-med-online1-206 kernel: [3360620] 0 3360620 1054 102 49152 0 -998 runsv
Oct 19 12:00:43 g1-med-online1-206 kernel: [3360621] 0 3360621 1054 130 45056 0 -998 runsv
Oct 19 12:00:43 g1-med-online1-206 kernel: [3360624] 0 3360624 832579 12065 565248 0 -998 calico-node
Oct 19 12:00:43 g1-med-online1-206 kernel: [3360625] 0 3360625 1164613 35682 921600 0 -998 calico-node
Oct 19 12:00:43 g1-med-online1-206 kernel: [3360805] 0 3360805 252 116 24576 0 -998 bird
Oct 19 12:00:43 g1-med-online1-206 kernel: [3360812] 0 3360812 200 76 32768 0 -998 bird6
Oct 19 12:00:43 g1-med-online1-206 kernel: [2580053] 0 2580053 19945 15461 204800 0 -998 iptables-legacy
Oct 19 12:00:43 g1-med-online1-206 kernel: Memory cgroup out of memory: Kill process 3360427 (runsvdir) score 0 or sacrifice child
Oct 19 12:00:43 g1-med-online1-206 kernel: Killed process 3360618 (runsv) total-vm:4216kB, anon-rss:64kB, file-rss:356kB, shmem-rss:0kB
Oct 19 12:00:43 g1-med-online1-206 kernel: iptables-legacy invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=-998
Oct 19 12:00:43 g1-med-online1-206 kernel: iptables-legacy cpuset=1292d8682e805865484d004a38aecab6813f3b8c323f62e8b4ead030a0b8bf07 mems_allowed=0-1
Oct 19 12:00:43 g1-med-online1-206 kernel: CPU: 27 PID: 2580053 Comm: iptables-legacy Kdump: loaded Not tainted 4.19.2-1.el7.elrepo.x86_64 #1
Oct 19 12:00:43 g1-med-online1-206 kernel: Hardware name: Lenovo ThinkSystem SR590 -[7X99CTO1WW]-/-[7X99CTO1WW]-, BIOS -[TEE160N-2.71]- 09/18/2020
Oct 19 12:00:43 g1-med-online1-206 kernel: Call Trace:
Oct 19 12:00:43 g1-med-online1-206 kernel: dump_stack+0x63/0x88
Oct 19 12:00:43 g1-med-online1-206 kernel: dump_header+0x78/0x2a4
Oct 19 12:00:43 g1-med-online1-206 kernel: ? mem_cgroup_scan_tasks+0x9c/0xf0
Oct 19 12:00:43 g1-med-online1-206 kernel: oom_kill_process+0x262/0x290
Oct 19 12:00:43 g1-med-online1-206 kernel: out_of_memory+0x140/0x4b0
Oct 19 12:00:43 g1-med-online1-206 kernel: mem_cgroup_out_of_memory+0x4b/0x80
Oct 19 12:00:43 g1-med-online1-206 kernel: try_charge+0x667/0x6f0
Oct 19 12:00:43 g1-med-online1-206 kernel: mem_cgroup_try_charge+0x8c/0x1e0
Oct 19 12:00:43 g1-med-online1-206 kernel: __add_to_page_cache_locked+0x94/0x1f0
Oct 19 12:00:43 g1-med-online1-206 kernel: add_to_page_cache_lru+0x52/0x100
Oct 19 12:00:43 g1-med-online1-206 kernel: filemap_fault+0x41b/0x6f0
Oct 19 12:00:43 g1-med-online1-206 kernel: ? radix_tree_next_chunk+0x10c/0x2e0
Oct 19 12:00:43 g1-med-online1-206 kernel: ? filemap_map_pages+0x2b2/0x410
Oct 19 12:00:43 g1-med-online1-206 kernel: ext4_filemap_fault+0x31/0x50 [ext4]
Oct 19 12:00:43 g1-med-online1-206 kernel: __do_fault+0x24/0x80
Oct 19 12:00:43 g1-med-online1-206 kernel: __handle_mm_fault+0xc0f/0xef0
Oct 19 12:00:43 g1-med-online1-206 kernel: ? hugepage_vma_check+0xb0/0xd0
Oct 19 12:00:43 g1-med-online1-206 kernel: handle_mm_fault+0x102/0x220
Oct 19 12:00:43 g1-med-online1-206 kernel: __do_page_fault+0x212/0x4e0
Oct 19 12:00:43 g1-med-online1-206 kernel: do_page_fault+0x37/0x140
Oct 19 12:00:43 g1-med-online1-206 kernel: ? page_fault+0x8/0x30
Oct 19 12:00:43 g1-med-online1-206 kernel: page_fault+0x1e/0x30
Oct 19 12:00:43 g1-med-online1-206 kernel: RIP: 0033:0x7f7156699e94
Oct 19 12:00:43 g1-med-online1-206 kernel: Code: 0f b7 4b 5a 49 89 c8 48 01 f1 48 39 d1 0f 84 e3 02 00 00 0f b7 53 58 b9 06 00 00 00 48 8d 3d 08 23 00 00 48 01 da 48 8d 72 02 <f3> a6 0f 97 c1 80 d9 00 84 c9 0f 84 2c 02 00 00 41 8b 7e 60 31 c9
Oct 19 12:00:43 g1-med-online1-206 kernel: RSP: 002b:00007ffc25020e30 EFLAGS: 00010216
Oct 19 12:00:43 g1-med-online1-206 kernel: RAX: 00007f7153979038 RBX: 00007f71556d8748 RCX: 0000000000000006
Oct 19 12:00:43 g1-med-online1-206 kernel: RDX: 00007f71556d87f0 RSI: 00007f71556d87f2 RDI: 00007f715669c195
Oct 19 12:00:43 g1-med-online1-206 kernel: RBP: 0000000001d5f710 R08: 00000000000000d0 R09: 0000000000000000
Oct 19 12:00:43 g1-med-online1-206 kernel: R10: 0000000000000019 R11: 00007f715586f070 R12: 000000000002285d
Oct 19 12:00:43 g1-med-online1-206 kernel: R13: 0000558163b01ed0 R14: 0000558161d15260 R15: 00007ffc25020e58
Oct 19 12:00:43 g1-med-online1-206 kernel: memory: usage 250000kB, limit 250000kB, failcnt 0
Oct 19 12:00:43 g1-med-online1-206 kernel: memory+swap: usage 250000kB, limit 250000kB, failcnt 201232
Oct 19 12:00:43 g1-med-online1-206 kernel: kmem: usage 44204kB, limit 9007199254740988kB, failcnt 0
Oct 19 12:00:43 g1-med-online1-206 kernel: Memory cgroup stats for /kubepods/burstable/podd9e034d2-cf48-48e5-b524-f9915b8f3464/1292d8682e805865484d004a38aecab6813f3b8c323f62e8b4ead030a0b8bf07: cache:0KB rss:205876KB rss_huge:45056KB shmem:0KB mapped_file:2376KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:205764KB inactive_file:0KB active_file:0KB unevictable:0KB
Oct 19 12:00:43 g1-med-online1-206 kernel: Tasks state (memory values in pages):
Oct 19 12:00:43 g1-med-online1-206 kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Oct 19 12:00:43 g1-med-online1-206 kernel: [3360427] 0 3360427 1092 150 53248 0 -998 runsvdir
Oct 19 12:00:43 g1-med-online1-206 kernel: [3360619] 0 3360619 1054 133 53248 0 -998 runsv
Oct 19 12:00:43 g1-med-online1-206 kernel: [3360620] 0 3360620 1054 102 49152 0 -998 runsv
Oct 19 12:00:43 g1-med-online1-206 kernel: [3360621] 0 3360621 1054 130 45056 0 -998 runsv
Oct 19 12:00:43 g1-med-online1-206 kernel: [3360624] 0 3360624 832579 12065 565248 0 -998 calico-node
Oct 19 12:00:43 g1-med-online1-206 kernel: [3360625] 0 3360625 1164613 35682 921600 0 -998 calico-node
Oct 19 12:00:43 g1-med-online1-206 kernel: [3360805] 0 3360805 252 116 24576 0 -998 bird
Oct 19 12:00:43 g1-med-online1-206 kernel: [3360812] 0 3360812 200 76 32768 0 -998 bird6
Oct 19 12:00:43 g1-med-online1-206 kernel: [2580053] 0 2580053 19978 15455 204800 0 -998 iptables-legacy
Oct 19 12:00:43 g1-med-online1-206 kernel: Memory cgroup out of memory: Kill process 3360427 (runsvdir) score 0 or sacrifice child
Oct 19 12:00:43 g1-med-online1-206 kernel: Killed process 3360619 (runsv) total-vm:4216kB, anon-rss:68kB, file-rss:464kB, shmem-rss:0kB
Oct 19 12:00:44 g1-med-online1-206 kernel: iptables-legacy invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=-998
Oct 19 12:00:44 g1-med-online1-206 kernel: iptables-legacy cpuset=1292d8682e805865484d004a38aecab6813f3b8c323f62e8b4ead030a0b8bf07 mems_allowed=0-1
Oct 19 12:00:44 g1-med-online1-206 kernel: CPU: 27 PID: 2580053 Comm: iptables-legacy Kdump: loaded Not tainted 4.19.2-1.el7.elrepo.x86_64 #1
Oct 19 12:00:44 g1-med-online1-206 kernel: Hardware name: Lenovo ThinkSystem SR590 -[7X99CTO1WW]-/-[7X99CTO1WW]-, BIOS -[TEE160N-2.71]- 09/18/2020
Oct 19 12:00:44 g1-med-online1-206 kernel: Call Trace:
Oct 19 12:00:44 g1-med-online1-206 kernel: dump_stack+0x63/0x88
Oct 19 12:00:44 g1-med-online1-206 kernel: dump_header+0x78/0x2a4
Oct 19 12:00:44 g1-med-online1-206 kernel: ? mem_cgroup_scan_tasks+0x9c/0xf0
Oct 19 12:00:44 g1-med-online1-206 kernel: oom_kill_process+0x262/0x290
Oct 19 12:00:44 g1-med-online1-206 kernel: out_of_memory+0x140/0x4b0
Oct 19 12:00:44 g1-med-online1-206 kernel: mem_cgroup_out_of_memory+0x4b/0x80
Oct 19 12:00:44 g1-med-online1-206 kernel: try_charge+0x667/0x6f0
Oct 19 12:00:44 g1-med-online1-206 kernel: ? read_pages+0x7b/0x1a0
Oct 19 12:00:44 g1-med-online1-206 kernel: mem_cgroup_try_charge+0x8c/0x1e0
Oct 19 12:00:44 g1-med-online1-206 kernel: __add_to_page_cache_locked+0x94/0x1f0
Oct 19 12:00:44 g1-med-online1-206 kernel: add_to_page_cache_lru+0x52/0x100
Oct 19 12:00:44 g1-med-online1-206 kernel: filemap_fault+0x41b/0x6f0
Oct 19 12:00:44 g1-med-online1-206 kernel: ? radix_tree_next_chunk+0x10c/0x2e0
Oct 19 12:00:44 g1-med-online1-206 kernel: ? filemap_map_pages+0x187/0x410
Oct 19 12:00:44 g1-med-online1-206 kernel: ext4_filemap_fault+0x31/0x50 [ext4]
Oct 19 12:00:44 g1-med-online1-206 kernel: __do_fault+0x24/0x80
Oct 19 12:00:44 g1-med-online1-206 kernel: __handle_mm_fault+0xc0f/0xef0
Oct 19 12:00:44 g1-med-online1-206 kernel: handle_mm_fault+0x102/0x220
Oct 19 12:00:44 g1-med-online1-206 kernel: __do_page_fault+0x212/0x4e0
Oct 19 12:00:44 g1-med-online1-206 kernel: do_page_fault+0x37/0x140
Oct 19 12:00:44 g1-med-online1-206 kernel: ? page_fault+0x8/0x30
Oct 19 12:00:44 g1-med-online1-206 kernel: page_fault+0x1e/0x30
Oct 19 12:00:44 g1-med-online1-206 kernel: RIP: 0033:0x7f715584c91c
Oct 19 12:00:44 g1-med-online1-206 kernel: Code: 77 c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 f3 0f 1e fa 89 f9 c5 f9 6e c6 c4 41 31 ef c9 c4 e2 7d 78 c0 83 e1 3f 83 f9 20 77 34 <c5> 7e 6f 07 c4 c1 7d 74 c8 c4 c1 35 74 d0 c5 ed eb c9 c5 fd d7 c1
Oct 19 12:00:44 g1-med-online1-206 kernel: RSP: 002b:00007ffc25020888 EFLAGS: 00010287
Oct 19 12:00:44 g1-med-online1-206 kernel: RAX: 0000558160d2ead9 RBX: 00007f7155aad780 RCX: 000000000000001a
Oct 19 12:00:44 g1-med-online1-206 kernel: RDX: 0000000000000006 RSI: 0000000000000025 RDI: 0000558160d2eada
Oct 19 12:00:44 g1-med-online1-206 kernel: RBP: 00007ffc25020e00 R08: 00007f715669c0b5 R09: 00007f715586d300
Oct 19 12:00:44 g1-med-online1-206 kernel: R10: 0000000000000000 R11: 00007f715586d380 R12: 0000000000000006
Oct 19 12:00:44 g1-med-online1-206 kernel: R13: 00007ffc25020e10 R14: 0000558160d2eada R15: 0000000000000006
Oct 19 12:00:44 g1-med-online1-206 kernel: Task in /kubepods/burstable/podd9e034d2-cf48-48e5-b524-f9915b8f3464/1292d8682e805865484d004a38aecab6813f3b8c323f62e8b4ead030a0b8bf07 killed as a result of limit of /kubepods/burstable/podd9e034d2-cf48-48e5-b524-f9915b8f3464/1292d8682e805865484d004a38aecab6813f3b8c323f62e8b4ead030a0b8bf07
Oct 19 12:00:44 g1-med-online1-206 kernel: memory: usage 250000kB, limit 250000kB, failcnt 0
Oct 19 12:00:44 g1-med-online1-206 kernel: memory+swap: usage 250000kB, limit 250000kB, failcnt 202685
Oct 19 12:00:44 g1-med-online1-206 kernel: kmem: usage 44124kB, limit 9007199254740988kB, failcnt 0
Oct 19 12:00:44 g1-med-online1-206 kernel: Memory cgroup stats for /kubepods/burstable/podd9e034d2-cf48-48e5-b524-f9915b8f3464/1292d8682e805865484d004a38aecab6813f3b8c323f62e8b4ead030a0b8bf07: cache:0KB rss:206008KB rss_huge:45056KB shmem:0KB mapped_file:2376KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:205808KB inactive_file:44KB active_file:24KB unevictable:0KB
Oct 19 12:00:44 g1-med-online1-206 kernel: Tasks state (memory values in pages):
Oct 19 12:00:44 g1-med-online1-206 kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Oct 19 12:00:44 g1-med-online1-206 kernel: [3360427] 0 3360427 1092 150 53248 0 -998 runsvdir
Oct 19 12:00:44 g1-med-online1-206 kernel: [3360620] 0 3360620 1054 102 49152 0 -998 runsv
Oct 19 12:00:44 g1-med-online1-206 kernel: [3360621] 0 3360621 1054 130 45056 0 -998 runsv
Oct 19 12:00:44 g1-med-online1-206 kernel: [3360624] 0 3360624 832579 12065 565248 0 -998 calico-node
Oct 19 12:00:44 g1-med-online1-206 kernel: [3360625] 0 3360625 1164613 35682 921600 0 -998 calico-node
Oct 19 12:00:44 g1-med-online1-206 kernel: [3360805] 0 3360805 252 116 24576 0 -998 bird
Oct 19 12:00:44 g1-med-online1-206 kernel: [3360812] 0 3360812 200 76 32768 0 -998 bird6
Oct 19 12:00:44 g1-med-online1-206 kernel: [2580053] 0 2580053 19978 15531 204800 0 -998 iptables-legacy
Oct 19 12:00:44 g1-med-online1-206 kernel: Memory cgroup out of memory: Kill process 3360427 (runsvdir) score 0 or sacrifice child
Oct 19 12:00:44 g1-med-online1-206 kernel: Killed process 3360620 (runsv) total-vm:4216kB, anon-rss:64kB, file-rss:344kB, shmem-rss:0kB
Oct 19 12:00:44 g1-med-online1-206 kubelet: I1019 12:00:44.325817 981242 setters.go:73] Using node IP: "10.16.25.206"
Oct 19 12:00:45 g1-med-online1-206 kubelet: E1019 12:00:45.906340 981242 pod_workers.go:191] Error syncing pod 5ff33e66-b4d8-4934-b036-6a757f72e9e5 ("budget-sys-go-serve-lsvxb-776bf76b7-rs5kv_bi(5ff33e66-b4d8-4934-b036-6a757f72e9e5)"), skipping: failed to "StartContainer" for "go-serve" with CrashLoopBackOff: "back-off 5m0s restarting failed container=go-serve pod=budget-sys-go-serve-lsvxb-776bf76b7-rs5kv_bi(5ff33e66-b4d8-4934-b036-6a757f72e9e5)"
Oct 19 12:00:46 g1-med-online1-206 kernel: iptables-legacy invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=-998
Oct 19 12:00:46 g1-med-online1-206 kernel: iptables-legacy cpuset=1292d8682e805865484d004a38aecab6813f3b8c323f62e8b4ead030a0b8bf07 mems_allowed=0-1
Oct 19 12:00:46 g1-med-online1-206 kernel: CPU: 7 PID: 2580315 Comm: iptables-legacy Kdump: loaded Not tainted 4.19.2-1.el7.elrepo.x86_64 #1
Oct 19 12:00:46 g1-med-online1-206 kernel: Hardware name: Lenovo ThinkSystem SR590 -[7X99CTO1WW]-/-[7X99CTO1WW]-, BIOS -[TEE160N-2.71]- 09/18/2020
Oct 19 12:00:46 g1-med-online1-206 kernel: Call Trace:
Oct 19 12:00:46 g1-med-online1-206 kernel: dump_stack+0x63/0x88
Oct 19 12:00:46 g1-med-online1-206 kernel: dump_header+0x78/0x2a4
Oct 19 12:00:46 g1-med-online1-206 kernel: ? mem_cgroup_scan_tasks+0x9c/0xf0
Oct 19 12:00:46 g1-med-online1-206 kernel: oom_kill_process+0x262/0x290
Oct 19 12:00:46 g1-med-online1-206 kernel: out_of_memory+0x140/0x4b0
Oct 19 12:00:46 g1-med-online1-206 kernel: mem_cgroup_out_of_memory+0x4b/0x80
Oct 19 12:00:46 g1-med-online1-206 kernel: try_charge+0x667/0x6f0
Oct 19 12:00:46 g1-med-online1-206 kernel: ? __alloc_pages_nodemask+0xf8/0x260
Oct 19 12:00:46 g1-med-online1-206 kernel: mem_cgroup_try_charge+0x8c/0x1e0
Oct 19 12:00:46 g1-med-online1-206 kernel: mem_cgroup_try_charge_delay+0x22/0x50
Oct 19 12:00:46 g1-med-online1-206 kernel: do_anonymous_page+0x11a/0x650
Oct 19 12:00:46 g1-med-online1-206 kernel: ? __handle_mm_fault+0xc5e/0xef0
Oct 19 12:00:46 g1-med-online1-206 kernel: __handle_mm_fault+0xc5e/0xef0
Oct 19 12:00:46 g1-med-online1-206 kernel: handle_mm_fault+0x102/0x220
Oct 19 12:00:46 g1-med-online1-206 kernel: __do_page_fault+0x212/0x4e0
Oct 19 12:00:46 g1-med-online1-206 kernel: do_page_fault+0x37/0x140
Oct 19 12:00:46 g1-med-online1-206 kernel: page_fault+0x1e/0x30
Oct 19 12:00:46 g1-med-online1-206 kernel: RIP: 0010:copy_user_enhanced_fast_string+0xe/0x20
Oct 19 12:00:46 g1-med-online1-206 kernel: Code: 89 d1 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 31 c0 0f 01 ca c3 0f 1f 80 00 00 00 00 0f 01 cb 83 fa 40 0f 82 70 ff ff ff 89 d1 <f3> a4 31 c0 0f 01 ca c3 66 2e 0f 1f 84 00 00 00 00 00 0f 01 cb 83
Oct 19 12:00:46 g1-med-online1-206 kernel: RSP: 0018:ffffc90024c8bd98 EFLAGS: 00050206
Oct 19 12:00:46 g1-med-online1-206 kernel: RAX: 00007f069b089020 RBX: ffffc90065ff1fb8 RCX: 0000000000000020
Oct 19 12:00:46 g1-med-online1-206 kernel: RDX: 0000000000000070 RSI: ffffc90065ff2008 RDI: 00007f069b089000
Oct 19 12:00:46 g1-med-online1-206 kernel: RBP: ffffc90024c8bda0 R08: 0000000000000004 R09: 0000000000000000
Oct 19 12:00:46 g1-med-online1-206 kernel: R10: 0000000000000001 R11: 0000000000000018 R12: ffff883fb47d7e40
Oct 19 12:00:46 g1-med-online1-206 kernel: R13: 0000000000c3af78 R14: ffffc90065ff1ea0 R15: 00000000000000a0
Oct 19 12:00:46 g1-med-online1-206 kernel: ? _copy_to_user+0x26/0x40
Oct 19 12:00:46 g1-med-online1-206 kernel: do_ipt_get_ctl+0x1fe/0x300 [ip_tables]
Oct 19 12:00:46 g1-med-online1-206 kernel: nf_getsockopt+0x4f/0x70
Oct 19 12:00:46 g1-med-online1-206 kernel: ip_getsockopt+0x70/0xb0
Oct 19 12:00:46 g1-med-online1-206 kernel: raw_getsockopt+0x22/0x40
Oct 19 12:00:46 g1-med-online1-206 kernel: sock_common_getsockopt+0x1a/0x20
Oct 19 12:00:46 g1-med-online1-206 kernel: __sys_getsockopt+0x7d/0xe0
Oct 19 12:00:46 g1-med-online1-206 kernel: __x64_sys_getsockopt+0x24/0x30
Oct 19 12:00:46 g1-med-online1-206 kernel: do_syscall_64+0x60/0x190
Oct 19 12:00:46 g1-med-online1-206 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Oct 19 12:00:46 g1-med-online1-206 kernel: RIP: 0033:0x7f069d6f727e
Oct 19 12:00:46 g1-med-online1-206 kernel: Code: 48 8b 0d 0d fc 2b 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 37 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d da fb 2b 00 f7 d8 64 89 01 48
Oct 19 12:00:46 g1-med-online1-206 kernel: RSP: 002b:00007fff03587268 EFLAGS: 00000202 ORIG_RAX: 0000000000000037
Oct 19 12:00:46 g1-med-online1-206 kernel: RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007f069d6f727e
Oct 19 12:00:46 g1-med-online1-206 kernel: RDX: 0000000000000041 RSI: 0000000000000000 RDI: 0000000000000004
Oct 19 12:00:46 g1-med-online1-206 kernel: RBP: 0000000000000001 R08: 00007fff03587290 R09: 0000000000000000
Oct 19 12:00:46 g1-med-online1-206 kernel: R10: 00007f069a44e010 R11: 0000000000000202 R12: 0000562035d358a8
Oct 19 12:00:46 g1-med-online1-206 kernel: R13: 0000000000000000 R14: 0000562035d358a0 R15: 00007fff035872a0
Oct 19 12:00:46 g1-med-online1-206 kernel: Task in /kubepods/burstable/podd9e034d2-cf48-48e5-b524-f9915b8f3464/1292d8682e805865484d004a38aecab6813f3b8c323f62e8b4ead030a0b8bf07 killed as a result of limit of /kubepods/burstable/podd9e034d2-cf48-48e5-b524-f9915b8f3464/1292d8682e805865484d004a38aecab6813f3b8c323f62e8b4ead030a0b8bf07
Oct 19 12:00:46 g1-med-online1-206 kernel: memory: usage 250000kB, limit 250000kB, failcnt 0
Oct 19 12:00:46 g1-med-online1-206 kernel: memory+swap: usage 250000kB, limit 250000kB, failcnt 205624
Oct 19 12:00:46 g1-med-online1-206 kernel: kmem: usage 45640kB, limit 9007199254740988kB, failcnt 0
Oct 19 12:00:46 g1-med-online1-206 kernel: Memory cgroup stats for /kubepods/burstable/podd9e034d2-cf48-48e5-b524-f9915b8f3464/1292d8682e805865484d004a38aecab6813f3b8c323f62e8b4ead030a0b8bf07: cache:0KB rss:204264KB rss_huge:45056KB shmem:0KB mapped_file:924KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:204248KB inactive_file:20KB active_file:24KB unevictable:0KB
Oct 19 12:00:46 g1-med-online1-206 kernel: Tasks state (memory values in pages):
Oct 19 12:00:46 g1-med-online1-206 kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Oct 19 12:00:46 g1-med-online1-206 kernel: [3360427] 0 3360427 1092 150 53248 0 -998 runsvdir
Oct 19 12:00:46 g1-med-online1-206 kernel: [3360621] 0 3360621 1054 130 45056 0 -998 runsv
Oct 19 12:00:46 g1-med-online1-206 kernel: [3360624] 0 3360624 832579 12065 565248 0 -998 calico-node
Oct 19 12:00:46 g1-med-online1-206 kernel: [3360625] 0 3360625 1164613 29612 921600 0 -998 calico-node
Oct 19 12:00:46 g1-med-online1-206 kernel: [3360805] 0 3360805 252 101 24576 0 -998 bird
Oct 19 12:00:46 g1-med-online1-206 kernel: [3360812] 0 3360812 202 47 32768 0 -998 bird6
Oct 19 12:00:46 g1-med-online1-206 kernel: [2580053] 0 2580053 24148 15621 237568 0 -998 iptables-legacy
Oct 19 12:00:46 g1-med-online1-206 kernel: [2580224] 0 2580224 1054 160 53248 0 -998 runsv
Oct 19 12:00:46 g1-med-online1-206 kernel: [2580225] 0 2580225 1054 131 49152 0 -998 runsv
Oct 19 12:00:46 g1-med-online1-206 kernel: [2580226] 0 2580226 1054 130 49152 0 -998 runsv
Oct 19 12:00:46 g1-med-online1-206 kernel: [2580236] 0 2580236 648057 7912 475136 0 -998 calico-node
Oct 19 12:00:46 g1-med-online1-206 kernel: [2580315] 0 2580315 17496 3515 151552 0 -998 iptables-legacy
Oct 19 12:00:46 g1-med-online1-206 kernel: Memory cgroup out of memory: Kill process 3360427 (runsvdir) score 0 or sacrifice child
Oct 19 12:00:46 g1-med-online1-206 kernel: Killed process 3360621 (runsv) total-vm:4216kB, anon-rss:64kB, file-rss:456kB, shmem-rss:0kB
Oct 19 12:00:46 g1-med-online1-206 kernel: iptables-legacy invoked oom-killer: gfp_mask=0x6000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=-998
Oct 19 12:00:46 g1-med-online1-206 kernel: iptables-legacy cpuset=1292d8682e805865484d004a38aecab6813f3b8c323f62e8b4ead030a0b8bf07 mems_allowed=0-1
Oct 19 12:00:46 g1-med-online1-206 kernel: CPU: 2 PID: 2580315 Comm: iptables-legacy Kdump: loaded Not tainted 4.19.2-1.el7.elrepo.x86_64 #1
Oct 19 12:00:46 g1-med-online1-206 kernel: Hardware name: Lenovo ThinkSystem SR590 -[7X99CTO1WW]-/-[7X99CTO1WW]-, BIOS -[TEE160N-2.71]- 09/18/2020
Oct 19 12:00:46 g1-med-online1-206 kernel: Call Trace:
Oct 19 12:00:46 g1-med-online1-206 kernel: dump_stack+0x63/0x88
Oct 19 12:00:46 g1-med-online1-206 kernel: dump_header+0x78/0x2a4
Oct 19 12:00:46 g1-med-online1-206 kernel: ? mem_cgroup_scan_tasks+0x9c/0xf0
Oct 19 12:00:46 g1-med-online1-206 kernel: oom_kill_process+0x262/0x290
Oct 19 12:00:46 g1-med-online1-206 kernel: out_of_memory+0x140/0x4b0
Oct 19 12:00:46 g1-med-online1-206 kernel: mem_cgroup_out_of_memory+0x4b/0x80
Oct 19 12:00:46 g1-med-online1-206 kernel: try_charge+0x667/0x6f0
Oct 19 12:00:46 g1-med-online1-206 kernel: ? __alloc_pages_nodemask+0xf8/0x260
Oct 19 12:00:46 g1-med-online1-206 kernel: mem_cgroup_try_charge+0x8c/0x1e0
Oct 19 12:00:46 g1-med-online1-206 kernel: mem_cgroup_try_charge_delay+0x22/0x50
Oct 19 12:00:46 g1-med-online1-206 kernel: do_anonymous_page+0x11a/0x650
Oct 19 12:00:46 g1-med-online1-206 kernel: __handle_mm_fault+0xc5e/0xef0
Oct 19 12:00:46 g1-med-online1-206 kernel: handle_mm_fault+0x102/0x220
Oct 19 12:00:46 g1-med-online1-206 kernel: __do_page_fault+0x212/0x4e0
Oct 19 12:00:46 g1-med-online1-206 kernel: do_page_fault+0x37/0x140
Oct 19 12:00:46 g1-med-online1-206 kernel: page_fault+0x1e/0x30
Oct 19 12:00:46 g1-med-online1-206 kernel: RIP: 0010:copy_user_enhanced_fast_string+0xe/0x20
Oct 19 12:00:46 g1-med-online1-206 kernel: RIP: 0033:0x7f069d6f727e
Oct 19 12:00:46 g1-med-online1-206 kernel: Code: 48 8b 0d 0d fc 2b 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 37 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d da fb 2b 00 f7 d8 64 89 01 48
Oct 19 12:00:46 g1-med-online1-206 kernel: RSP: 002b:00007fff03587268 EFLAGS: 00000202 ORIG_RAX: 0000000000000037
Oct 19 12:00:46 g1-med-online1-206 kernel: RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007f069d6f727e
Oct 19 12:00:46 g1-med-online1-206 kernel: RDX: 0000000000000041 RSI: 0000000000000000 RDI: 0000000000000004
Oct 19 12:00:46 g1-med-online1-206 kernel: RBP: 0000000000000001 R08: 00007fff03587290 R09: 0000000000000000
Oct 19 12:00:46 g1-med-online1-206 kernel: R10: 00007f069a44e010 R11: 0000000000000202 R12: 0000562035d358a8
Oct 19 12:00:46 g1-med-online1-206 kernel: R13: 0000000000000000 R14: 0000562035d358a0 R15: 00007fff035872a0
Oct 19 12:00:46 g1-med-online1-206 kernel: Task in /kubepods/burstable/podd9e034d2-cf48-48e5-b524-f9915b8f3464/1292d8682e805865484d004a38aecab6813f3b8c323f62e8b4ead030a0b8bf07 killed as a result of limit of /kubepods/burstable/podd9e034d2-cf48-48e5-b524-f9915b8f3464/1292d8682e805865484d004a38aecab6813f3b8c323f62e8b4ead030a0b8bf07
Oct 19 12:00:46 g1-med-online1-206 kernel: memory: usage 250000kB, limit 250000kB, failcnt 0
Oct 19 12:00:46 g1-med-online1-206 kernel: memory+swap: usage 250000kB, limit 250000kB, failcnt 205647
Oct 19 12:00:46 g1-med-online1-206 kernel: kmem: usage 45540kB, limit 9007199254740988kB, failcnt 0
Oct 19 12:00:46 g1-med-online1-206 kernel: Memory cgroup stats for /kubepods/burstable/podd9e034d2-cf48-48e5-b524-f9915b8f3464/1292d8682e805865484d004a38aecab6813f3b8c323f62e8b4ead030a0b8bf07: cache:0KB rss:204396KB rss_huge:45056KB shmem:0KB mapped_file:924KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:204360KB inactive_file:20KB active_file:12KB unevictable:0KB
Oct 19 12:00:46 g1-med-online1-206 kernel: Tasks state (memory values in pages):
Oct 19 12:00:46 g1-med-online1-206 kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Oct 19 12:00:46 g1-med-online1-206 kernel: [3360427] 0 3360427 1092 150 53248 0 -998 runsvdir
Oct 19 12:00:46 g1-med-online1-206 kernel: [3360624] 0 3360624 832579 12065 565248 0 -998 calico-node
Oct 19 12:00:46 g1-med-online1-206 kernel: [3360625] 0 3360625 1164613 29612 921600 0 -998 calico-node
Oct 19 12:00:46 g1-med-online1-206 kernel: [3360805] 0 3360805 252 101 24576 0 -998 bird
Oct 19 12:00:46 g1-med-online1-206 kernel: [3360812] 0 3360812 202 47 32768 0 -998 bird6
Oct 19 12:00:46 g1-med-online1-206 kernel: [2580053] 0 2580053 24148 15621 237568 0 -998 iptables-legacy
Oct 19 12:00:46 g1-med-online1-206 kernel: [2580224] 0 2580224 1054 160 53248 0 -998 runsv
Oct 19 12:00:46 g1-med-online1-206 kernel: [2580225] 0 2580225 1054 131 49152 0 -998 runsv
Oct 19 12:00:46 g1-med-online1-206 kernel: [2580226] 0 2580226 1054 130 49152 0 -998 runsv
Oct 19 12:00:46 g1-med-online1-206 kernel: [2580236] 0 2580236 648057 7912 475136 0 -998 calico-node
Oct 19 12:00:46 g1-med-online1-206 kernel: [2580315] 0 2580315 17496 3581 151552 0 -998 iptables-legacy
Oct 19 12:00:46 g1-med-online1-206 kernel: Memory cgroup out of memory: Kill process 3360427 (runsvdir) score 0 or sacrifice child
Oct 19 12:00:46 g1-med-online1-206 kernel: Killed process 3360625 (calico-node) total-vm:4658452kB, anon-rss:89188kB, file-rss:29888kB, shmem-rss:0kB
Oct 19 12:00:46 g1-med-online1-206 kernel: oom_reaper: reaped process 3360625 (calico-node), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
标签：分析,med,00,12,online1,g1,oom,Oct,calico 来源： https://www.cnblogs.com/FengGeBlog/p/15424313.html