Hello, I had an F11 host with virt-preview repo configured and I have used it a lot without particular problems. Now I'm on F12 on the same host after upgrade and I'm experiencing guests' death caused by oom killer.
Example. - all VM (4) are qemu/kvm and x86_64 - boot of host on 26/11 about 19:20. - two centos 5.3 guests are configured to startup automatically and they indeed start In messages I see this on their startup: Nov 26 19:23:01 virtfed kernel: kvm: 3458: cpu0 unimplemented perfctr wrmsr: 0x186 data 0x130079 Nov 26 19:23:01 virtfed kernel: kvm: 3458: cpu0 unimplemented perfctr wrmsr: 0xc1 data 0xffcfadc8 Nov 26 19:23:01 virtfed kernel: kvm: 3458: cpu0 unimplemented perfctr wrmsr: 0x186 data 0x530079 Nov 26 19:23:01 virtfed kernel: kvm: 3458: cpu1 unimplemented perfctr wrmsr: 0x186 data 0x130079 Nov 26 19:23:01 virtfed kernel: kvm: 3458: cpu1 unimplemented perfctr wrmsr: 0xc1 data 0xffcfadc8 Nov 26 19:23:01 virtfed kernel: kvm: 3458: cpu1 unimplemented perfctr wrmsr: 0x186 data 0x530079 Nov 26 19:23:01 virtfed kernel: kvm: 3501: cpu0 unimplemented perfctr wrmsr: 0x186 data 0x130079 Nov 26 19:23:01 virtfed kernel: kvm: 3501: cpu0 unimplemented perfctr wrmsr: 0xc1 data 0xffcfadc8 Nov 26 19:23:01 virtfed kernel: kvm: 3501: cpu0 unimplemented perfctr wrmsr: 0x186 data 0x530079 Nov 26 19:23:01 virtfed kernel: kvm: 3501: cpu1 unimplemented perfctr wrmsr: 0x186 data 0x130079 Nov 26 19:23:36 virtfed kernel: kvm: emulating exchange as write
- On 27/11 at 12:03 I start other two guests in centos 5.4 Again I get rows similar to the above ones plus: Nov 27 12:03:19 virtfed kernel: __ratelimit: 2 callbacks suppressed
The day after at 06:55
Nov 28 03:27:06 virtfed logrotate: ALERT exited abnormally with [1] Nov 28 06:55:01 virtfed kernel: qemu-kvm invoked oom-killer: gfp_mask=0x201da, order=0, oomkilladj=0 Nov 28 06:55:01 virtfed kernel: qemu-kvm cpuset=/ mems_allowed=0 Nov 28 06:55:01 virtfed kernel: Pid: 405, comm: qemu-kvm Tainted: G W 2.6.31.5-127.fc12.x86_64 #1 Nov 28 06:55:01 virtfed kernel: Call Trace: Nov 28 06:55:01 virtfed kernel: [<ffffffff8108a7d7>] ? cpuset_print_task_mems_allowed+0x91/0x9d Nov 28 06:55:01 virtfed kernel: [<ffffffff810c38e9>] oom_kill_process+0x98/0x256 Nov 28 06:55:01 virtfed kernel: [<ffffffff810c3d6a>] ? select_bad_process+0xa3/0x102 Nov 28 06:55:01 virtfed kernel: [<ffffffff810c3e53>] __out_of_memory+0x8a/0x99 Nov 28 06:55:01 virtfed kernel: [<ffffffff810c3fc5>] out_of_memory+0x163/0x195 Nov 28 06:55:01 virtfed kernel: [<ffffffff810c75c1>] __alloc_pages_nodemask+0x491/0x584 Nov 28 06:55:01 virtfed kernel: [<ffffffff810eb1c4>] alloc_pages_current+0x95/0x9e Nov 28 06:55:01 virtfed kernel: [<ffffffff810c15c1>] __page_cache_alloc+0x5f/0x61 Nov 28 06:55:01 virtfed kernel: [<ffffffff810c9235>] __do_page_cache_readahead+0x98/0x176 Nov 28 06:55:01 virtfed kernel: [<ffffffff810c9334>] ra_submit+0x21/0x25 Nov 28 06:55:01 virtfed kernel: [<ffffffff810c1abd>] filemap_fault+0x193/0x317 Nov 28 06:55:01 virtfed kernel: [<ffffffff810d6641>] __do_fault+0x54/0x3c4 Nov 28 06:55:01 virtfed kernel: [<ffffffff81017bc1>] ? read_tsc+0x9/0x1b Nov 28 06:55:01 virtfed kernel: [<ffffffff810d881a>] handle_mm_fault+0x2f6/0x705 Nov 28 06:55:01 virtfed kernel: [<ffffffff8106a9ff>] ? __hrtimer_start_range_ns+0x2bb/0x2cd Nov 28 06:55:01 virtfed kernel: [<ffffffff8141cfe1>] do_page_fault+0x281/0x299 Nov 28 06:55:01 virtfed kernel: [<ffffffff8141af75>] page_fault+0x25/0x30 Nov 28 06:55:01 virtfed kernel: Mem-Info: Nov 28 06:55:01 virtfed kernel: Node 0 DMA per-cpu: Nov 28 06:55:01 virtfed kernel: CPU 0: hi: 0, btch: 1 usd: 0 Nov 28 06:55:01 virtfed kernel: CPU 1: hi: 0, btch: 1 usd: 0 Nov 28 06:55:01 virtfed kernel: CPU 2: hi: 0, btch: 1 usd: 0 Nov 28 06:55:01 virtfed kernel: CPU 3: hi: 0, btch: 1 usd: 0 Nov 28 06:55:01 virtfed kernel: CPU 4: hi: 0, btch: 1 usd: 0 Nov 28 06:55:01 virtfed kernel: CPU 5: hi: 0, btch: 1 usd: 0 Nov 28 06:55:01 virtfed kernel: CPU 6: hi: 0, btch: 1 usd: 0 Nov 28 06:55:01 virtfed kernel: CPU 7: hi: 0, btch: 1 usd: 0 Nov 28 06:55:01 virtfed kernel: Node 0 DMA32 per-cpu: Nov 28 06:55:01 virtfed kernel: CPU 0: hi: 186, btch: 31 usd: 86 Nov 28 06:55:01 virtfed kernel: CPU 1: hi: 186, btch: 31 usd: 136 Nov 28 06:55:01 virtfed kernel: CPU 2: hi: 186, btch: 31 usd: 51 Nov 28 06:55:01 virtfed kernel: CPU 3: hi: 186, btch: 31 usd: 126 Nov 28 06:55:01 virtfed kernel: CPU 4: hi: 186, btch: 31 usd: 86 Nov 28 06:55:01 virtfed kernel: CPU 5: hi: 186, btch: 31 usd: 34 Nov 28 06:55:01 virtfed kernel: CPU 6: hi: 186, btch: 31 usd: 171 Nov 28 06:55:01 virtfed kernel: CPU 7: hi: 186, btch: 31 usd: 169 Nov 28 06:55:01 virtfed kernel: Node 0 Normal per-cpu: Nov 28 06:55:01 virtfed kernel: CPU 0: hi: 186, btch: 31 usd: 173 Nov 28 06:55:01 virtfed kernel: CPU 1: hi: 186, btch: 31 usd: 42 Nov 28 06:55:01 virtfed kernel: CPU 2: hi: 186, btch: 31 usd: 160 Nov 28 06:55:01 virtfed kernel: CPU 3: hi: 186, btch: 31 usd: 109 Nov 28 06:55:01 virtfed kernel: CPU 4: hi: 186, btch: 31 usd: 112 Nov 28 06:55:01 virtfed kernel: CPU 5: hi: 186, btch: 31 usd: 112 Nov 28 06:55:01 virtfed kernel: CPU 6: hi: 186, btch: 31 usd: 174 Nov 28 06:55:01 virtfed kernel: CPU 7: hi: 186, btch: 31 usd: 118 Nov 28 06:55:01 virtfed kernel: Active_anon:2537459 active_file:3 inactive_anon:364350 Nov 28 06:55:01 virtfed kernel: inactive_file:160 unevictable:12562 dirty:0 writeback:0 unstable:0 Nov 28 06:55:01 virtfed kernel: free:16256 slab:29833 mapped:7872 pagetables:10697 bounce:0 Nov 28 06:55:01 virtfed kernel: Node 0 DMA free:15824kB min:16kB low:20kB high:24kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB present:15320kB pages_scanned:0 all_unreclaimable? yes Nov 28 06:55:01 virtfed kernel: lowmem_reserve[]: 0 3254 12092 12092 Nov 28 06:55:01 virtfed kernel: Node 0 DMA32 free:39060kB min:3784kB low:4728kB high:5676kB active_anon:2510452kB inactive_anon:502304kB active_file:12kB inactive_file:244kB unevictable:520kB present:3332660kB pages_scanned:120 all_unreclaimable? no Nov 28 06:55:01 virtfed kernel: lowmem_reserve[]: 0 0 8837 8837 Nov 28 06:55:01 virtfed kernel: Node 0 Normal free:10140kB min:10280kB low:12848kB high:15420kB active_anon:7639384kB inactive_anon:955096kB active_file:0kB inactive_file:396kB unevictable:49728kB present:9049596kB pages_scanned:0 all_unreclaimable? no Nov 28 06:55:01 virtfed kernel: lowmem_reserve[]: 0 0 0 0 Nov 28 06:55:01 virtfed kernel: Node 0 DMA: 2*4kB 1*8kB 2*16kB 1*32kB 2*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15824kB Nov 28 06:55:01 virtfed kernel: Node 0 DMA32: 3793*4kB 2168*8kB 68*16kB 11*32kB 4*64kB 6*128kB 3*256kB 2*512kB 2*1024kB 0*2048kB 0*4096kB = 38820kB Nov 28 06:55:01 virtfed kernel: Node 0 Normal: 1948*4kB 5*8kB 0*16kB 0*32kB 2*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 10264kB Nov 28 06:55:01 virtfed kernel: 26703 total pagecache pages Nov 28 06:55:01 virtfed kernel: 18747 pages in swap cache Nov 28 06:55:01 virtfed kernel: Swap cache stats: add 1236041, delete 1217294, find 52896/69574 Nov 28 06:55:01 virtfed kernel: Free swap = 0kB Nov 28 06:55:01 virtfed kernel: Total swap = 4194296kB Nov 28 06:55:01 virtfed kernel: 3145727 pages RAM Nov 28 06:55:01 virtfed kernel: 63020 pages reserved Nov 28 06:55:01 virtfed kernel: 76044 pages shared Nov 28 06:55:01 virtfed kernel: 3053641 pages non-shared Nov 28 06:55:01 virtfed kernel: virbr0: port 3(vnet4) entering disabled state Nov 28 06:55:01 virtfed kernel: device vnet4 left promiscuous mode Nov 28 06:55:01 virtfed kernel: virbr0: port 3(vnet4) entering disabled state
About 1 hour and a half after, at 08:14 I get the same for the other 5.4 vm. The same happens for other processes
Any hints on this and on how to debug? Any other one? I don't know if it can be related, but I also tried to reboot disabling ksm and ksmtuned: [root@virtfed ~]# chkconfig --list ksmtuned ksmtuned 0:off 1:off 2:off 3:off 4:off 5:off 6:off [root@virtfed ~]# chkconfig --list ksm ksm 0:off 1:off 2:off 3:off 4:off 5:off 6:off
getting the same problem... strangely, I get anyway this: [root@virtfed ~]# service ksm status ksm is running
Why ksm is running? How can I configure a VM to be or not to be managed with ksm? Or is it an overall feature to enable/disable?
On 11/30/2009 04:00 PM, Gianluca Cecchi wrote:
Hello, I had an F11 host with virt-preview repo configured and I have used it a lot without particular problems. Now I'm on F12 on the same host after upgrade and I'm experiencing guests' death caused by oom killer.
Example.
- all VM (4) are qemu/kvm and x86_64
- boot of host on 26/11 about 19:20.
- two centos 5.3 guests are configured to startup automatically and
they indeed start
How much memory have you allocated to guests, and how much memory do you have in your system? How much swap space?
On Mon, Nov 30, 2009 at 3:38 PM, Avi Kivity avi@redhat.com wrote:
How much memory have you allocated to guests, and how much memory do you have in your system? How much swap space?
The system has 10Gb of ram. Swap is configured for 4Gb 2 guests have 1Gb of ram configured 2 guests have 3,5Gb of ram configured. no other guests running Now I have the 2 guests with 1Gb of ram up and running and the situation is
[root@virtfed log]# free total used free shared buffers cached Mem: 12330828 6005764 6325064 0 1099052 234008 -/+ buffers/cache: 4672704 7658124 Swap: 4194296 15880 4178416
In the mean time I have just installed sysstat to check memory usage during time...
On Mon, Nov 30, 2009 at 3:48 PM, Gianluca Cecchi gianluca.cecchi@gmail.com wrote:
Now I have the 2 guests with 1Gb of ram up and running and the situation is
[root@virtfed log]# free total used free shared buffers cached Mem: 12330828 6005764 6325064 0 1099052 234008 -/+ buffers/cache: 4672704 7658124 Swap: 4194296 15880 4178416
In the mean time I have just installed sysstat to check memory usage during time...
I wrote about host having 10Gb of ram. Actually it has 12Gb of ram (as seen btw from the output of the "free" command). And after starting the other two guests (3,5Gb of ram configured for each one): [root@virtfed cluster]# free total used free shared buffers cached Mem: 12330828 7233912 5096916 0 1567600 245032 -/+ buffers/cache: 5421280 6909548 Swap: 4194296 15596 4178700
* Gianluca Cecchi gianluca.cecchi@gmail.com [20091130 15:59]:
On Mon, Nov 30, 2009 at 3:48 PM, Gianluca Cecchi
[snip]
I wrote about host having 10Gb of ram. Actually it has 12Gb of ram (as seen btw from the output of the "free" command). And after starting the other two guests (3,5Gb of ram configured for each one): [root@virtfed cluster]# free total used free shared buffers cached Mem: 12330828 7233912 5096916 0 1567600 245032 -/+ buffers/cache: 5421280 6909548 Swap: 4194296 15596 4178700
It would not perchance be BZ#540330 you're hitting would it?
Worth checking what the size is of libvirtd periodically perhaps, if nothing else, to rule out that it's this BZ.
Just my ¢2 :)
/Anders
On Mon, 30 Nov 2009 21:23:18 +0100, Anders Rayner-Karlsson wrote:
It would not perchance be BZ#540330 you're hitting would it?
Worth checking what the size is of libvirtd periodically perhaps, if nothing else, to rule out that it's this BZ. Just my ¢2 :) /Anders
I think it is exactly it... ;-( On my server I have now PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2904 root 19 -1 10.6g 9.8g 2712 S 0.0 83.6 16:12.42 libvirtd
[root@virtfed ~]# ps -eo rss,pid,cmd|egrep libvirtd|egrep -v egrep 10309532 2904 libvirtd --daemon --listen
Based on bugzilla info, in the mean time I restarted libvirtd. I'm going to follow the bug and add my data to it...
Thanks for the pointer! Gianluca
-
Anders Rayner-Karlsson -
Avi Kivity -
Gianluca Cecchi