Having F11 + fedora-virt-preview repo, with some Linux guests running without problems. Then updated to F12 After some pains with preupgrade and upgrade, finally I was on F12 Actually I probably have to prune the duplicates because:
[root@ ~]# rpm -qa libvirt* qemu* virt-manager* qemu-common-0.11.0-11.fc11.x86_64 virt-manager-0.8.0-7.fc12.noarch libvirt-client-0.7.1-15.fc12.x86_64 qemu-common-0.11.0-11.fc12.x86_64 qemu-img-0.11.0-11.fc11.x86_64 qemu-img-0.11.0-11.fc12.x86_64 qemu-kvm-0.11.0-11.fc12.x86_64 libvirt-python-0.7.1-15.fc12.x86_64 qemu-system-x86-0.11.0-11.fc12.x86_64 libvirt-0.7.1-15.fc12.x86_64
(in the mean time I disabled fedora-virt-preview repo) So on 18th of November I booted the server with Nov 18 17:30:12 virtfed kernel: Linux version 2.6.31.5-127.fc12.x86_64 (mockbuild@x86-4.fedora.phx.redhat.com) (gcc version 4.4.2 20091027 (Red Hat 4.4.2-7) (GCC) ) #1 SMP Sat Nov 7 21:11:14 EST 2009 . I found this trace at boot: Nov 18 17:30:12 virtfed kernel: ------------[ cut here ]------------ Nov 18 17:30:12 virtfed kernel: WARNING: at arch/x86/kernel/cpu/mtrr/generic.c:456 generic_get_mtrr+0xcc/0x10a() (Not tainted) Nov 18 17:30:12 virtfed kernel: Hardware name: ProLiant BL480c G1 Nov 18 17:30:12 virtfed kernel: mtrr: your BIOS has set up an incorrect mask, fixing it up. Nov 18 17:30:12 virtfed kernel: Modules linked in: radeon(+) ttm drm_kms_helper drm i2c_algo_bit i2c_core Nov 18 17:30:12 virtfed kernel: Pid: 183, comm: work_for_cpu Not tainted 2.6.31.5-127.fc12.x86_64 #1 Nov 18 17:30:12 virtfed kernel: Call Trace: Nov 18 17:30:12 virtfed kernel: [<ffffffff81051694>] warn_slowpath_common+0x84/0x9c Nov 18 17:30:12 virtfed kernel: [<ffffffff81051703>] warn_slowpath_fmt+0x41/0x43 Nov 18 17:30:12 virtfed kernel: [<ffffffff812aab35>] ? bus_find_device+0x88/0x98 Nov 18 17:30:12 virtfed kernel: [<ffffffff810227a4>] generic_get_mtrr+0xcc/0x10a Nov 18 17:30:12 virtfed kernel: [<ffffffff8102170e>] mtrr_add_page+0x16c/0x34a Nov 18 17:30:12 virtfed kernel: [<ffffffff81021934>] mtrr_add+0x48/0x54 Nov 18 17:30:12 virtfed kernel: [<ffffffffa007782a>] radeon_object_init+0x2e/0x87 [radeon] Nov 18 17:30:12 virtfed kernel: [<ffffffffa0089b3b>] r100_init+0x19d/0x23c [radeon] Nov 18 17:30:12 virtfed kernel: [<ffffffff8129e035>] ? vga_client_register+0x72/0x7d Nov 18 17:30:12 virtfed kernel: [<ffffffffa006c107>] radeon_device_init+0x203/0x27d [radeon] Nov 18 17:30:12 virtfed kernel: [<ffffffffa006cb35>] radeon_driver_load_kms+0xff/0x13a [radeon] Nov 18 17:30:12 virtfed kernel: [<ffffffffa00183ba>] drm_get_dev+0x36e/0x46f [drm] Nov 18 17:30:12 virtfed kernel: [<ffffffff81063af6>] ? do_work_for_cpu+0x0/0x2a Nov 18 17:30:12 virtfed kernel: [<ffffffffa009f670>] radeon_pci_probe+0x15/0x269 [radeon] Nov 18 17:30:12 virtfed kernel: [<ffffffff812101e1>] local_pci_probe+0x17/0x1b Nov 18 17:30:12 virtfed kernel: [<ffffffff81063b0e>] do_work_for_cpu+0x18/0x2a Nov 18 17:30:12 virtfed kernel: [<ffffffff81067765>] kthread+0x91/0x99 Nov 18 17:30:12 virtfed kernel: [<ffffffff81012daa>] child_rip+0xa/0x20 Nov 18 17:30:12 virtfed kernel: [<ffffffff810676d4>] ? kthread+0x0/0x99 Nov 18 17:30:12 virtfed kernel: [<ffffffff81012da0>] ? child_rip+0x0/0x20 Nov 18 17:30:12 virtfed kernel: ---[ end trace 5173c14fe23b4391 ]---
the system is a standalone node of a cluster (based on cman/rgmanager as provided in F11, now F12)
I started two CentOS 5.4 vms and started ksm and ksmtuned [root@virtfed ~]# cat /sys/kernel/mm/ksm/pages_shared 2000 This morning I found the two guests in shutdown mode and these messages:
Nov 20 08:22:06 virtfed kernel: script.sh invoked oom-killer: gfp_mask=0x201da, order=0, oomkilladj=0 Nov 20 08:22:06 virtfed kernel: script.sh cpuset=/ mems_allowed=0 Nov 20 08:22:06 virtfed kernel: Pid: 14360, comm: script.sh Tainted: G W 2.6.31.5-127.fc12.x86_64 #1 Nov 20 08:22:06 virtfed kernel: Call Trace: Nov 20 08:22:06 virtfed kernel: [<ffffffff8108a7d7>] ? cpuset_print_task_mems_allowed+0x91/0x9d Nov 20 08:22:06 virtfed kernel: [<ffffffff810c38e9>] oom_kill_process+0x98/0x256 Nov 20 08:22:06 virtfed kernel: [<ffffffff810c3d6a>] ? select_bad_process+0xa3/0x102 Nov 20 08:22:06 virtfed kernel: [<ffffffff810c3e53>] __out_of_memory+0x8a/0x99 Nov 20 08:22:06 virtfed kernel: [<ffffffff810c3fc5>] out_of_memory+0x163/0x195 Nov 20 08:22:06 virtfed kernel: [<ffffffff810c75c1>] __alloc_pages_nodemask+0x491/0x584 Nov 20 08:22:06 virtfed kernel: [<ffffffff810eb1c4>] alloc_pages_current+0x95/0x9e Nov 20 08:22:06 virtfed kernel: [<ffffffff810c15c1>] __page_cache_alloc+0x5f/0x61 Nov 20 08:22:06 virtfed kernel: [<ffffffff810c9235>] __do_page_cache_readahead+0x98/0x176 Nov 20 08:22:06 virtfed kernel: [<ffffffff810c9334>] ra_submit+0x21/0x25 Nov 20 08:22:06 virtfed kernel: [<ffffffff810c1abd>] filemap_fault+0x193/0x317 Nov 20 08:22:06 virtfed kernel: [<ffffffff810d6641>] __do_fault+0x54/0x3c4 Nov 20 08:22:06 virtfed kernel: [<ffffffff810d881a>] handle_mm_fault+0x2f6/0x705 Nov 20 08:22:06 virtfed kernel: [<ffffffff810f185d>] ? virt_to_head_page+0xe/0x2f Nov 20 08:22:06 virtfed kernel: [<ffffffff811012ec>] ? free_bprm+0x44/0x49 Nov 20 08:22:06 virtfed kernel: [<ffffffff8141cfe1>] do_page_fault+0x281/0x299 Nov 20 08:22:06 virtfed kernel: [<ffffffff8141af75>] page_fault+0x25/0x30 Nov 20 08:22:06 virtfed kernel: Mem-Info: Nov 20 08:22:06 virtfed kernel: Node 0 DMA per-cpu: Nov 20 08:22:06 virtfed kernel: CPU 0: hi: 0, btch: 1 usd: 0 Nov 20 08:22:06 virtfed kernel: CPU 1: hi: 0, btch: 1 usd: 0 Nov 20 08:22:06 virtfed kernel: CPU 2: hi: 0, btch: 1 usd: 0 Nov 20 08:22:06 virtfed kernel: CPU 3: hi: 0, btch: 1 usd: 0 Nov 20 08:22:06 virtfed kernel: CPU 4: hi: 0, btch: 1 usd: 0 Nov 20 08:22:06 virtfed kernel: CPU 5: hi: 0, btch: 1 usd: 0 Nov 20 08:22:06 virtfed kernel: CPU 6: hi: 0, btch: 1 usd: 0 Nov 20 08:22:06 virtfed kernel: CPU 7: hi: 0, btch: 1 usd: 0 Nov 20 08:22:06 virtfed kernel: Node 0 DMA32 per-cpu: Nov 20 08:22:06 virtfed kernel: CPU 0: hi: 186, btch: 31 usd: 164 Nov 20 08:22:06 virtfed kernel: CPU 1: hi: 186, btch: 31 usd: 23 Nov 20 08:22:06 virtfed kernel: CPU 2: hi: 186, btch: 31 usd: 100 Nov 20 08:22:06 virtfed kernel: CPU 3: hi: 186, btch: 31 usd: 0 Nov 20 08:22:06 virtfed kernel: CPU 4: hi: 186, btch: 31 usd: 47 Nov 20 08:22:06 virtfed kernel: CPU 5: hi: 186, btch: 31 usd: 51 Nov 20 08:22:06 virtfed kernel: CPU 6: hi: 186, btch: 31 usd: 181 Nov 20 08:22:06 virtfed kernel: CPU 7: hi: 186, btch: 31 usd: 48 Nov 20 08:22:06 virtfed kernel: Node 0 Normal per-cpu: Nov 20 08:22:06 virtfed kernel: CPU 0: hi: 186, btch: 31 usd: 160 Nov 20 08:22:06 virtfed kernel: CPU 1: hi: 186, btch: 31 usd: 5 Nov 20 08:22:06 virtfed kernel: CPU 2: hi: 186, btch: 31 usd: 181 Nov 20 08:22:06 virtfed kernel: CPU 3: hi: 186, btch: 31 usd: 0 Nov 20 08:22:06 virtfed kernel: CPU 4: hi: 186, btch: 31 usd: 73 Nov 20 08:22:06 virtfed kernel: CPU 5: hi: 186, btch: 31 usd: 38 Nov 20 08:22:06 virtfed kernel: CPU 6: hi: 186, btch: 31 usd: 156 Nov 20 08:22:06 virtfed kernel: CPU 7: hi: 186, btch: 31 usd: 31 Nov 20 08:22:06 virtfed kernel: Active_anon:2563037 active_file:0 inactive_anon:368313 Nov 20 08:22:06 virtfed kernel: inactive_file:31 unevictable:12620 dirty:0 writeback:0 unstable:0 Nov 20 08:22:06 virtfed kernel: free:16269 slab:25512 mapped:7756 pagetables:10201 bounce:0 Nov 20 08:22:06 virtfed kernel: Node 0 DMA free:15824kB min:16kB low:20kB high:24kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB present:15320kB pages_scanned:0 all_unreclaimable? yes Nov 20 08:22:06 virtfed kernel: lowmem_reserve[]: 0 3254 12092 12092 Nov 20 08:22:06 virtfed kernel: Node 0 DMA32 free:39040kB min:3784kB low:4728kB high:5676kB active_anon:2557420kB inactive_anon:511296kB active_file:0kB inactive_file:0kB unevictable:804kB present:3332660kB pages_scanned:0 all_unreclaimable? no Nov 20 08:22:06 virtfed kernel: lowmem_reserve[]: 0 0 8837 8837 Nov 20 08:22:06 virtfed kernel: Node 0 Normal free:10212kB min:10280kB low:12848kB high:15420kB active_anon:7694728kB inactive_anon:961956kB active_file:0kB inactive_file:236kB unevictable:49676kB present:9049596kB pages_scanned:0 all_unreclaimable? no Nov 20 08:22:06 virtfed kernel: lowmem_reserve[]: 0 0 0 0 Nov 20 08:22:06 virtfed kernel: Node 0 DMA: 2*4kB 1*8kB 2*16kB 1*32kB 2*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15824kB Nov 20 08:22:06 virtfed kernel: Node 0 DMA32: 1068*4kB 1263*8kB 511*16kB 161*32kB 25*64kB 2*128kB 1*256kB 6*512kB 0*1024kB 1*2048kB 1*4096kB = 39032kB Nov 20 08:22:06 virtfed kernel: Node 0 Normal: 1595*4kB 0*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 10460kB Nov 20 08:22:06 virtfed kernel: 25465 total pagecache pages Nov 20 08:22:06 virtfed kernel: 17663 pages in swap cache Nov 20 08:22:06 virtfed kernel: Swap cache stats: add 1210138, delete 1192475, find 119729/133852 Nov 20 08:22:06 virtfed kernel: Free swap = 0kB Nov 20 08:22:06 virtfed kernel: Total swap = 4194296kB Nov 20 08:22:06 virtfed kernel: 3145727 pages RAM Nov 20 08:22:06 virtfed kernel: 63020 pages reserved Nov 20 08:22:06 virtfed kernel: 32665 pages shared Nov 20 08:22:06 virtfed kernel: 3054335 pages non-shared Nov 20 08:22:06 virtfed kernel: Out of memory: kill process 14357 (qemu-kvm) score 994666 or a child Nov 20 08:22:06 virtfed kernel: Killed process 14357 (qemu-kvm) Nov 20 08:22:06 virtfed kernel: virbr0: port 2(vnet1) entering disabled state Nov 20 08:22:06 virtfed kernel: device vnet1 left promiscuous mode Nov 20 08:22:06 virtfed kernel: virbr0: port 2(vnet1) entering disabled state
and then similar about 1h 20minutes after for the other one....
Now I have: [root@virtfed ~]# free total used free shared buffers cached Mem: 12330828 1866848 10463980 0 546088 158820 -/+ buffers/cache: 1161940 11168888 Swap: 4194296 30024 4164272
and [root@ ~]# cat /sys/kernel/mm/ksm/pages_shared 0
It seems script.sh is the one provided by cluster manager but following messages were about other processed Nov 20 09:45:25 virtfed kernel: ksmd invoked oom-killer: gfp_mask=0x200d2, order=0, oomkilladj=0 Nov 20 09:45:25 virtfed kernel: ksmd cpuset=/ mems_allowed=0
Nov 20 10:58:35 virtfed kernel: awk invoked oom-killer: gfp_mask=0x201da, order=0, oomkilladj=0 Nov 20 10:58:35 virtfed kernel: awk cpuset=/ mems_allowed=0
Nov 20 10:58:35 virtfed kernel: libvirtd invoked oom-killer: gfp_mask=0x201da, order=0, oomkilladj=0 Nov 20 10:58:35 virtfed kernel: libvirtd cpuset=/ mems_allowed=0
/etc/sysconfig/ksm is empty (apart comments) at this moment: [root@virtfed log]# service ksm status ksm is not running
[root@virtfed log]# service ksmtuned status ksmtuned (pid 8672) is running...
Any hints? Thanks, Gianluca
-
Gianluca Cecchi