4:20 a.m.
Having F11 + fedora-virt-preview repo, with some Linux guests running
without problems.
Then updated to F12
After some pains with preupgrade and upgrade, finally I was on F12
Actually I probably have to prune the duplicates because:
[root@ ~]# rpm -qa libvirt\* qemu\* virt-manager\*
qemu-common-0.11.0-11.fc11.x86_64
virt-manager-0.8.0-7.fc12.noarch
libvirt-client-0.7.1-15.fc12.x86_64
qemu-common-0.11.0-11.fc12.x86_64
qemu-img-0.11.0-11.fc11.x86_64
qemu-img-0.11.0-11.fc12.x86_64
qemu-kvm-0.11.0-11.fc12.x86_64
libvirt-python-0.7.1-15.fc12.x86_64
qemu-system-x86-0.11.0-11.fc12.x86_64
libvirt-0.7.1-15.fc12.x86_64
(in the mean time I disabled fedora-virt-preview repo)
So on 18th of November I booted the server with
Nov 18 17:30:12 virtfed kernel: Linux version 2.6.31.5-127.fc12.x86_64
(mockbuild(a)x86-4.fedora.phx.redhat.com) (gcc version 4.4.2 20091027
(Red Hat 4.4.2-7) (GCC) ) #1 SMP Sat Nov 7 21:11:14 EST 2009
.
I found this trace at boot:
Nov 18 17:30:12 virtfed kernel: ------------[ cut here ]------------
Nov 18 17:30:12 virtfed kernel: WARNING: at
arch/x86/kernel/cpu/mtrr/generic.c:456 generic_get_mtrr+0xcc/0x10a()
(Not tainted)
Nov 18 17:30:12 virtfed kernel: Hardware name: ProLiant BL480c G1
Nov 18 17:30:12 virtfed kernel: mtrr: your BIOS has set up an
incorrect mask, fixing it up.
Nov 18 17:30:12 virtfed kernel: Modules linked in: radeon(+) ttm
drm_kms_helper drm i2c_algo_bit i2c_core
Nov 18 17:30:12 virtfed kernel: Pid: 183, comm: work_for_cpu Not
tainted 2.6.31.5-127.fc12.x86_64 #1
Nov 18 17:30:12 virtfed kernel: Call Trace:
Nov 18 17:30:12 virtfed kernel: [<ffffffff81051694>]
warn_slowpath_common+0x84/0x9c
Nov 18 17:30:12 virtfed kernel: [<ffffffff81051703>] warn_slowpath_fmt+0x41/0x43
Nov 18 17:30:12 virtfed kernel: [<ffffffff812aab35>] ? bus_find_device+0x88/0x98
Nov 18 17:30:12 virtfed kernel: [<ffffffff810227a4>] generic_get_mtrr+0xcc/0x10a
Nov 18 17:30:12 virtfed kernel: [<ffffffff8102170e>] mtrr_add_page+0x16c/0x34a
Nov 18 17:30:12 virtfed kernel: [<ffffffff81021934>] mtrr_add+0x48/0x54
Nov 18 17:30:12 virtfed kernel: [<ffffffffa007782a>]
radeon_object_init+0x2e/0x87 [radeon]
Nov 18 17:30:12 virtfed kernel: [<ffffffffa0089b3b>]
r100_init+0x19d/0x23c [radeon]
Nov 18 17:30:12 virtfed kernel: [<ffffffff8129e035>] ?
vga_client_register+0x72/0x7d
Nov 18 17:30:12 virtfed kernel: [<ffffffffa006c107>]
radeon_device_init+0x203/0x27d [radeon]
Nov 18 17:30:12 virtfed kernel: [<ffffffffa006cb35>]
radeon_driver_load_kms+0xff/0x13a [radeon]
Nov 18 17:30:12 virtfed kernel: [<ffffffffa00183ba>]
drm_get_dev+0x36e/0x46f [drm]
Nov 18 17:30:12 virtfed kernel: [<ffffffff81063af6>] ? do_work_for_cpu+0x0/0x2a
Nov 18 17:30:12 virtfed kernel: [<ffffffffa009f670>]
radeon_pci_probe+0x15/0x269 [radeon]
Nov 18 17:30:12 virtfed kernel: [<ffffffff812101e1>] local_pci_probe+0x17/0x1b
Nov 18 17:30:12 virtfed kernel: [<ffffffff81063b0e>] do_work_for_cpu+0x18/0x2a
Nov 18 17:30:12 virtfed kernel: [<ffffffff81067765>] kthread+0x91/0x99
Nov 18 17:30:12 virtfed kernel: [<ffffffff81012daa>] child_rip+0xa/0x20
Nov 18 17:30:12 virtfed kernel: [<ffffffff810676d4>] ? kthread+0x0/0x99
Nov 18 17:30:12 virtfed kernel: [<ffffffff81012da0>] ? child_rip+0x0/0x20
Nov 18 17:30:12 virtfed kernel: ---[ end trace 5173c14fe23b4391 ]---
the system is a standalone node of a cluster (based on cman/rgmanager
as provided in F11, now F12)
I started two CentOS 5.4 vms and started ksm and ksmtuned
[root@virtfed ~]# cat /sys/kernel/mm/ksm/pages_shared
2000
This morning I found the two guests in shutdown mode and these messages:
Nov 20 08:22:06 virtfed kernel: script.sh invoked oom-killer:
gfp_mask=0x201da, order=0, oomkilladj=0
Nov 20 08:22:06 virtfed kernel: script.sh cpuset=/ mems_allowed=0
Nov 20 08:22:06 virtfed kernel: Pid: 14360, comm: script.sh Tainted: G
W 2.6.31.5-127.fc12.x86_64 #1
Nov 20 08:22:06 virtfed kernel: Call Trace:
Nov 20 08:22:06 virtfed kernel: [<ffffffff8108a7d7>] ?
cpuset_print_task_mems_allowed+0x91/0x9d
Nov 20 08:22:06 virtfed kernel: [<ffffffff810c38e9>] oom_kill_process+0x98/0x256
Nov 20 08:22:06 virtfed kernel: [<ffffffff810c3d6a>] ?
select_bad_process+0xa3/0x102
Nov 20 08:22:06 virtfed kernel: [<ffffffff810c3e53>] __out_of_memory+0x8a/0x99
Nov 20 08:22:06 virtfed kernel: [<ffffffff810c3fc5>] out_of_memory+0x163/0x195
Nov 20 08:22:06 virtfed kernel: [<ffffffff810c75c1>]
__alloc_pages_nodemask+0x491/0x584
Nov 20 08:22:06 virtfed kernel: [<ffffffff810eb1c4>]
alloc_pages_current+0x95/0x9e
Nov 20 08:22:06 virtfed kernel: [<ffffffff810c15c1>]
__page_cache_alloc+0x5f/0x61
Nov 20 08:22:06 virtfed kernel: [<ffffffff810c9235>]
__do_page_cache_readahead+0x98/0x176
Nov 20 08:22:06 virtfed kernel: [<ffffffff810c9334>] ra_submit+0x21/0x25
Nov 20 08:22:06 virtfed kernel: [<ffffffff810c1abd>] filemap_fault+0x193/0x317
Nov 20 08:22:06 virtfed kernel: [<ffffffff810d6641>] __do_fault+0x54/0x3c4
Nov 20 08:22:06 virtfed kernel: [<ffffffff810d881a>] handle_mm_fault+0x2f6/0x705
Nov 20 08:22:06 virtfed kernel: [<ffffffff810f185d>] ?
virt_to_head_page+0xe/0x2f
Nov 20 08:22:06 virtfed kernel: [<ffffffff811012ec>] ? free_bprm+0x44/0x49
Nov 20 08:22:06 virtfed kernel: [<ffffffff8141cfe1>] do_page_fault+0x281/0x299
Nov 20 08:22:06 virtfed kernel: [<ffffffff8141af75>] page_fault+0x25/0x30
Nov 20 08:22:06 virtfed kernel: Mem-Info:
Nov 20 08:22:06 virtfed kernel: Node 0 DMA per-cpu:
Nov 20 08:22:06 virtfed kernel: CPU 0: hi: 0, btch: 1 usd: 0
Nov 20 08:22:06 virtfed kernel: CPU 1: hi: 0, btch: 1 usd: 0
Nov 20 08:22:06 virtfed kernel: CPU 2: hi: 0, btch: 1 usd: 0
Nov 20 08:22:06 virtfed kernel: CPU 3: hi: 0, btch: 1 usd: 0
Nov 20 08:22:06 virtfed kernel: CPU 4: hi: 0, btch: 1 usd: 0
Nov 20 08:22:06 virtfed kernel: CPU 5: hi: 0, btch: 1 usd: 0
Nov 20 08:22:06 virtfed kernel: CPU 6: hi: 0, btch: 1 usd: 0
Nov 20 08:22:06 virtfed kernel: CPU 7: hi: 0, btch: 1 usd: 0
Nov 20 08:22:06 virtfed kernel: Node 0 DMA32 per-cpu:
Nov 20 08:22:06 virtfed kernel: CPU 0: hi: 186, btch: 31 usd: 164
Nov 20 08:22:06 virtfed kernel: CPU 1: hi: 186, btch: 31 usd: 23
Nov 20 08:22:06 virtfed kernel: CPU 2: hi: 186, btch: 31 usd: 100
Nov 20 08:22:06 virtfed kernel: CPU 3: hi: 186, btch: 31 usd: 0
Nov 20 08:22:06 virtfed kernel: CPU 4: hi: 186, btch: 31 usd: 47
Nov 20 08:22:06 virtfed kernel: CPU 5: hi: 186, btch: 31 usd: 51
Nov 20 08:22:06 virtfed kernel: CPU 6: hi: 186, btch: 31 usd: 181
Nov 20 08:22:06 virtfed kernel: CPU 7: hi: 186, btch: 31 usd: 48
Nov 20 08:22:06 virtfed kernel: Node 0 Normal per-cpu:
Nov 20 08:22:06 virtfed kernel: CPU 0: hi: 186, btch: 31 usd: 160
Nov 20 08:22:06 virtfed kernel: CPU 1: hi: 186, btch: 31 usd: 5
Nov 20 08:22:06 virtfed kernel: CPU 2: hi: 186, btch: 31 usd: 181
Nov 20 08:22:06 virtfed kernel: CPU 3: hi: 186, btch: 31 usd: 0
Nov 20 08:22:06 virtfed kernel: CPU 4: hi: 186, btch: 31 usd: 73
Nov 20 08:22:06 virtfed kernel: CPU 5: hi: 186, btch: 31 usd: 38
Nov 20 08:22:06 virtfed kernel: CPU 6: hi: 186, btch: 31 usd: 156
Nov 20 08:22:06 virtfed kernel: CPU 7: hi: 186, btch: 31 usd: 31
Nov 20 08:22:06 virtfed kernel: Active_anon:2563037 active_file:0
inactive_anon:368313
Nov 20 08:22:06 virtfed kernel: inactive_file:31 unevictable:12620
dirty:0 writeback:0 unstable:0
Nov 20 08:22:06 virtfed kernel: free:16269 slab:25512 mapped:7756
pagetables:10201 bounce:0
Nov 20 08:22:06 virtfed kernel: Node 0 DMA free:15824kB min:16kB
low:20kB high:24kB active_anon:0kB inactive_anon:0kB active_file:0kB
inactive_file:0kB unevictable:0kB present:15320kB pages_scanned:0
all_unreclaimable? yes
Nov 20 08:22:06 virtfed kernel: lowmem_reserve[]: 0 3254 12092 12092
Nov 20 08:22:06 virtfed kernel: Node 0 DMA32 free:39040kB min:3784kB
low:4728kB high:5676kB active_anon:2557420kB inactive_anon:511296kB
active_file:0kB inactive_file:0kB unevictable:804kB present:3332660kB
pages_scanned:0 all_unreclaimable? no
Nov 20 08:22:06 virtfed kernel: lowmem_reserve[]: 0 0 8837 8837
Nov 20 08:22:06 virtfed kernel: Node 0 Normal free:10212kB min:10280kB
low:12848kB high:15420kB active_anon:7694728kB inactive_anon:961956kB
active_file:0kB inactive_file:236kB unevictable:49676kB
present:9049596kB pages_scanned:0 all_unreclaimable? no
Nov 20 08:22:06 virtfed kernel: lowmem_reserve[]: 0 0 0 0
Nov 20 08:22:06 virtfed kernel: Node 0 DMA: 2*4kB 1*8kB 2*16kB 1*32kB
2*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15824kB
Nov 20 08:22:06 virtfed kernel: Node 0 DMA32: 1068*4kB 1263*8kB
511*16kB 161*32kB 25*64kB 2*128kB 1*256kB 6*512kB 0*1024kB 1*2048kB
1*4096kB = 39032kB
Nov 20 08:22:06 virtfed kernel: Node 0 Normal: 1595*4kB 0*8kB 1*16kB
1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB =
10460kB
Nov 20 08:22:06 virtfed kernel: 25465 total pagecache pages
Nov 20 08:22:06 virtfed kernel: 17663 pages in swap cache
Nov 20 08:22:06 virtfed kernel: Swap cache stats: add 1210138, delete
1192475, find 119729/133852
Nov 20 08:22:06 virtfed kernel: Free swap = 0kB
Nov 20 08:22:06 virtfed kernel: Total swap = 4194296kB
Nov 20 08:22:06 virtfed kernel: 3145727 pages RAM
Nov 20 08:22:06 virtfed kernel: 63020 pages reserved
Nov 20 08:22:06 virtfed kernel: 32665 pages shared
Nov 20 08:22:06 virtfed kernel: 3054335 pages non-shared
Nov 20 08:22:06 virtfed kernel: Out of memory: kill process 14357
(qemu-kvm) score 994666 or a child
Nov 20 08:22:06 virtfed kernel: Killed process 14357 (qemu-kvm)
Nov 20 08:22:06 virtfed kernel: virbr0: port 2(vnet1) entering disabled state
Nov 20 08:22:06 virtfed kernel: device vnet1 left promiscuous mode
Nov 20 08:22:06 virtfed kernel: virbr0: port 2(vnet1) entering disabled state
and then similar about 1h 20minutes after for the other one....
Now I have:
[root@virtfed ~]# free
total used free shared buffers cached
Mem: 12330828 1866848 10463980 0 546088 158820
-/+ buffers/cache: 1161940 11168888
Swap: 4194296 30024 4164272
and
[root@ ~]# cat /sys/kernel/mm/ksm/pages_shared
0
It seems script.sh is the one provided by cluster manager but
following messages were about other processed
Nov 20 09:45:25 virtfed kernel: ksmd invoked oom-killer:
gfp_mask=0x200d2, order=0, oomkilladj=0
Nov 20 09:45:25 virtfed kernel: ksmd cpuset=/ mems_allowed=0
Nov 20 10:58:35 virtfed kernel: awk invoked oom-killer:
gfp_mask=0x201da, order=0, oomkilladj=0
Nov 20 10:58:35 virtfed kernel: awk cpuset=/ mems_allowed=0
Nov 20 10:58:35 virtfed kernel: libvirtd invoked oom-killer:
gfp_mask=0x201da, order=0, oomkilladj=0
Nov 20 10:58:35 virtfed kernel: libvirtd cpuset=/ mems_allowed=0
/etc/sysconfig/ksm is empty (apart comments)
at this moment:
[root@virtfed log]# service ksm status
ksm is not running
[root@virtfed log]# service ksmtuned status
ksmtuned (pid 8672) is running...
Any hints?
Thanks,
Gianluca