11:19 p.m.
Hi,
I've posted this same problem on the fedora-xen list, and the fedora
forums. Sorry to anybody who is getting duplicates.
Additional log info is available at
http://forums.fedoraforum.org/showthread.php?p=1149972&posted=1#post1149
972
It is also formatted a lot better and may be easier to follow.
------------------------------------------------------------------
I have two machines running fresh installs of f8 with the xen. Kernel
and all software versions are the same on both.
Specifically:
[root@machineA boot]# uname -a
Linux machineA 2.6.21.7-5.fc8xen #1 SMP Thu Aug 7 12:44:22 EDT 2008
x86_64 x86_64 x86_64 GNU/Linux
[root@machineA boot]# virsh version
Compiled against library: libvir 0.4.4
Using library: libvir 0.4.4
Using API: Xen 3.0.1
Running hypervisor: Xen 3.1.0
------------------------------------------------------------------
And:
------------------------------------------------------------------
[root@machineB ~]# uname -a
Linux machineB 2.6.21.7-5.fc8xen #1 SMP Thu Aug 7 12:44:22 EDT 2008
x86_64 x86_64 x86_64 GNU/Linux
[root@machineB ~]# virsh version
Compiled against library: libvir 0.4.4
Using library: libvir 0.4.4
Using API: Xen 3.0.1
Running hypervisor: Xen 3.1.0
MachineA has two AMD Opteron 275s. MachineB has four Intel(R) Xeon(TM)
CPU 2.80GHz processors.
Both machines are as up to date as possible.
I can boot or create x86_64 f10 guests on MachineA with no trouble
whatsoever.
MachineB will not boot/create x86_64 f10 guests.
The configuration files are created in the same manner, but as soon as
Xen tries to unpause the newly created domain, it crashes pretty much
instantly.
------------------------------------------------------------------
/var/log/xen/xend.log relevant output:
[2009-01-16 14:45:32 4120] DEBUG (DevController:150) Waiting for devices
vtpm.
[2009-01-16 14:45:32 4120] INFO (XendDomain:1130) Domain f10testB (21)
unpaused.
[2009-01-16 14:45:32 4120] WARNING (XendDomainInfo:1203) Domain has
crashed: name=f10testB id=21.
[2009-01-16 14:45:32 4120] DEBUG (XendDomainInfo:1802)
XendDomainInfo.destroy: domid=21
[2009-01-16 14:45:32 4120] DEBUG (XendDomainInfo:1821)
XendDomainInfo.destroyDomain(21)
------------------------------------------------------------------
I've also tried moving a functional guest from MachineA to MachineB to
boot it there, with the same results. Guest will not boot on MachineB.
f8 64bit guests will boot on MachineB with no problems.
f10 32bit guests will boot on MachineB with no problems.
Only 64bit machines seem to be borked.
Mark on the fedora-xen list suggested running xenctx on the crashed
domain. Output is as follows:
------------------------------------------------------------------
xenctx output:
/usr/lib64/xen/bin/xenctx -s System.map-2.6.27.5-117.fc10.x86_64 46
rip: ffffffff8100b8a2 set_page_prot+0x6d
rsp: ffffffff81573f08
rax: ffffffea rbx: 000016e1 rcx: 00000055 rdx: 00000000
rsi: 800000014ffc6061 rdi: ffffffff816e1000 rbp: ffffffff81573f68
r8: 0000000f r9: ffffffff817eb450 r10: ffffffff817eb650 r11:
00000010
r12: ffffffff816e1000 r13: 800000014ffc6061 r14: 8000000000000161
r15: 00000016
cs: 0000e033 ds: 00000000 fs: 00000000 gs: 00000000
Stack:
0000000000000055 0000000000000010 ffffffff8100b8a2 000000010000e030
0000000000010082 ffffffff81573f48 000000000000e02b ffffffff8100b89e
0000000000000200 ffffffff816e4000 0000000000000800 0000000000002c00
ffffffff81573ff8 ffffffff815a3c60 0000000000002c00 0000000000000000
Code:
7b 4a 1d 00 4c 89 e7 4c 89 ee 31 d2 e8 22 d9 ff ff 85 c0 74 04 <0f> 0b
eb fe 5b 41 5c 41 5d 41 5e
Call Trace:
[<ffffffff8100b8a2>] set_page_prot+0x6d <--
[<ffffffff8100b8a2>] set_page_prot+0x6d
[<ffffffff8100b89e>] set_page_prot+0x69
[<ffffffff815a3c60>] xen_start_kernel+0x5dd
------------------------------------------------------------------
I also finally figured out you can look at the Xen dmesg, which includes
the following line:
(XEN) traps.c:405:d44 Unhandled invalid opcode fault/trap [#6] in domain
46 on VCPU 0 [ec=0000]
The domain does install so the following bug does not seem to be the
cause of the current issues:
http://fedoraproject.org/wiki/Bugs/F10Common#Installing_Fedora_10_DomU_o
n_Fedora_8_Dom0_Fails
Any information / help / insight as to why this is happening would be
very much appreciated. The machines are pretty similar, and since the
guests are paravirtualized it does not really make sense for the
processors to be the cause of the problem.
Thanks,
jon
4:27 a.m.
New subject: f10 x86_64 xen VM guests fail to boot on f8 host (guest setting NX bit in L1 PTE?)
(Jeremy/Ian - here's some more info on the bug reported here:
http://lists.xensource.com/archives/html/xen-devel/2009-01/msg00176.html
)
Hi Jon/Phill,
Thanks for all the info.
Here's the important bits:
1) Host kernel is 2.6.21.7-5.fc8xen, that means the hypervisor is
xen-3.1.4
2) The guest kernel is 2.6.27.5-117.fc10.x86_64
3) Phill points out the faulting instruction is UD2. That just means
the guest kernel is hitting a BUG() assertion. See /asm-x86/bug.h:
#define BUG() \
do { \
asm volatile("ud2"); \
for (;;) ; \
} while (0)
4) The backtrace shows the fault happens in set_page_prot()
5) Jon's dmesg contains:
(XEN) mm.c:1362:d46 Bad L1 flags 800000
That means the guest is faulting here:
static void set_page_prot(void *addr, pgprot_t prot)
{
....
if (HYPERVISOR_update_va_mapping((unsigned long)addr, pte, 0))
BUG();
}
because the PTE update is failing in the HV here:
static int mod_l1_entry(l1_pgentry_t *pl1e, l1_pgentry_t nl1e,
unsigned long gl1mfn)
{
...
if ( unlikely(l1e_get_flags(nl1e) & L1_DISALLOW_MASK) )
{
MEM_LOG("Bad L1 flags %x",
l1e_get_flags(nl1e) & L1_DISALLOW_MASK);
return 0;
}
...
}
the PTE flags are 800000 which corresponds to:
#define _PAGE_NX_BIT (1U<<23)
Jon/Phill - can one of you two file a bug (bugzilla.redhat.com) with all
this info?
Thanks,
Mark.
Previous posts, for reference:
http://www.redhat.com/archives/fedora-xen/2009-January/thread.html#00022
http://www.redhat.com/archives/fedora-virt/2009-January/thread.html#00013
9:55 a.m.
New subject: f10 x86_64 xen VM guests fail to boot on f8 host (guest setting NX bit in L1 PTE?)
On Tue, 2009-01-20 at 10:27 +0000, Mark McLoughlin wrote:
if ( unlikely(l1e_get_flags(nl1e) & L1_DISALLOW_MASK) )
{
MEM_LOG("Bad L1 flags %x",
l1e_get_flags(nl1e) & L1_DISALLOW_MASK);
return 0;
}
...
}
the PTE flags are 800000 which corresponds to:
#define _PAGE_NX_BIT (1U<<23)
At least in xen-unstable (and I think for much longer) L1_DISALLOW_MASK
contains _PAGE_NX_BIT dynamically depending on the processor
capabilities.
#define _PAGE_NX (cpu_has_nx ? _PAGE_NX_BIT : 0)
...
/*
* Disallow unused flag bits plus PAT/PSE, PCD, PWT and GLOBAL.
* Permit the NX bit if the hardware supports it.
*/
#define BASE_DISALLOW_MASK (0xFFFFF198U & ~_PAGE_NX)
#define L1_DISALLOW_MASK (BASE_DISALLOW_MASK | _PAGE_GNTTAB)
Does the hardware support NX? What does /proc/cpuinfo in dom0 think?
The guest kernel should be setting up __supported_pte_mask appropriately
to match the hardware and hence shouldn't be using NX if it isn't
available. There's a command line option to force NX, can you try
noexec=off on the guest command line.
My guess would be that the guest is getting a wrong EFER from
somewhere...
Ian.
11:03 a.m.
New subject: f10 x86_64 xen VM guests fail to boot on f8 host (guest setting NX bit in L1 PTE?)
(resending with original xen-devel thread participants on CC, please
reply to this subthread, I'll forward you guys Mark's original mail in a
second)
On Tue, 2009-01-20 at 10:27 +0000, Mark McLoughlin wrote:
if ( unlikely(l1e_get_flags(nl1e) & L1_DISALLOW_MASK) )
{
MEM_LOG("Bad L1 flags %x",
l1e_get_flags(nl1e) & L1_DISALLOW_MASK);
return 0;
}
...
}
the PTE flags are 800000 which corresponds to:
#define _PAGE_NX_BIT (1U<<23)
At least in xen-unstable (and I think for much longer) L1_DISALLOW_MASK
contains _PAGE_NX_BIT dynamically depending on the processor
capabilities.
#define _PAGE_NX (cpu_has_nx ? _PAGE_NX_BIT : 0)
...
/*
* Disallow unused flag bits plus PAT/PSE, PCD, PWT and GLOBAL.
* Permit the NX bit if the hardware supports it.
*/
#define BASE_DISALLOW_MASK (0xFFFFF198U & ~_PAGE_NX)
#define L1_DISALLOW_MASK (BASE_DISALLOW_MASK | _PAGE_GNTTAB)
Does the hardware support NX? What does /proc/cpuinfo in dom0 think?
The guest kernel should be setting up __supported_pte_mask appropriately
to match the hardware and hence shouldn't be using NX if it isn't
available. There's a command line option to force NX, can you try
noexec=off on the guest command line.
My guess would be that the guest is getting a wrong EFER from
somewhere...
Ian.
6:45 p.m.
New subject: f10 x86_64 xen VM guests fail to boot on f8 host (guest setting NX bit in L1 PTE?)
Hi Ian,
Indeed nx is on one and not the other! However, that doesn't help...
Broken CPU:
/proc/cpuinfo:
model name : Intel(R) Xeon(TM) CPU 3.00GHz
flags : fpu tsc msr pae mce cx8 apic mtrr mca cmov pat pse36 clflush dts acpi
mmx fxsr sse sse2 ss ht tm syscall lm constant_tsc up pni monitor ds_cpl cid cx16 xtpr
Good CPU:
model name : AMD Athlon(tm) 64 X2 Dual Core Processor 3600+
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36
clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow rep_good
pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy 3dnowprefetch
So, ran it again with noexec=0,
[ Minimal BASH-like line editing is supported.
ESC at any time cancels. ENTER at any time accepts your changes. ]
kernel /vmlinuz-2.6.27.9-159.fc10.x86_64 ro root=LABEL=/ selinux=0
noipv6 nomodeset noexec=off
Results:
[ root@office64 xen ]# /usr/lib64/xen/bin/xenctx -s System.map-2.6.27.9-159.fc10.x86_64
119
rip: ffffffff8100b8a2 set_page_prot+0x6d
rsp: ffffffff81575f08
rax: ffffffea rbx: 000016e4 rcx: 00000055 rdx: 00000000
rsi: 800000014a293061 rdi: ffffffff816e4000 rbp: ffffffff81575f68
r8: 0000000f r9: ffffffff817ee350 r10: ffffffff817ee550 r11: 00000010
r12: ffffffff816e4000 r13: 800000014a293061 r14: 8000000000000161 r15: 00002c00
cs: 0000e033 ds: 00000000 fs: 00000000 gs: 00000000
Stack:
0000000000000055 0000000000000010 ffffffff8100b8a2 000000010000e030
0000000000010082 ffffffff81575f48 000000000000e02b ffffffff8100b89e
0000000000000200 ffffffff816e7000 0000000000000800 0000000000000016
ffffffff81575ff8 ffffffff815a5c60 0000000000002c00 0000000000000000
Code:
df 54 1d 00 4c 89 e7 4c 89 ee 31 d2 e8 22 d9 ff ff 85 c0 74 04 <0f> 0b eb fe 5b 41
5c 41 5d 41 5e
Call Trace:
[<ffffffff8100b8a2>] set_page_prot+0x6d <--
[<ffffffff8100b8a2>] set_page_prot+0x6d
[<ffffffff8100b89e>] set_page_prot+0x69
[<ffffffff815a5c60>] xen_start_kernel+0x5dd
Battling with bugzilla trying to get a new account. It doesn't like me :-(
Might have to leave it up to Jon to do the bugzilla thing.
Cheers
Phill.
On Tue, 2009-01-20 at 17:03 +0000, Ian Campbell wrote:
(resending with original xen-devel thread participants on CC, please
reply to this subthread, I'll forward you guys Mark's original mail in a
second)
On Tue, 2009-01-20 at 10:27 +0000, Mark McLoughlin wrote:
> if ( unlikely(l1e_get_flags(nl1e) & L1_DISALLOW_MASK) )
> {
> MEM_LOG("Bad L1 flags %x",
> l1e_get_flags(nl1e) & L1_DISALLOW_MASK);
> return 0;
> }
> ...
> }
>
> the PTE flags are 800000 which corresponds to:
>
> #define _PAGE_NX_BIT (1U<<23)
At least in xen-unstable (and I think for much longer) L1_DISALLOW_MASK
contains _PAGE_NX_BIT dynamically depending on the processor
capabilities.
#define _PAGE_NX (cpu_has_nx ? _PAGE_NX_BIT : 0)
...
/*
* Disallow unused flag bits plus PAT/PSE, PCD, PWT and GLOBAL.
* Permit the NX bit if the hardware supports it.
*/
#define BASE_DISALLOW_MASK (0xFFFFF198U & ~_PAGE_NX)
#define L1_DISALLOW_MASK (BASE_DISALLOW_MASK | _PAGE_GNTTAB)
Does the hardware support NX? What does /proc/cpuinfo in dom0 think?
The guest kernel should be setting up __supported_pte_mask appropriately
to match the hardware and hence shouldn't be using NX if it isn't
available. There's a command line option to force NX, can you try
noexec=off on the guest command line.
My guess would be that the guest is getting a wrong EFER from
somewhere...
Ian.
11:42 a.m.
New subject: f10 x86_64 xen VM guests fail to boot on f8 host (guest setting NX bit in L1 PTE?)
Hi,
I can confirm this. With NX enabled in BIOS the domU boots fine (tested
with 2.6.28).
If I disable NX in BIOS the domU will crash.
- Valtteri Kiviniemi
Virtualization kirjoitti:
Hi Ian,
Indeed nx is on one and not the other! However, that doesn't help...
Broken CPU:
/proc/cpuinfo:
model name : Intel(R) Xeon(TM) CPU 3.00GHz
flags : fpu tsc msr pae mce cx8 apic mtrr mca cmov pat pse36 clflush dts acpi
mmx fxsr sse sse2 ss ht tm syscall lm constant_tsc up pni monitor ds_cpl cid cx16 xtpr
Good CPU:
model name : AMD Athlon(tm) 64 X2 Dual Core Processor 3600+
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36
clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow rep_good
pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy 3dnowprefetch
So, ran it again with noexec=0,
[ Minimal BASH-like line editing is supported.
ESC at any time cancels. ENTER at any time accepts your changes. ]
> kernel /vmlinuz-2.6.27.9-159.fc10.x86_64 ro root=LABEL=/ selinux=0 noipv6
nomodeset noexec=off
Results:
[ root@office64 xen ]# /usr/lib64/xen/bin/xenctx -s System.map-2.6.27.9-159.fc10.x86_64
119
rip: ffffffff8100b8a2 set_page_prot+0x6d
rsp: ffffffff81575f08
rax: ffffffea rbx: 000016e4 rcx: 00000055 rdx: 00000000
rsi: 800000014a293061 rdi: ffffffff816e4000 rbp: ffffffff81575f68
r8: 0000000f r9: ffffffff817ee350 r10: ffffffff817ee550 r11: 00000010
r12: ffffffff816e4000 r13: 800000014a293061 r14: 8000000000000161 r15: 00002c00
cs: 0000e033 ds: 00000000 fs: 00000000 gs: 00000000
Stack:
0000000000000055 0000000000000010 ffffffff8100b8a2 000000010000e030
0000000000010082 ffffffff81575f48 000000000000e02b ffffffff8100b89e
0000000000000200 ffffffff816e7000 0000000000000800 0000000000000016
ffffffff81575ff8 ffffffff815a5c60 0000000000002c00 0000000000000000
Code:
df 54 1d 00 4c 89 e7 4c 89 ee 31 d2 e8 22 d9 ff ff 85 c0 74 04 <0f> 0b eb fe 5b 41
5c 41 5d 41 5e
Call Trace:
[<ffffffff8100b8a2>] set_page_prot+0x6d <--
[<ffffffff8100b8a2>] set_page_prot+0x6d
[<ffffffff8100b89e>] set_page_prot+0x69
[<ffffffff815a5c60>] xen_start_kernel+0x5dd
Battling with bugzilla trying to get a new account. It doesn't like me :-(
Might have to leave it up to Jon to do the bugzilla thing.
Cheers
Phill.
On Tue, 2009-01-20 at 17:03 +0000, Ian Campbell wrote:
> (resending with original xen-devel thread participants on CC, please
> reply to this subthread, I'll forward you guys Mark's original mail in a
> second)
>
> On Tue, 2009-01-20 at 10:27 +0000, Mark McLoughlin wrote:
>> if ( unlikely(l1e_get_flags(nl1e) & L1_DISALLOW_MASK) )
>> {
>> MEM_LOG("Bad L1 flags %x",
>> l1e_get_flags(nl1e) & L1_DISALLOW_MASK);
>> return 0;
>> }
>> ...
>> }
>>
>> the PTE flags are 800000 which corresponds to:
>>
>> #define _PAGE_NX_BIT (1U<<23)
> At least in xen-unstable (and I think for much longer) L1_DISALLOW_MASK
> contains _PAGE_NX_BIT dynamically depending on the processor
> capabilities.
>
> #define _PAGE_NX (cpu_has_nx ? _PAGE_NX_BIT : 0)
> ...
> /*
> * Disallow unused flag bits plus PAT/PSE, PCD, PWT and GLOBAL.
> * Permit the NX bit if the hardware supports it.
> */
> #define BASE_DISALLOW_MASK (0xFFFFF198U & ~_PAGE_NX)
>
> #define L1_DISALLOW_MASK (BASE_DISALLOW_MASK | _PAGE_GNTTAB)
>
> Does the hardware support NX? What does /proc/cpuinfo in dom0 think?
>
> The guest kernel should be setting up __supported_pte_mask appropriately
> to match the hardware and hence shouldn't be using NX if it isn't
> available. There's a command line option to force NX, can you try
> noexec=off on the guest command line.
>
> My guess would be that the guest is getting a wrong EFER from
> somewhere...
>
> Ian.
>
>
5:40 a.m.
New subject: f10 x86_64 xen VM guests fail to boot on f8 host (guest setting NX bit in L1 PTE?)
On Tue, 2009-01-27 at 19:42 +0200, Valtteri Kiviniemi wrote:
Hi,
I can confirm this. With NX enabled in BIOS the domU boots fine (tested
with 2.6.28).
If I disable NX in BIOS the domU will crash.
Thanks for confirming. Did you get a chance to try the patch I sent?
Ian.
- Valtteri Kiviniemi
Virtualization kirjoitti:
> Hi Ian,
>
> Indeed nx is on one and not the other! However, that doesn't help...
>
> Broken CPU:
> /proc/cpuinfo:
> model name : Intel(R) Xeon(TM) CPU 3.00GHz
> flags : fpu tsc msr pae mce cx8 apic mtrr mca cmov pat pse36 clflush dts
acpi mmx fxsr sse sse2 ss ht tm syscall lm constant_tsc up pni monitor ds_cpl cid cx16
xtpr
>
> Good CPU:
> model name : AMD Athlon(tm) 64 X2 Dual Core Processor 3600+
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm 3dnowext 3dnow
rep_good pni cx16 lahf_lm cmp_legacy svm extapic cr8_legacy 3dnowprefetch
>
> So, ran it again with noexec=0,
>
> [ Minimal BASH-like line editing is supported.
> ESC at any time cancels. ENTER at any time accepts your changes. ]
>
>
> > kernel /vmlinuz-2.6.27.9-159.fc10.x86_64 ro root=LABEL=/ selinux=0 noipv6
nomodeset noexec=off
>
> Results:
>
> [ root@office64 xen ]# /usr/lib64/xen/bin/xenctx -s
System.map-2.6.27.9-159.fc10.x86_64 119
> rip: ffffffff8100b8a2 set_page_prot+0x6d
> rsp: ffffffff81575f08
> rax: ffffffea rbx: 000016e4 rcx: 00000055 rdx: 00000000
> rsi: 800000014a293061 rdi: ffffffff816e4000 rbp: ffffffff81575f68
> r8: 0000000f r9: ffffffff817ee350 r10: ffffffff817ee550 r11: 00000010
> r12: ffffffff816e4000 r13: 800000014a293061 r14: 8000000000000161 r15:
00002c00
> cs: 0000e033 ds: 00000000 fs: 00000000 gs: 00000000
>
> Stack:
> 0000000000000055 0000000000000010 ffffffff8100b8a2 000000010000e030
> 0000000000010082 ffffffff81575f48 000000000000e02b ffffffff8100b89e
> 0000000000000200 ffffffff816e7000 0000000000000800 0000000000000016
> ffffffff81575ff8 ffffffff815a5c60 0000000000002c00 0000000000000000
>
> Code:
> df 54 1d 00 4c 89 e7 4c 89 ee 31 d2 e8 22 d9 ff ff 85 c0 74 04 <0f> 0b eb fe
5b 41 5c 41 5d 41 5e
>
> Call Trace:
> [<ffffffff8100b8a2>] set_page_prot+0x6d <--
> [<ffffffff8100b8a2>] set_page_prot+0x6d
> [<ffffffff8100b89e>] set_page_prot+0x69
> [<ffffffff815a5c60>] xen_start_kernel+0x5dd
>
> Battling with bugzilla trying to get a new account. It doesn't like me :-(
>
> Might have to leave it up to Jon to do the bugzilla thing.
>
> Cheers
> Phill.
>
> On Tue, 2009-01-20 at 17:03 +0000, Ian Campbell wrote:
>> (resending with original xen-devel thread participants on CC, please
>> reply to this subthread, I'll forward you guys Mark's original mail in
a
>> second)
>>
>> On Tue, 2009-01-20 at 10:27 +0000, Mark McLoughlin wrote:
>>> if ( unlikely(l1e_get_flags(nl1e) & L1_DISALLOW_MASK) )
>>> {
>>> MEM_LOG("Bad L1 flags %x",
>>> l1e_get_flags(nl1e) & L1_DISALLOW_MASK);
>>> return 0;
>>> }
>>> ...
>>> }
>>>
>>> the PTE flags are 800000 which corresponds to:
>>>
>>> #define _PAGE_NX_BIT (1U<<23)
>> At least in xen-unstable (and I think for much longer) L1_DISALLOW_MASK
>> contains _PAGE_NX_BIT dynamically depending on the processor
>> capabilities.
>>
>> #define _PAGE_NX (cpu_has_nx ? _PAGE_NX_BIT : 0)
>> ...
>> /*
>> * Disallow unused flag bits plus PAT/PSE, PCD, PWT and GLOBAL.
>> * Permit the NX bit if the hardware supports it.
>> */
>> #define BASE_DISALLOW_MASK (0xFFFFF198U & ~_PAGE_NX)
>>
>> #define L1_DISALLOW_MASK (BASE_DISALLOW_MASK | _PAGE_GNTTAB)
>>
>> Does the hardware support NX? What does /proc/cpuinfo in dom0 think?
>>
>> The guest kernel should be setting up __supported_pte_mask appropriately
>> to match the hardware and hence shouldn't be using NX if it isn't
>> available. There's a command line option to force NX, can you try
>> noexec=off on the guest command line.
>>
>> My guess would be that the guest is getting a wrong EFER from
>> somewhere...
>>
>> Ian.
>>
>>
>
9:22 p.m.
New subject: f10 x86_64 xen VM guests fail to boot on f8 host (guest setting NX bit in L1 PTE?)
Thank you again Mark, Ian, and Phil.
As Phil pointed out, my working host has the NX feature, the broken host
does not.
I also could not create a bugzilla account, but was able to use a
co-worker's credentials to create this ticket:
https://bugzilla.redhat.com/show_bug.cgi?id=480880
Ian, I didn't have any luck with the noexec=off.
xm dmesg output is the same as posted earlier.
(XEN) mm.c:1362:d52 Bad L1 flags 800000
(XEN) traps.c:405:d52 Unhandled invalid opcode fault/trap [#6] in domain
52 on VCPU 0 [ec=0000]
(XEN) domain_crash_sync called from entry.S
(XEN) Domain 52 (vcpu#0) crashed on cpu#3:
...
xenctx output:
/usr/lib64/xen/bin/xenctx -s System.map-2.6.27.5-117.fc10.x86_64 52
rip: ffffffff8100b8a2 set_page_prot+0x6d
rsp: ffffffff81573f08
rax: ffffffea rbx: 000016e1 rcx: 00000054 rdx: 00000000
rsi: 80000000a7be7061 rdi: ffffffff816e1000 rbp: ffffffff81573f68
r8: 0000000f r9: ffffffff817eb450 r10: ffffffff817eb650 r11:
00000010
r12: ffffffff816e1000 r13: 80000000a7be7061 r14: 8000000000000161
r15: 00000016
cs: 0000e033 ds: 00000000 fs: 00000000 gs: 00000000
Stack:
0000000000000054 0000000000000010 ffffffff8100b8a2 000000010000e030
0000000000010082 ffffffff81573f48 000000000000e02b ffffffff8100b89e
0000000000000200 ffffffff816e4000 0000000000000800 0000000000002c00
ffffffff81573ff8 ffffffff815a3c60 0000000000002c00 0000000000000000
Code:
7b 4a 1d 00 4c 89 e7 4c 89 ee 31 d2 e8 22 d9 ff ff 85 c0 74 04 <0f> 0b
eb fe 5b 41 5c 41 5d 41 5e
Call Trace:
[<ffffffff8100b8a2>] set_page_prot+0x6d <--
[<ffffffff8100b8a2>] set_page_prot+0x6d
[<ffffffff8100b89e>] set_page_prot+0x69
[<ffffffff815a3c60>] xen_start_kernel+0x5dd
Thanks,
jon
-----Original Message-----
From: Mark McLoughlin [mailto:markmc@redhat.com]
Sent: Tuesday, January 20, 2009 7:27 PM
To: Jon Swanson
Cc: fedora-virt(a)redhat.com; virtualization(a)webwombat.com.au; Jeremy
Fitzhardinge; Ian Campbell
Subject: Re: [fedora-virt] f10 x86_64 xen VM guests fail to boot on f8
host (guest setting NX bit in L1 PTE?)
(Jeremy/Ian - here's some more info on the bug reported here:
http://lists.xensource.com/archives/html/xen-devel/2009-01/msg00176.html
)
Hi Jon/Phill,
Thanks for all the info.
Here's the important bits:
1) Host kernel is 2.6.21.7-5.fc8xen, that means the hypervisor is
xen-3.1.4
2) The guest kernel is 2.6.27.5-117.fc10.x86_64
3) Phill points out the faulting instruction is UD2. That just means
the guest kernel is hitting a BUG() assertion. See /asm-x86/bug.h:
#define BUG() \
do { \
asm volatile("ud2"); \
for (;;) ; \
} while (0)
4) The backtrace shows the fault happens in set_page_prot()
5) Jon's dmesg contains:
(XEN) mm.c:1362:d46 Bad L1 flags 800000
That means the guest is faulting here:
static void set_page_prot(void *addr, pgprot_t prot) { ....
if (HYPERVISOR_update_va_mapping((unsigned long)addr, pte, 0))
BUG();
}
because the PTE update is failing in the HV here:
static int mod_l1_entry(l1_pgentry_t *pl1e, l1_pgentry_t nl1e,
unsigned long gl1mfn) { ...
if ( unlikely(l1e_get_flags(nl1e) & L1_DISALLOW_MASK) )
{
MEM_LOG("Bad L1 flags %x",
l1e_get_flags(nl1e) & L1_DISALLOW_MASK);
return 0;
}
...
}
the PTE flags are 800000 which corresponds to:
#define _PAGE_NX_BIT (1U<<23)
Jon/Phill - can one of you two file a bug (bugzilla.redhat.com) with all
this info?
Thanks,
Mark.
Previous posts, for reference:
http://www.redhat.com/archives/fedora-xen/2009-January/thread.html#00022
http://www.redhat.com/archives/fedora-virt/2009-January/thread.html#0001
3
9:59 a.m.
New subject: f10 x86_64 xen VM guests fail to boot on f8 host (guest setting NX bit in L1 PTE?)
On Wed, 2009-01-21 at 12:22 +0900, Jon Swanson wrote:
Thank you again Mark, Ian, and Phil.
As Phil pointed out, my working host has the NX feature, the broken host
does not.
I also could not create a bugzilla account, but was able to use a
co-worker's credentials to create this ticket:
https://bugzilla.redhat.com/show_bug.cgi?id=480880
Ian, I didn't have any luck with the noexec=off.
It turns out that nonx_setup() and check_efer() both run quite a while
after all the set_page_prot calls in xen_setup_kernel() which include
_PAGE_NX via PAGE_KERNEL_RO.
On 32 bit __supported_pte_mask starts off without NX in it and it gets
added later if the system supports it. This is safe but means that the
pages frobbed by the early Xen setup won't have NX set when they could
(unless they all get frobbed again later?)
On 64 bit __supported_pte_mask contains NX at start of day and it is
taken away later on if the system turns out not to support it.
Native seems to mainly use _KERNPG_TABLE which does not include NX, can
you try this patch? (lots of printks because I don't have any non-NX
hardware to test properly).
diff -r ec792b22009f arch/x86/mm/init_64.c
--- a/arch/x86/mm/init_64.c Fri Jan 23 15:27:45 2009 +0000
+++ b/arch/x86/mm/init_64.c Fri Jan 23 15:58:03 2009 +0000
@@ -103,12 +103,15 @@
*/
static int __init nonx_setup(char *str)
{
+ printk(KERN_CRIT "noexec_setup %s\n", str);
if (!str)
return -EINVAL;
if (!strncmp(str, "on", 2)) {
+ printk(KERN_CRIT "noexec_setup: enabling NX\n");
__supported_pte_mask |= _PAGE_NX;
do_not_nx = 0;
} else if (!strncmp(str, "off", 3)) {
+ printk(KERN_CRIT "noexec_setup: disabling NX\n");
do_not_nx = 1;
__supported_pte_mask &= ~_PAGE_NX;
}
@@ -121,8 +124,13 @@
unsigned long efer;
rdmsrl(MSR_EFER, efer);
- if (!(efer & EFER_NX) || do_not_nx)
+ if (!(efer & EFER_NX) || do_not_nx) {
+ printk(KERN_CRIT "check_efer: disabling NX\n");
__supported_pte_mask &= ~_PAGE_NX;
+ } else
+ printk(KERN_CRIT "check_efer: leaving NX alone. supported_pte_mask %s the NX
bit\n",
+ __supported_pte_mask & _PAGE_NX ? "includes" :
"excludes");
+
}
int force_personality32;
diff -r ec792b22009f arch/x86/xen/enlighten.c
--- a/arch/x86/xen/enlighten.c Fri Jan 23 15:27:45 2009 +0000
+++ b/arch/x86/xen/enlighten.c Fri Jan 23 15:58:03 2009 +0000
@@ -54,6 +54,9 @@
#include "xen-ops.h"
#include "mmu.h"
#include "multicalls.h"
+
+#define _KERNPG_TABLE_RO __pgprot(_KERNPG_TABLE & ~_PAGE_RW)
+//#define _KERNPG_TABLE_RO (_KERNPG_TABLE)
EXPORT_SYMBOL_GPL(hypercall_page);
@@ -1476,6 +1479,15 @@
{
unsigned long pfn = __pa(addr) >> PAGE_SHIFT;
pte_t pte = pfn_pte(pfn, prot);
+ static int once = 5;
+
+ if (once > 0 && pte_val(pte) & _PAGE_NX) {
+ once--;
+ printk(KERN_CRIT "set_page_prot to %#lx (incl NX) supported_pte_mask %#lx %s the
NX bit\n",
+ pgprot_val(prot), __supported_pte_mask, __supported_pte_mask & _PAGE_NX ?
"includes" : "excludes");
+ printk(KERN_CRIT "pte is %#lx\n", pte_val(pte));
+ WARN_ON(1);
+ }
if (HYPERVISOR_update_va_mapping((unsigned long)addr, pte, 0))
BUG();
@@ -1522,9 +1534,9 @@
}
for (pteidx = 0; pteidx < ident_pte; pteidx += PTRS_PER_PTE)
- set_page_prot(&level1_ident_pgt[pteidx], PAGE_KERNEL_RO);
+ set_page_prot(&level1_ident_pgt[pteidx], _KERNPG_TABLE_RO);
- set_page_prot(pmd, PAGE_KERNEL_RO);
+ set_page_prot(pmd, _KERNPG_TABLE_RO);
}
static __init void xen_ident_map_ISA(void)
@@ -1601,12 +1613,12 @@
xen_map_identity_early(level2_ident_pgt, max_pfn);
/* Make pagetable pieces RO */
- set_page_prot(init_level4_pgt, PAGE_KERNEL_RO);
- set_page_prot(level3_ident_pgt, PAGE_KERNEL_RO);
- set_page_prot(level3_kernel_pgt, PAGE_KERNEL_RO);
- set_page_prot(level3_user_vsyscall, PAGE_KERNEL_RO);
- set_page_prot(level2_kernel_pgt, PAGE_KERNEL_RO);
- set_page_prot(level2_fixmap_pgt, PAGE_KERNEL_RO);
+ set_page_prot(init_level4_pgt, _KERNPG_TABLE_RO);
+ set_page_prot(level3_ident_pgt, _KERNPG_TABLE_RO);
+ set_page_prot(level3_kernel_pgt, _KERNPG_TABLE_RO);
+ set_page_prot(level3_user_vsyscall, _KERNPG_TABLE_RO);
+ set_page_prot(level2_kernel_pgt, _KERNPG_TABLE_RO);
+ set_page_prot(level2_fixmap_pgt, _KERNPG_TABLE_RO);
/* Pin down new L4 */
pin_pagetable_pfn(MMUEXT_PIN_L4_TABLE,
@@ -1670,9 +1682,9 @@
set_pgd(&swapper_pg_dir[KERNEL_PGD_BOUNDARY],
__pgd(__pa(level2_kernel_pgt) | _PAGE_PRESENT));
- set_page_prot(level2_kernel_pgt, PAGE_KERNEL_RO);
- set_page_prot(swapper_pg_dir, PAGE_KERNEL_RO);
- set_page_prot(empty_zero_page, PAGE_KERNEL_RO);
+ set_page_prot(level2_kernel_pgt, _KERNPG_TABLE_RO);
+ set_page_prot(swapper_pg_dir, _KERNPG_TABLE_RO);
+ set_page_prot(empty_zero_page, _KERNPG_TABLE_RO);
pin_pagetable_pfn(MMUEXT_UNPIN_TABLE, PFN_DOWN(__pa(pgd)));
2:52 a.m.
New subject: f10 x86_64 xen VM guests fail to boot on f8 host(guest setting NX bit in L1 PTE?)
Hey Ian,
Probably incompetence on my part, but I am unable to get the patch to
apply. What specific version of the kernel are you doing this on?
I'm getting the following error:
------------------------------------------------------------------------
---------------------------------------------
~/rpmbuild/BUILD> patch --dry-run -p0 < ~/f10xenNoNX.patch.txt
patching file a/arch/x86/mm/init_64.c
patch: **** malformed patch at line 19: @@ -121,8 +124,13 @@
------------------------------------------------------------------------
---------------------------------------------
I've tried with the following kernel versions, and duplicated your
directory structure:
kernel-2.6.27.5-117.fc10.src.rpm
kernel-2.6.27.9-159.fc10.src.rpm
I've also tried just patching one of the files:
------------------------------------------------------------------------
---------------------------------------------
~/rpmbuild/BUILD>patch --dry-run --verbose a/arch/x86/mm/init_64.c
~/f10init_64.c.patch
Hmm... Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|diff -r ec792b22009f arch/x86/mm/init_64.c
|--- a/arch/x86/mm/init_64.c Fri Jan 23 15:27:45 2009 +0000
|+++ b/arch/x86/mm/init_64.c Fri Jan 23 15:58:03 2009 +0000
--------------------------
Patching file a/arch/x86/mm/init_64.c using Plan A...
patch: **** malformed patch at line 19: @@ -121,8 +124,13 @@
------------------------------------------------------------------------
---------------------------------------------
I opened up the file and was about to do it manually, but it seems
radically different so stopped.
------------------------------------------------------------------------
---------------------------------------------
~/rpmbuild/BUILD> sed -n '120,125p' a/arch/x86/mm/init_64.c
pud = pud_page + pud_index(vaddr);
if (pud_none(*pud)) {
pmd = (pmd_t *) spp_getpage();
pud_populate(&init_mm, pud, pmd);
if (pmd != pmd_offset(pud, 0)) {
------------------------------------------------------------------------
---------------------------------------------
I'm sorry if I'm just being an idiot, but any insight you can share
would be greatly appreciated.
Thanks,
jon
-----Original Message-----
From: Ian Campbell [mailto:Ian.Campbell@citrix.com]
Sent: Saturday, January 24, 2009 12:59 AM
To: Jon Swanson
Cc: Mark McLoughlin; fedora-virt(a)redhat.com;
virtualization(a)webwombat.com.au; Jeremy Fitzhardinge
Subject: RE: [fedora-virt] f10 x86_64 xen VM guests fail to boot on f8
host(guest setting NX bit in L1 PTE?)
On Wed, 2009-01-21 at 12:22 +0900, Jon Swanson wrote:
Thank you again Mark, Ian, and Phil.
As Phil pointed out, my working host has the NX feature, the broken
host does not.
I also could not create a bugzilla account, but was able to use a
co-worker's credentials to create this ticket:
https://bugzilla.redhat.com/show_bug.cgi?id=480880
Ian, I didn't have any luck with the noexec=off.
It turns out that nonx_setup() and check_efer() both run quite a while
after all the set_page_prot calls in xen_setup_kernel() which include
_PAGE_NX via PAGE_KERNEL_RO.
On 32 bit __supported_pte_mask starts off without NX in it and it gets
added later if the system supports it. This is safe but means that the
pages frobbed by the early Xen setup won't have NX set when they could
(unless they all get frobbed again later?)
On 64 bit __supported_pte_mask contains NX at start of day and it is
taken away later on if the system turns out not to support it.
Native seems to mainly use _KERNPG_TABLE which does not include NX, can
you try this patch? (lots of printks because I don't have any non-NX
hardware to test properly).
diff -r ec792b22009f arch/x86/mm/init_64.c
--- a/arch/x86/mm/init_64.c Fri Jan 23 15:27:45 2009 +0000
+++ b/arch/x86/mm/init_64.c Fri Jan 23 15:58:03 2009 +0000
@@ -103,12 +103,15 @@
*/
static int __init nonx_setup(char *str) {
+ printk(KERN_CRIT "noexec_setup %s\n", str);
if (!str)
return -EINVAL;
if (!strncmp(str, "on", 2)) {
+ printk(KERN_CRIT "noexec_setup: enabling NX\n");
__supported_pte_mask |= _PAGE_NX;
do_not_nx = 0;
} else if (!strncmp(str, "off", 3)) {
+ printk(KERN_CRIT "noexec_setup: disabling
NX\n");
do_not_nx = 1;
__supported_pte_mask &= ~_PAGE_NX;
}
@@ -121,8 +124,13 @@
unsigned long efer;
rdmsrl(MSR_EFER, efer);
- if (!(efer & EFER_NX) || do_not_nx)
+ if (!(efer & EFER_NX) || do_not_nx) {
+ printk(KERN_CRIT "check_efer: disabling NX\n");
__supported_pte_mask &= ~_PAGE_NX;
+ } else
+ printk(KERN_CRIT "check_efer: leaving NX alone.
supported_pte_mask %s the NX bit\n",
+ __supported_pte_mask & _PAGE_NX ? "includes" :
"excludes");
+
}
int force_personality32;
diff -r ec792b22009f arch/x86/xen/enlighten.c
--- a/arch/x86/xen/enlighten.c Fri Jan 23 15:27:45 2009 +0000
+++ b/arch/x86/xen/enlighten.c Fri Jan 23 15:58:03 2009 +0000
@@ -54,6 +54,9 @@
#include "xen-ops.h"
#include "mmu.h"
#include "multicalls.h"
+
+#define _KERNPG_TABLE_RO __pgprot(_KERNPG_TABLE & ~_PAGE_RW) //#define
+_KERNPG_TABLE_RO (_KERNPG_TABLE)
EXPORT_SYMBOL_GPL(hypercall_page);
@@ -1476,6 +1479,15 @@
{
unsigned long pfn = __pa(addr) >> PAGE_SHIFT;
pte_t pte = pfn_pte(pfn, prot);
+ static int once = 5;
+
+ if (once > 0 && pte_val(pte) & _PAGE_NX) {
+ once--;
+ printk(KERN_CRIT "set_page_prot to %#lx (incl NX)
supported_pte_mask %#lx %s the NX bit\n",
+ pgprot_val(prot), __supported_pte_mask,
__supported_pte_mask & _PAGE_NX ? "includes" : "excludes");
+ printk(KERN_CRIT "pte is %#lx\n", pte_val(pte));
+ WARN_ON(1);
+ }
if (HYPERVISOR_update_va_mapping((unsigned long)addr, pte, 0))
BUG();
@@ -1522,9 +1534,9 @@
}
for (pteidx = 0; pteidx < ident_pte; pteidx += PTRS_PER_PTE)
- set_page_prot(&level1_ident_pgt[pteidx],
PAGE_KERNEL_RO);
+ set_page_prot(&level1_ident_pgt[pteidx],
_KERNPG_TABLE_RO);
- set_page_prot(pmd, PAGE_KERNEL_RO);
+ set_page_prot(pmd, _KERNPG_TABLE_RO);
}
static __init void xen_ident_map_ISA(void) @@ -1601,12 +1613,12 @@
xen_map_identity_early(level2_ident_pgt, max_pfn);
/* Make pagetable pieces RO */
- set_page_prot(init_level4_pgt, PAGE_KERNEL_RO);
- set_page_prot(level3_ident_pgt, PAGE_KERNEL_RO);
- set_page_prot(level3_kernel_pgt, PAGE_KERNEL_RO);
- set_page_prot(level3_user_vsyscall, PAGE_KERNEL_RO);
- set_page_prot(level2_kernel_pgt, PAGE_KERNEL_RO);
- set_page_prot(level2_fixmap_pgt, PAGE_KERNEL_RO);
+ set_page_prot(init_level4_pgt, _KERNPG_TABLE_RO);
+ set_page_prot(level3_ident_pgt, _KERNPG_TABLE_RO);
+ set_page_prot(level3_kernel_pgt, _KERNPG_TABLE_RO);
+ set_page_prot(level3_user_vsyscall, _KERNPG_TABLE_RO);
+ set_page_prot(level2_kernel_pgt, _KERNPG_TABLE_RO);
+ set_page_prot(level2_fixmap_pgt, _KERNPG_TABLE_RO);
/* Pin down new L4 */
pin_pagetable_pfn(MMUEXT_PIN_L4_TABLE,
@@ -1670,9 +1682,9 @@
set_pgd(&swapper_pg_dir[KERNEL_PGD_BOUNDARY],
__pgd(__pa(level2_kernel_pgt) | _PAGE_PRESENT));
- set_page_prot(level2_kernel_pgt, PAGE_KERNEL_RO);
- set_page_prot(swapper_pg_dir, PAGE_KERNEL_RO);
- set_page_prot(empty_zero_page, PAGE_KERNEL_RO);
+ set_page_prot(level2_kernel_pgt, _KERNPG_TABLE_RO);
+ set_page_prot(swapper_pg_dir, _KERNPG_TABLE_RO);
+ set_page_prot(empty_zero_page, _KERNPG_TABLE_RO);
pin_pagetable_pfn(MMUEXT_UNPIN_TABLE, PFN_DOWN(__pa(pgd)));
5564
days inactive
5574
days old
9 comments
5 participants
participants (5)
-
Ian Campbell -
Jon Swanson -
Mark McLoughlin -
Valtteri Kiviniemi -
Virgil