kernel-xen0-2.6.18-1.2200.fc5 amusingly unhappy

[Matt Bernstein]

[root@hyperion ~]# cat /proc/version
Linux version 2.6.18-1.2200.fc5xen0 (brewbuilder(a)hs20-bc2-3.build.redhat.com) (gcc version
4.1.1 20060525 (Red Hat 4.1.1-1)) #1 SMP Sat Oct 14 17:49:47 EDT 2006
[root@hyperion ~]# getenforce
Permissive
[root@hyperion ~]# xm list
Error: Error connecting to xend: No such file or directory.  Is xend running?
[root@hyperion ~]# /etc/init.d/xend status
xend is running

Reverting to 2.6.17-1.2157_FC5xen0 fixes this problem.

[Ben]

My understanding from https://bugzilla.redhat.com/bugzilla/ 
show_bug.cgi?id=199944 is that to make this kernel work you have to  
have a new xen package, which I haven't seen an update for yet.

On Oct 17, 2006, at 3:19 AM, Matt Bernstein wrote:

>  [root@hyperion ~]# cat /proc/version
>  Linux version 2.6.18-1.2200.fc5xen0 (brewbuilder@hs20- 
>  bc2-3.build.redhat.com) (gcc version 4.1.1 20060525 (Red Hat  
>  4.1.1-1)) #1 SMP Sat Oct 14 17:49:47 EDT 2006
>  [root@hyperion ~]# getenforce
>  Permissive
>  [root@hyperion ~]# xm list
>  Error: Error connecting to xend: No such file or directory.  Is  
>  xend running?
>  [root@hyperion ~]# /etc/init.d/xend status
>  xend is running
>
>  Reverting to 2.6.17-1.2157_FC5xen0 fixes this problem.
>
>  --
>  Fedora-xen mailing list
>  Fedora-xen(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-xen

AW: kernel-xen0-2.6.18-1.2200.fc5 amusingly unhappy

[Thomas von Steiger]

There is now bugzilla for this problem:
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=208529

My last running kernel with current xen rpm's is 2.6.17-1.2174_FC5_xen0
becose m'm not able to find this kernel version rpm again on the fedora
mirrors.
I found the latest 5 kernel version should be safe on the mirrors ?
Can anybody tell me where I can find this kernel rpm ?

Regards,
Thomas

>  -----Ursprüngliche Nachricht-----
>  Von: fedora-xen-bounces(a)redhat.com [mailto:[e-mail]]
>  Im Auftrag von Ben
>  Gesendet: Dienstag, 17. Oktober 2006 17:08
>  An: Matt Bernstein
>  Cc: fedora-xen(a)redhat.com
>  Betreff: Re: [Fedora-xen] kernel-xen0-2.6.18-1.2200.fc5 amusingly unhappy
>  
>  My understanding from https://bugzilla.redhat.com/bugzilla/
>  show_bug.cgi?id=199944 is that to make this kernel work you have to
>  have a new xen package, which I haven't seen an update for yet.
>  
>  On Oct 17, 2006, at 3:19 AM, Matt Bernstein wrote:
>  
>  > [root@hyperion ~]# cat /proc/version
>  > Linux version 2.6.18-1.2200.fc5xen0 (brewbuilder@hs20-
>  > bc2-3.build.redhat.com) (gcc version 4.1.1 20060525 (Red Hat
>  > 4.1.1-1)) #1 SMP Sat Oct 14 17:49:47 EDT 2006
>  > [root@hyperion ~]# getenforce
>  > Permissive
>  > [root@hyperion ~]# xm list
>  > Error: Error connecting to xend: No such file or directory.  Is
>  > xend running?
>  > [root@hyperion ~]# /etc/init.d/xend status
>  > xend is running
>  >
>  > Reverting to 2.6.17-1.2157_FC5xen0 fixes this problem.
>  >
>  > --
>  > Fedora-xen mailing list
>  > Fedora-xen(a)redhat.com
>  > https://www.redhat.com/mailman/listinfo/fedora-xen
>  
>  --
>  Fedora-xen mailing list
>  Fedora-xen(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-xen
>  
>  
>  --
>  No virus found in this incoming message.
>  Checked by AVG Free Edition.
>  Version: 7.1.408 / Virus Database: 268.13.4/480 - Release Date: 17.10.2006

[Daniel P. Berrange]

On Tue, Oct 17, 2006 at 11:19:21AM +0100, Matt Bernstein wrote:
>  [root@hyperion ~]# cat /proc/version
>  Linux version 2.6.18-1.2200.fc5xen0 
>  (brewbuilder(a)hs20-bc2-3.build.redhat.com) (gcc version 4.1.1 20060525 (Red 
>  Hat 4.1.1-1)) #1 SMP Sat Oct 14 17:49:47 EDT 2006
>  [root@hyperion ~]# getenforce
>  Permissive
>  [root@hyperion ~]# xm list
>  Error: Error connecting to xend: No such file or directory.  Is xend 
>  running?
>  [root@hyperion ~]# /etc/init.d/xend status
>  xend is running
>  
>  Reverting to 2.6.17-1.2157_FC5xen0 fixes this problem. 
We had a little snafu when pushing out the updates. The 2.6.18 kernel
which was pushed out includes Xen 3.0.3 APIs, but we forgot to push
the corresponding userspace bits (xen & libvirt RPMs). Because upstream
Xen developers saw fit to break ABI compatability for Dom0 control APIs
in 3.0.3 the older bits don't work :-(

We've got the neccessary updates in  the 'updates-testing' repository
as of a few hours ago, so if you want to try them out use:

  yum --enablerepo=updates-testing install xen libvirt


Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  [phone] -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|

[Tom Wilkie]

Yup thats fixed it.  Cheers

Tom

On 18 Oct 2006, at 17:08, Daniel P. Berrange wrote:

>  On Tue, Oct 17, 2006 at 11:19:21AM +0100, Matt Bernstein wrote:
> > [root@hyperion ~]# cat /proc/version
> > Linux version 2.6.18-1.2200.fc5xen0
> > (brewbuilder(a)hs20-bc2-3.build.redhat.com) (gcc version 4.1.1  
> > 20060525 (Red
> > Hat 4.1.1-1)) #1 SMP Sat Oct 14 17:49:47 EDT 2006
> > [root@hyperion ~]# getenforce
> > Permissive
> > [root@hyperion ~]# xm list
> > Error: Error connecting to xend: No such file or directory.  Is xend
> > running?
> > [root@hyperion ~]# /etc/init.d/xend status
> > xend is running
> >
> > Reverting to 2.6.17-1.2157_FC5xen0 fixes this problem.
>
>  We had a little snafu when pushing out the updates. The 2.6.18 kernel
>  which was pushed out includes Xen 3.0.3 APIs, but we forgot to push
>  the corresponding userspace bits (xen & libvirt RPMs). Because  
>  upstream
>  Xen developers saw fit to break ABI compatability for Dom0 control  
>  APIs
>  in 3.0.3 the older bits don't work :-(
>
>  We've got the neccessary updates in  the 'updates-testing' repository
>  as of a few hours ago, so if you want to try them out use:
>
>    yum --enablerepo=updates-testing install xen libvirt
>
>
>  Regards,
>  Dan.
>  -- 
>  |=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978  
>  392 2496 -=|
>  |=-           Perl modules: http://search.cpan.org/ 
>  ~danberr/              -=|
>  |=-               Projects: http://freshmeat.net/ 
>  ~danielpb/               -=|
>  |=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B  
>  9505  -=|
>
>  --
>  Fedora-xen mailing list
>  Fedora-xen(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-xen

[Robert Story]

On Wed, 18 Oct 2006 17:08:21 +0100 Daniel wrote:
DPB> We had a little snafu when pushing out the updates. The 2.6.18 kernel
DPB> which was pushed out includes Xen 3.0.3 APIs, but we forgot to push
DPB> the corresponding userspace bits (xen & libvirt RPMs). Because upstream
DPB> Xen developers saw fit to break ABI compatability for Dom0 control APIs
DPB> in 3.0.3 the older bits don't work :-(
DPB> 
DPB> We've got the neccessary updates in  the 'updates-testing' repository
DPB> as of a few hours ago, so if you want to try them out use:
DPB> 
DPB>   yum --enablerepo=updates-testing install xen libvirt

What versions *should* we have? I tried this yum update, but it didn't grab
anything new. I'm still seeing the "No such file or directory" error for
all
xm commands..


# uname -a
Linux server 2.6.18-1.2200.fc5xen0 #1 SMP Sat Oct 14 17:49:47 EDT 2006 i686
athlon i386 GNU/Linux

# rpm -q xen libvirt
xen-3.0.2-5.fc5
libvirt-0.1.7-2.FC5
# yum --enablerepo=updates-testing install xen libvirt
core                                                                 [1/4]
extras                                                               [2/4]
updates-released                                                     [3/4]
updates-testing                                                      [4/4]
Reading repository metadata in from local files

Parsing package install arguments
Nothing to do


# service xend start
Starting xend:                                             [  OK  ]

# xm list
Error: Error connecting to xend: No such file or directory.  Is xend running?


# ps -ef|grep xen
root         9     7  0 Oct26 ?        00:00:00 [xenwatch]
root        10     7  0 Oct26 ?        00:00:00 [xenbus]
root      2600     1  0 Oct26 ?        00:00:00 python /usr/sbin/xend start
root      2601  2600  0 Oct26 ?        00:00:00 python /usr/sbin/xend start

[itamar]

I am using xen on Fc6 and is running very fine for me

why you don't upgrade to FC6 ?

--------------------

Itamar Reis Peixoto

e-mail: itamar(a)ispbrasil.com.br
msn: itamarjp(a)starmedia.com
skype: itamarjp
icq: 81053601
[phone]
[phone]
----- Original Message ----- 
From: "Robert Story" <rstory-l(a)2006.revelstone.com&gt;
To: "Daniel P. Berrange" <berrange(a)redhat.com&gt;
Cc: <fedora-xen(a)redhat.com&gt;
Sent: Friday, October 27, 2006 10:33 AM
Subject: Re: [Fedora-xen] kernel-xen0-2.6.18-1.2200.fc5 amusingly unhappy


>  On Wed, 18 Oct 2006 17:08:21 +0100 Daniel wrote:
>  DPB> We had a little snafu when pushing out the updates. The 2.6.18 kernel
>  DPB> which was pushed out includes Xen 3.0.3 APIs, but we forgot to push
>  DPB> the corresponding userspace bits (xen & libvirt RPMs). Because 
>  upstream
>  DPB> Xen developers saw fit to break ABI compatability for Dom0 control 
>  APIs
>  DPB> in 3.0.3 the older bits don't work :-(
>  DPB>
>  DPB> We've got the neccessary updates in  the 'updates-testing'
> repository
>  DPB> as of a few hours ago, so if you want to try them out use:
>  DPB>
>  DPB>   yum --enablerepo=updates-testing install xen libvirt
>
>  What versions *should* we have? I tried this yum update, but it didn't 
>  grab
>  anything new. I'm still seeing the "No such file or directory" error for 
>  all
>  xm commands..
>
>
>  # uname -a
>  Linux server 2.6.18-1.2200.fc5xen0 #1 SMP Sat Oct 14 17:49:47 EDT 2006 
>  i686
>  athlon i386 GNU/Linux
>
>  # rpm -q xen libvirt
>  xen-3.0.2-5.fc5
>  libvirt-0.1.7-2.FC5
>  # yum --enablerepo=updates-testing install xen libvirt
>  core                                                                 [1/4]
>  extras                                                               [2/4]
>  updates-released                                                     [3/4]
>  updates-testing                                                      [4/4]
>  Reading repository metadata in from local files
>
>  Parsing package install arguments
>  Nothing to do
>
>
>  # service xend start
>  Starting xend:                                             [  OK  ]
>
>  # xm list
>  Error: Error connecting to xend: No such file or directory.  Is xend 
>  running?
>
>
>  # ps -ef|grep xen
>  root         9     7  0 Oct26 ?        00:00:00 [xenwatch]
>  root        10     7  0 Oct26 ?        00:00:00 [xenbus]
>  root      2600     1  0 Oct26 ?        00:00:00 python /usr/sbin/xend 
>  start
>  root      2601  2600  0 Oct26 ?        00:00:00 python /usr/sbin/xend 
>  start
>
>  --
>  Fedora-xen mailing list
>  Fedora-xen(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-xen
>

[Robert Story]

On Fri, 27 Oct 2006 11:29:53 -0300 Itamar wrote:
IRP> I am using xen on Fc6 and is running very fine for me
IRP> 
IRP> why you don't upgrade to FC6 ?

I like to give major new releases of any software a few weeks to settle down...

[Daniel P. Berrange]

On Fri, Oct 27, 2006 at 09:33:13AM -0400, Robert Story wrote:
>  On Wed, 18 Oct 2006 17:08:21 +0100 Daniel wrote:
>  DPB> We had a little snafu when pushing out the updates. The 2.6.18 kernel
>  DPB> which was pushed out includes Xen 3.0.3 APIs, but we forgot to push
>  DPB> the corresponding userspace bits (xen & libvirt RPMs). Because upstream
>  DPB> Xen developers saw fit to break ABI compatability for Dom0 control APIs
>  DPB> in 3.0.3 the older bits don't work :-(
>  DPB> 
>  DPB> We've got the neccessary updates in  the 'updates-testing'
> repository
>  DPB> as of a few hours ago, so if you want to try them out use:
>  DPB> 
>  DPB>   yum --enablerepo=updates-testing install xen libvirt
>  
>  What versions *should* we have? I tried this yum update, but it didn't grab
>  anything new. I'm still seeing the "No such file or directory" error for
> all
>  xm commands..
>  
>  
>  # uname -a
>  Linux server 2.6.18-1.2200.fc5xen0 #1 SMP Sat Oct 14 17:49:47 EDT 2006 i686
>  athlon i386 GNU/Linux
>  
>  # rpm -q xen libvirt
>  xen-3.0.2-5.fc5
>  libvirt-0.1.7-2.FC5
>  # yum --enablerepo=updates-testing install xen libvirt
>  core                                                                 [1/4]
>  extras                                                               [2/4]
>  updates-released                                                     [3/4]
>  updates-testing                                                      [4/4]
>  Reading repository metadata in from local files
>  
>  Parsing package install arguments
>  Nothing to do
>  
>  
>  # service xend start
>  Starting xend:                                             [  OK  ]
>  
>  # xm list
>  Error: Error connecting to xend: No such file or directory.  Is xend running? 
Could you do 'strace xm list' and post the log as an attachment. 

Also,  I'm wondering if there is an SELinux polcy fix we missed out. Could
you tell me if you have selinux enabled, and in enforcing or permissive 
mode. If enforcing, see if there's anything interesting (AVC denials) in
/var/log/messages, or /var/log/audit/audit.log

>  # ps -ef|grep xen
>  root         9     7  0 Oct26 ?        00:00:00 [xenwatch]
>  root        10     7  0 Oct26 ?        00:00:00 [xenbus]
>  root      2600     1  0 Oct26 ?        00:00:00 python /usr/sbin/xend start
>  root      2601  2600  0 Oct26 ?        00:00:00 python /usr/sbin/xend start 
An 'lsof -p 2601' would be handy to let us see what ports xend is listening
for connections on.

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  [phone] -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|

[Robert Story]

On Fri, 27 Oct 2006 15:44:17 +0100 Daniel wrote:
DPB> > # xm list
DPB> > Error: Error connecting to xend: No such file or directory.  Is xend
running?
DPB> 
DPB> Could you do 'strace xm list' and post the log as an attachment. 

attached.

DPB> Also,  I'm wondering if there is an SELinux polcy fix we missed out. Could
DPB> you tell me if you have selinux enabled, and in enforcing or permissive 
DPB> mode. If enforcing, see if there's anything interesting (AVC denials) in
DPB> /var/log/messages, or /var/log/audit/audit.log

Sorry, I forgot to mention that. SELinux is disabled.

 # getenforce
 Disabled

DPB> An 'lsof -p 2601' would be handy to let us see what ports xend is
listening
DPB> for connections on.

Also attached..

[Robert Story]

On Fri, 27 Oct 2006 11:02:14 -0400 Robert wrote:
RS> DPB> An 'lsof -p 2601' would be handy to let us see what ports xend is
listening
RS> DPB> for connections on.
RS> 
RS> Also attached..

Note, I had rebooted, so the pid changed.. I also just did a lsof on the
child, which is listening on a TCP port.. attached...

[Ben]

On Oct 27, 2006, at 7:44 AM, Daniel P. Berrange wrote:

>  On Fri, Oct 27, 2006 at 09:33:13AM -0400, Robert Story wrote:
> >
> > # service xend start
> > Starting xend:                                             [  OK  ]
> >
> > # xm list
> > Error: Error connecting to xend: No such file or directory.  Is  
> > xend running?
>
>  Could you do 'strace xm list' and post the log as an attachment.
>
>  Also,  I'm wondering if there is an SELinux polcy fix we missed  
>  out. Could
>  you tell me if you have selinux enabled, and in enforcing or  
>  permissive
>  mode. If enforcing, see if there's anything interesting (AVC  
>  denials) in
>  /var/log/messages, or /var/log/audit/audit.log
>  
I think the problem (and fix) has been described pretty well already  
by Guillaume Bougard:

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=211745

[Daniel P. Berrange]

On Fri, Oct 27, 2006 at 08:05:14AM -0700, Ben wrote:
>  
>  On Oct 27, 2006, at 7:44 AM, Daniel P. Berrange wrote:
>  
>  >On Fri, Oct 27, 2006 at 09:33:13AM -0400, Robert Story wrote:
>  >>
>  >># service xend start
>  >>Starting xend:                                             [  OK  ]
>  >>
>  >># xm list
>  >>Error: Error connecting to xend: No such file or directory.  Is  
>  >>xend running?
>  >
>  >Could you do 'strace xm list' and post the log as an attachment.
>  >
>  >Also,  I'm wondering if there is an SELinux polcy fix we missed  
>  >out. Could
>  >you tell me if you have selinux enabled, and in enforcing or  
>  >permissive
>  >mode. If enforcing, see if there's anything interesting (AVC  
>  >denials) in
>  >/var/log/messages, or /var/log/audit/audit.log
>  >
>  
>  I think the problem (and fix) has been described pretty well already  
>  by Guillaume Bougard:
>  
>  https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=211745 
I find it very hard to believe that changing 'Xend changeset: unavailable"
to list an explicit changeset has anything at all todo with the problem.
All fedora & RHEL xen builds have the changeset info disabled, and it is
not used in any functional codepath path related to the hypervisor or 
kernel. It is merely a bit of user facing info.

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  [phone] -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|

[Daniel P. Berrange]

On Fri, Oct 27, 2006 at 11:04:54AM -0400, Robert Story wrote:
>  On Fri, 27 Oct 2006 11:02:14 -0400 Robert wrote:
>  RS> DPB> An 'lsof -p 2601' would be handy to let us see what ports xend is
> listening
>  RS> DPB> for connections on.
>  RS> 
>  RS> Also attached..
>  
>  Note, I had rebooted, so the pid changed.. I also just did a lsof on the
>  child, which is listening on a TCP port.. attached... 
Ok, this is useful. The strace shows xm connecting to

connect(3, {sa_family=AF_FILE, path="/var/run/xend/xmlrpc.sock"}, 27) = -1
ENOENT (No such file or directory)

But the xend is only listening on port 8002, not unix sockets.

Can you post /etc/xen/xend-config.sxp 

Also, do you have an /etc/xen/xend-config.sxp.rpmnew perhaps ? I think
the new config file may not have been deployed during the RPM upgrade

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  [phone] -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|

[Robert Story]

On Fri, 27 Oct 2006 17:03:19 +0100 Daniel wrote:
DPB> Ok, this is useful. The strace shows xm connecting to
DPB> 
DPB> connect(3, {sa_family=AF_FILE, path="/var/run/xend/xmlrpc.sock"}, 27) =
-1 ENOENT (No such file or directory)
DPB> 
DPB> But the xend is only listening on port 8002, not unix sockets.
DPB> 
DPB> Can you post /etc/xen/xend-config.sxp 

Attached...

DPB> Also, do you have an /etc/xen/xend-config.sxp.rpmnew perhaps ? I think
DPB> the new config file may not have been deployed during the RPM upgrade

Nope, no rpmnew files in that directory..

[Guillaume Bougard]

Le vendredi 27 octobre 2006 à 16:58 +0100, Daniel P. Berrange a écrit :

>  > I think the problem (and fix) has been described pretty well
> already  
>  > by Guillaume Bougard:
>  > 
>  > https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=211745
>  
>  I find it very hard to believe that changing 'Xend changeset: unavailable"
>  to list an explicit changeset has anything at all todo with the problem.
>  All fedora & RHEL xen builds have the changeset info disabled, and it is
>  not used in any functional codepath path related to the hypervisor or 
>  kernel. It is merely a bit of user facing info.
>  
>  Dan. 

Hi all,

just a comment here, what I found is not "changing something to list an
explicit changeset", for me this was just a clue to understand the
problem.

In fact, when I study a little the xen patch really applied with the
kernel-xen0-2.6.18-1.2200.fc5, I discovered the patch is based on xen
3.0.3 which has totally different API than the one expected by the
xen-3.0.2 installed in FC5 at that time... What I do to fix the problem
is to find the really patch that should be applied (and it's based on
xen-3.0.2) and recompiled the kernel, and it was working.

So I think you should completly forget that kernel as it should has not
been released and it seems nobody is working on it in the CVS (maybe the
guy is on holidays after the release of FC6...) so be patient... I think
this would be fix in few days. (hope)

Best regards,

G. Bougard

[Daniel P. Berrange]

On Fri, Oct 27, 2006 at 11:43:59PM +0200, Guillaume Bougard wrote:
>  Le vendredi 27 octobre 2006 à 16:58 +0100, Daniel P. Berrange a écrit
> :
>  
>  > > I think the problem (and fix) has been described pretty well already  
>  > > by Guillaume Bougard:
>  > > 
>  > > https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=211745
>  > 
>  > I find it very hard to believe that changing 'Xend changeset: unavailable"
>  > to list an explicit changeset has anything at all todo with the problem.
>  > All fedora & RHEL xen builds have the changeset info disabled, and it is
>  > not used in any functional codepath path related to the hypervisor or 
>  > kernel. It is merely a bit of user facing info.
>  > 
>  > Dan.
>  
>  
>  Hi all,
>  
>  just a comment here, what I found is not "changing something to list an
>  explicit changeset", for me this was just a clue to understand the
>  problem.
>  
>  In fact, when I study a little the xen patch really applied with the
>  kernel-xen0-2.6.18-1.2200.fc5, I discovered the patch is based on xen
>  3.0.3 which has totally different API than the one expected by the
>  xen-3.0.2 installed in FC5 at that time... What I do to fix the problem
>  is to find the really patch that should be applied (and it's based on
>  xen-3.0.2) and recompiled the kernel, and it was working. 
The RPM version number here is a little misleading. The xen-3.0.2-5.FC5
RPM is actually based off a very recent pre-release of the 3.0.3 tree
(changeset 11540). The kernel-xen0-2.6.18-1.20.fc5 is also based off
a pre-release of the 3.0.3 tree (the same changeset  11540) so they
match exactly.

I think, however, I may have just found the problem - it looks like the
/dev/xen/evtchn  device is being created with the wrong major-minor
numbers, causing xenstored to fail, and thus breaking xend. The reason
it /appeared/ to work ok for some people is that xenstored did not get
restarted in %post section of the xen RPM. Going to investigate some 
more to confirm

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  [phone] -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|

[Daniel P. Berrange]

On Fri, Oct 27, 2006 at 11:13:40PM +0100, Daniel P. Berrange wrote:
>  On Fri, Oct 27, 2006 at 11:43:59PM +0200, Guillaume Bougard wrote:
>  > Le vendredi 27 octobre 2006 à 16:58 +0100, Daniel P. Berrange a écrit :
>  > 
>  > > > I think the problem (and fix) has been described pretty well already  
>  > > > by Guillaume Bougard:
>  > > > 
>  > > > https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=211745
>  > > 
>  > > I find it very hard to believe that changing 'Xend changeset:
> unavailable"
>  > > to list an explicit changeset has anything at all todo with the problem.
>  > > All fedora & RHEL xen builds have the changeset info disabled, and it is
>  > > not used in any functional codepath path related to the hypervisor or 
>  > > kernel. It is merely a bit of user facing info.
>  > > 
>  > > Dan.
>  > 
>  > 
>  > Hi all,
>  > 
>  > just a comment here, what I found is not "changing something to list an
>  > explicit changeset", for me this was just a clue to understand the
>  > problem.
>  > 
>  > In fact, when I study a little the xen patch really applied with the
>  > kernel-xen0-2.6.18-1.2200.fc5, I discovered the patch is based on xen
>  > 3.0.3 which has totally different API than the one expected by the
>  > xen-3.0.2 installed in FC5 at that time... What I do to fix the problem
>  > is to find the really patch that should be applied (and it's based on
>  > xen-3.0.2) and recompiled the kernel, and it was working.
>  
>  The RPM version number here is a little misleading. The xen-3.0.2-5.FC5
>  RPM is actually based off a very recent pre-release of the 3.0.3 tree
>  (changeset 11540). The kernel-xen0-2.6.18-1.20.fc5 is also based off
>  a pre-release of the 3.0.3 tree (the same changeset  11540) so they
>  match exactly. 
Ok, the CVS logs were confusing me here - the hypervisor is on 11540,
but the actual kernel is on 11740 - as Guillaume correctly points out
in his BZ ticket. Since 11740 is only 30 changesets before the 3.0.3
release tar.gz, and those 30 changesets contain some pretty important
bugfixes I plan to just sync the userspace xen RPM upto 3.0.3 official
tar.gz. This should give FC5 a fully operational Xen stack again.

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  [phone] -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|

[Robert Story]

On Sat, 28 Oct 2006 01:19:43 +0100 Daniel wrote:
DPB> Ok, the CVS logs were confusing me here - the hypervisor is on 11540,
DPB> but the actual kernel is on 11740 - as Guillaume correctly points out
DPB> in his BZ ticket. Since 11740 is only 30 changesets before the 3.0.3
DPB> release tar.gz, and those 30 changesets contain some pretty important
DPB> bugfixes I plan to just sync the userspace xen RPM upto 3.0.3 official
DPB> tar.gz. This should give FC5 a fully operational Xen stack again.

Hi Dan,

Glad to know you've figured it out. Please let us know when there is something
in updates-testing we can try! Thanks!

[Daniel P. Berrange]

On Sat, Oct 28, 2006 at 05:52:21AM -0400, Robert Story wrote:
>  On Sat, 28 Oct 2006 01:19:43 +0100 Daniel wrote:
>  DPB> Ok, the CVS logs were confusing me here - the hypervisor is on 11540,
>  DPB> but the actual kernel is on 11740 - as Guillaume correctly points out
>  DPB> in his BZ ticket. Since 11740 is only 30 changesets before the 3.0.3
>  DPB> release tar.gz, and those 30 changesets contain some pretty important
>  DPB> bugfixes I plan to just sync the userspace xen RPM upto 3.0.3 official
>  DPB> tar.gz. This should give FC5 a fully operational Xen stack again.
>  
>  Hi Dan,
>  
>  Glad to know you've figured it out. Please let us know when there is something
>  in updates-testing we can try! Thanks! 
It'll take me a little while to get a formal build made & pushed through the
errata process, because I need to do some further testing before committing
it to Fedora CVS. So if anyone is interested in testing in the meantime, I've
put my unofficial prototype build up here http://people.redhat.com/berrange/xen/
I'll remove RPMs once the formal build is in updates-testing.

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  [phone] -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|

[Adrian Chadd]

On Sat, Oct 28, 2006, Daniel P. Berrange wrote:

>  > Glad to know you've figured it out. Please let us know when
> there is something
>  > in updates-testing we can try! Thanks!
>  
>  It'll take me a little while to get a formal build made & pushed through the
>  errata process, because I need to do some further testing before committing
>  it to Fedora CVS. So if anyone is interested in testing in the meantime, I've
>  put my unofficial prototype build up here http://people.redhat.com/berrange/xen/
>  I'll remove RPMs once the formal build is in updates-testing. 
Works for me - both on a FC5 machine thats been upgraded over time and on a
freshly-installed FC5 that's been upgraded just once. I'm running a slightly
modified config on one to support >1 vlan and physical interface; I've also
disabled the Xen TCP services and stuck to the UNIX services.

Thanks Dan!



Adrian

[Adrian Chadd]

On Sun, Oct 29, 2006, Adrian Chadd wrote:
>  On Sat, Oct 28, 2006, Daniel P. Berrange wrote:
>  
>  > > Glad to know you've figured it out. Please let us know when there is
> something
>  > > in updates-testing we can try! Thanks!
>  > 
>  > It'll take me a little while to get a formal build made & pushed through
> the
>  > errata process, because I need to do some further testing before committing
>  > it to Fedora CVS. So if anyone is interested in testing in the meantime, I've
>  > put my unofficial prototype build up here http://people.redhat.com/berrange/xen/
>  > I'll remove RPMs once the formal build is in updates-testing.
>  
>  Works for me - both on a FC5 machine thats been upgraded over time and on a
>  freshly-installed FC5 that's been upgraded just once. I'm running a slightly
>  modified config on one to support >1 vlan and physical interface; I've also
>  disabled the Xen TCP services and stuck to the UNIX services. 
I spoke too soon: Now I'm seeing lots of these:

4gb seg fixup, process named (pid 1266), cs:ip 73:00561378
printk: 484562 messages suppressed.
4gb seg fixup, process poller.php (pid 1663), cs:ip 73:00540b0b
printk: 1448 messages suppressed.
4gb seg fixup, process syslogd (pid 1235), cs:ip 73:00b0e81a

I'm running Ubuntu/Debian in some of my test Xen DomU's. This has apparently been
fixed in 'etch' (with a "xen-aware" libc6) but why did this change
between
revision 2187 and revision 2200 of the xen kernels?

I can't go around and upgrade all of these machines running older OSes and I
can't
stick with the older kernels for (possible) security reasons.



Adrian

[Daniel P. Berrange]

On Sun, Oct 29, 2006 at 10:23:20AM +0800, Adrian Chadd wrote:
>  On Sun, Oct 29, 2006, Adrian Chadd wrote:
>  > On Sat, Oct 28, 2006, Daniel P. Berrange wrote:
>  > 
>  > > > Glad to know you've figured it out. Please let us know when there is
> something
>  > > > in updates-testing we can try! Thanks!
>  > > 
>  > > It'll take me a little while to get a formal build made & pushed
> through the
>  > > errata process, because I need to do some further testing before committing
>  > > it to Fedora CVS. So if anyone is interested in testing in the meantime,
> I've
>  > > put my unofficial prototype build up here
> http://people.redhat.com/berrange/xen/
>  > > I'll remove RPMs once the formal build is in updates-testing.
>  > 
>  > Works for me - both on a FC5 machine thats been upgraded over time and on a
>  > freshly-installed FC5 that's been upgraded just once. I'm running a
> slightly
>  > modified config on one to support >1 vlan and physical interface; I've also
>  > disabled the Xen TCP services and stuck to the UNIX services.
>  
>  I spoke too soon: Now I'm seeing lots of these:
>  
>  4gb seg fixup, process named (pid 1266), cs:ip 73:00561378
>  printk: 484562 messages suppressed.
>  4gb seg fixup, process poller.php (pid 1663), cs:ip 73:00540b0b
>  printk: 1448 messages suppressed.
>  4gb seg fixup, process syslogd (pid 1235), cs:ip 73:00b0e81a
>  
>  I'm running Ubuntu/Debian in some of my test Xen DomU's. This has apparently
> been
>  fixed in 'etch' (with a "xen-aware" libc6) but why did this change
> between
>  revision 2187 and revision 2200 of the xen kernels? 
This issue has been present in Xen kernels ever since the 3.0.x series, if not
before. Basically libc uses -ve segment addressing, which means Xen hypervisor
has to do some magic tricks with segmentation to make things work. These tricks,
however, have a horrific performance impact on the guests so the Xen kernels
will warn about any process which does this -ve addressing. 

Since the vast majority of apps in Fedora are now fixed to not do -ve segment
addressing by default, we have the current kernels setup to print this warning
for any process which still uses the slow form of addressing, enabling easy
identification of apps which need fixing.

It sounds like you are using the Fedora kernels with a Debian distro which 
doesn't have any of the Xen fixups, hence you're seeing these warnings that
wouldn't normally be seen by Fedora users.

There's only really two solutions - either install a newer libc which has a 
'nosegneg' variant, or get rid of the existing 'tls' variant of libc and
have
the guests use the older plain 'i686'  variant which doesn't do -ve segment
addressing.

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  [phone] -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|

>  On Sun, Oct 29, 2006 at 10:23:20AM +0800, Adrian Chadd wrote:
> > On Sun, Oct 29, 2006, Adrian Chadd wrote:
> > > On Sat, Oct 28, 2006, Daniel P. Berrange wrote:
> >
> >  
> > > > > Glad to know you've figured it out.
> Please let us know when there
> > is something
> > > > > in updates-testing we can try! Thanks!
> > >
> >  
> > > > It'll take me a
> little while to get a formal build made & pushed
> > through the
> > > > errata process, because I need to do some further testing before
> > committing
> > > > it to Fedora CVS. So if anyone is interested in testing in the
> > meantime, I've
> > > > put my unofficial prototype build up here
> > http://people.redhat.com/berrange/xen/
> > > > I'll remove RPMs once the formal build is in updates-testing.
> >
> >  
> > > Works for me - both on a FC5 machine thats been
> upgraded over time and
> > on a
> > > freshly-installed FC5 that's been upgraded just once. I'm running a
> > slightly
> > > modified config on one to support >1 vlan and physical interface; I've
> > also
> > > disabled the Xen TCP services and stuck to the UNIX services.
> >  
> > I spoke too soon: Now I'm seeing lots of these:
> >  
> > 4gb seg fixup, process named (pid 1266), cs:ip
> 73:00561378
> > printk: 484562 messages suppressed.
> > 4gb seg fixup, process poller.php (pid 1663), cs:ip 73:00540b0b
> > printk: 1448 messages suppressed.
> > 4gb seg fixup, process syslogd (pid 1235), cs:ip 73:00b0e81a
> >  
> > I'm running Ubuntu/Debian in some of my test Xen
> DomU's. This has
> > apparently been
> > fixed in 'etch' (with a "xen-aware" libc6) but why did this change
> > between
> > revision 2187 and revision 2200 of the xen kernels?
>
>  This issue has been present in Xen kernels ever since the 3.0.x series, if
>  not
>  before. Basically libc uses -ve segment addressing, which means Xen
>  hypervisor
>  has to do some magic tricks with segmentation to make things work. These
>  tricks,
>  however, have a horrific performance impact on the guests so the Xen
>  kernels
>  will warn about any process which does this -ve addressing.
>
>  Since the vast majority of apps in Fedora are now fixed to not do -ve
>  segment
>  addressing by default, we have the current kernels setup to print this
>  warning
>  for any process which still uses the slow form of addressing, enabling
>  easy
>  identification of apps which need fixing.
>
>  It sounds like you are using the Fedora kernels with a Debian distro which
>  doesn't have any of the Xen fixups, hence you're seeing these warnings
>  that
>  wouldn't normally be seen by Fedora users.
>
>  There's only really two solutions - either install a newer libc which has
>  a
>  'nosegneg' variant, or get rid of the existing 'tls' variant of libc and
>  have
>  the guests use the older plain 'i686'  variant which doesn't do -ve
>  segment
>  addressing.
>
>  Regards,
>  Dan.
>  --
>  |=- Red Hat, Engineering, Emerging Technologies, Boston.  [phone]
>  -=|
>  |=-           Perl modules: http://search.cpan.org/~danberr/
>  -=|
>  |=-               Projects: http://freshmeat.net/~danielpb/
>  -=|
>  |=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505
>  -=|
>
>  --
>  Fedora-xen mailing list
>  Fedora-xen(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-xen 
Unfortunately, I'm seeing the same behavior (printk messages suppressed,
4GB fixup, etc.) on a completely "stock" FC5 install. No debian or Ubuntu
here...

My dom0 now runs xm commands, but starting any guest results in the
messages and the guest never completes the boot process -- no login
prompt.

>

[Daniel P. Berrange]

On Sun, Oct 29, 2006 at 01:19:53PM -0800, master(a)bradleyland.com wrote:
>  > On Sun, Oct 29, 2006 at 10:23:20AM +0800, Adrian Chadd wrote:
>  >> On Sun, Oct 29, 2006, Adrian Chadd wrote:
>  >> > On Sat, Oct 28, 2006, Daniel P. Berrange wrote:
>  > This issue has been present in Xen kernels ever since the 3.0.x series, if
>  > not
>  > before. Basically libc uses -ve segment addressing, which means Xen
>  > hypervisor
>  > has to do some magic tricks with segmentation to make things work. These
>  > tricks,
>  > however, have a horrific performance impact on the guests so the Xen
>  > kernels
>  > will warn about any process which does this -ve addressing.
>  >
>  > Since the vast majority of apps in Fedora are now fixed to not do -ve
>  > segment
>  > addressing by default, we have the current kernels setup to print this
>  > warning
>  > for any process which still uses the slow form of addressing, enabling
>  > easy
>  > identification of apps which need fixing.
>  >
>  > It sounds like you are using the Fedora kernels with a Debian distro which
>  > doesn't have any of the Xen fixups, hence you're seeing these warnings
>  > that
>  > wouldn't normally be seen by Fedora users.
>  >
>  > There's only really two solutions - either install a newer libc which has
>  > a
>  > 'nosegneg' variant, or get rid of the existing 'tls' variant of libc
> and
>  > have
>  > the guests use the older plain 'i686'  variant which doesn't do -ve
>  > segment
>  > addressing.
>  
>  Unfortunately, I'm seeing the same behavior (printk messages suppressed,
>  4GB fixup, etc.) on a completely "stock" FC5 install. No debian or Ubuntu
>  here...
>  
>  My dom0 now runs xm commands, but starting any guest results in the
>  messages and the guest never completes the boot process -- no login
>  prompt. 
Ok, so you're seeing these messages in the guest only - you're not seeing
any of these messages in the Dom0 host too ?

Can you check to see whether ldconfig has correctly cached the nosegneg
versions of the libraries. For example, you should see something similar
to this:

# ldconfig -p | grep segneg
    libthread_db.so.1 (libc6, hwcap: 0x0018000000000000, OS ABI: Linux 2.6.9) =>
/lib/i686/nosegneg/libthread_db.so.1
    librt.so.1 (libc6, hwcap: 0x0018000000000000, OS ABI: Linux 2.6.9) =>
/lib/i686/nosegneg/librt.so.1
    libpthread.so.0 (libc6, hwcap: 0x0018000000000000, OS ABI: Linux 2.6.9) =>
/lib/i686/nosegneg/libpthread.so.0
    libm.so.6 (libc6, hwcap: 0x0018000000000000, OS ABI: Linux 2.6.9) =>
/lib/i686/nosegneg/libm.so.6
    libc.so.6 (libc6, hwcap: 0x0018000000000000, OS ABI: Linux 2.6.9) =>
/lib/i686/nosegneg/libc.so.6
 
There should be a file in /etc/ld.so.conf.d/kernelcaps-???? matching your
live kernel version which has the following setting:
 
  hwcap 0 nosegneg

If ldconfig doesn't show the nosegneg versions, try forcing it to rebuild
the cache, by just running 'ldconfig' with no args.

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  [phone] -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|

[Adrian Chadd]

On Sun, Oct 29, 2006, Daniel P. Berrange wrote:

>  > Unfortunately, I'm seeing the same behavior (printk messages
> suppressed,
>  > 4GB fixup, etc.) on a completely "stock" FC5 install. No debian or Ubuntu
>  > here...
>  > 
>  > My dom0 now runs xm commands, but starting any guest results in the
>  > messages and the guest never completes the boot process -- no login
>  > prompt.
>  
>  Ok, so you're seeing these messages in the guest only - you're not seeing
>  any of these messages in the Dom0 host too ? 
ACPI: Power Button (FF) [PWRF]
ACPI: Power Button (CM) [PWRB]
ACPI: Sleep Button (CM) [SLPB]
ibm_acpi: ec object not found
printk: 29482 messages suppressed.
4gb seg fixup, process nash (pid 950), cs:ip 73:080f3a06
4gb seg fixup, process nash (pid 950), cs:ip 73:080f692d
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
device-mapper: ioctl: 4.7.0-ioctl (2006-06-24) initialised: dm-devel(a)redhat.com
EXT3 FS on sda2, internal journal
kjournald starting.  Commit interval 5 seconds
EXT3 FS on hda1, internal journal
EXT3-fs: mounted filesystem with ordered data mode.

Thats during bootup before / is mounted..

There's more messages but I can't tell which are due to FC5 (some of them)
and which are due to my chroot-into-debian-install-to-try-and-fix mucking
about.



Adrian

Correct -- it seems the messages only appear on the guest's console during
boot. No messages like that on dom0. The results of the commands you
suggested look as expected(?):

[root@zeus ~]# ldconfig -p|grep segneg
        libthread_db.so.1 (libc6, hwcap: 0x0018000000000000, OS ABI: Linux
2.6.9) => /lib/i686/nosegneg/libthread_db.so.1
        librt.so.1 (libc6, hwcap: 0x0018000000000000, OS ABI: Linux 2.6.9)
=> /lib/i686/nosegneg/librt.so.1
        libpthread.so.0 (libc6, hwcap: 0x0018000000000000, OS ABI: Linux
2.6.9) => /lib/i686/nosegneg/libpthread.so.0
        libm.so.6 (libc6, hwcap: 0x0018000000000000, OS ABI: Linux 2.6.9)
=> /lib/i686/nosegneg/libm.so.6
        libc.so.6 (libc6, hwcap: 0x0018000000000000, OS ABI: Linux 2.6.9)
=> /lib/i686/nosegneg/libc.so.6

[root@zeus ld.so.conf.d]# cat kernelcap-2.6.18-1.2200.fc5.conf
# This directive teaches ldconfig to search in nosegneg subdirectories
# and cache the DSOs there with extra bit 0 set in their hwcap match
# fields.  In Xen guest kernels, the vDSO tells the dynamic linker to
# search in nosegneg subdirectories and to match this extra hwcap bit
# in the ld.so.cache file.
hwcap 0 nosegneg

[root@zeus ld.so.conf.d]# uname -a
Linux zeus.bradleyland.com 2.6.18-1.2200.fc5xen0 #1 SMP Sat Oct 14
17:49:47 EDT 2006 i686 athlon i386 GNU/Linux

I tried running ldconfig without arguments in any case and then tried to
start a guest. I don't have the modules for the kernel in the guest yet,
but it should still boot. Here's the results:

0MB HIGHMEM available.
136MB LOWMEM available.
Using x86 segment limits to approximate NX protection
Built 1 zonelists.  Total pages: 34816
Kernel command line:  root=/dev/sda1 ro 3
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
CPU 0 irqstacks, hard=c06df000 soft=c06bf000
PID hash table entries: 1024 (order: 10, 4096 bytes)
Xen reported: 1460.452 MHz processor.
Console: colour dummy device 80x25
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Software IO TLB disabled
vmalloc area: c9000000-fb3fe000, maxmem 33ffe000
Memory: 122664k/139264k available (1721k kernel code, 8268k reserved, 634k
data, 144k init, 0k highmem)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
Calibrating delay using timer specific routine.. 3664.60 BogoMIPS
(lpj=7329214)
Security Framework v1.0.0 initialized
SELinux:  Initializing.
SELinux:  Starting in permissive mode
selinux_register_security:  Registering secondary module capability
Capability LSM initialized as secondary
Mount-cache hash table entries: 512
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 256K (64 bytes/line)
Checking 'hlt' instruction... OK.
SMP alternatives: switching to UP code
Freeing SMP alternatives: 12k freed
Brought up 1 CPUs
checking if image is initramfs... it is
Freeing initrd memory: 3203k freed
Grant table initialized
NET: Registered protocol family 16
Brought up 1 CPUs
xen_mem: Initialising balloon driver.
NET: Registered protocol family 2
IP route cache hash table entries: 2048 (order: 1, 8192 bytes)
TCP established hash table entries: 8192 (order: 5, 163840 bytes)
TCP bind hash table entries: 4096 (order: 4, 81920 bytes)
TCP: Hash tables configured (established 8192 bind 4096)
TCP reno registered
audit: initializing netlink socket (disabled)
audit(1162178779.185:1): initialized
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
SELinux:  Registering netfilter hooks
Initializing Cryptographic API
ksign: Installing public key data
Loading keyring
- Added public key AFB81CC8AAE721BE
- User ID: Red Hat, Inc. (Kernel Module GPG key)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
rtc: IRQ 8 is not free.
Non-volatile memory driver v1.2
RAMDISK driver initialized: 16 RAM disks of 16384K size 4096 blocksize
Xen virtual console successfully installed as xvc0
Linux version 2.6.18-1.2200.fc5xenU
(brewbuilder(a)hs20-bc2-3.build.redhat.com) (gcc version 4.1.1 20060525 (Red
Hat 4.1.1-1)) #1 SMP Sat Oct 14 18:06:38 EDT 2006
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 0000000008800000 (usable)
0MB HIGHMEM available.
136MB LOWMEM available.
Using x86 segment limits to approximate NX protection
Built 1 zonelists.  Total pages: 34816
Kernel command line:  root=/dev/sda1 ro 3
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
CPU 0 irqstacks, hard=c06df000 soft=c06bf000
PID hash table entries: 1024 (order: 10, 4096 bytes)
Xen reported: 1460.452 MHz processor.
Console: colour dummy device 80x25
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Software IO TLB disabled
vmalloc area: c9000000-fb3fe000, maxmem 33ffe000
Memory: 122664k/139264k available (1721k kernel code, 8268k reserved, 634k
data, 144k init, 0k highmem)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
Calibrating delay using timer specific routine.. 3664.60 BogoMIPS
(lpj=7329214)
Security Framework v1.0.0 initialized
SELinux:  Initializing.
SELinux:  Starting in permissive mode
selinux_register_security:  Registering secondary module capability
Capability LSM initialized as secondary
Mount-cache hash table entries: 512
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 256K (64 bytes/line)
Checking 'hlt' instruction... OK.
SMP alternatives: switching to UP code
Freeing SMP alternatives: 12k freed
Brought up 1 CPUs
checking if image is initramfs... it is
Freeing initrd memory: 3203k freed
Grant table initialized
NET: Registered protocol family 16
Brought up 1 CPUs
xen_mem: Initialising balloon driver.
NET: Registered protocol family 2
IP route cache hash table entries: 2048 (order: 1, 8192 bytes)
TCP established hash table entries: 8192 (order: 5, 163840 bytes)
TCP bind hash table entries: 4096 (order: 4, 81920 bytes)
TCP: Hash tables configured (established 8192 bind 4096)
TCP reno registered
audit: initializing netlink socket (disabled)
audit(1162178779.185:1): initialized
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
SELinux:  Registering netfilter hooks
Initializing Cryptographic API
ksign: Installing public key data
Loading keyring
- Added public key AFB81CC8AAE721BE
- User ID: Red Hat, Inc. (Kernel Module GPG key)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
rtc: IRQ 8 is not free.
Non-volatile memory driver v1.2
RAMDISK driver initialized: 16 RAM disks of 16384K size 4096 blocksize
Xen virtual console successfully installed as xvc0
Event-channel device installed.
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 50MHz system bus speed for PIO modes; override with idebus=xx
4gb seg fixup, process modprobe (pid 154), cs:ip 73:080f3a06
4gb seg fixup, process modprobe (pid 154), cs:ip 73:08117bae
4gb seg fixup, process modprobe (pid 154), cs:ip 73:080f692d
4gb seg fixup, process modprobe (pid 154), cs:ip 73:080f693b
4gb seg fixup, process modprobe (pid 154), cs:ip 73:080f692d
4gb seg fixup, process modprobe (pid 154), cs:ip 73:080f693b
4gb seg fixup, process modprobe (pid 154), cs:ip 73:08120d22
4gb seg fixup, process modprobe (pid 154), cs:ip 73:080a8501
4gb seg fixup, process modprobe (pid 154), cs:ip 73:080f692d
4gb seg fixup, process modprobe (pid 154), cs:ip 73:080f693b
Registering block device major 8
ide-floppy driver 0.99.newide
i8042.c: No controller found.
mice: PS/2 mouse device common for all mice
md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: bitmap version 4.39
TCP bic registered
Initializing IPsec netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
Using IPI No-Shortcut mode
XENBUS: Device with no driver: device/vif/0
XENBUS: Device with no driver: device/vif/1
XENBUS: Device with no driver: device/vif/2
Freeing unused kernel memory: 144k freed
Write protecting the kernel read-only data: 258k
Red Hat nash version 5.0.32 starting
Mounting proc filesystem
Mounting sysfs filesystem
Creating /dev
Creating initial device nodes
Setting up hotplug.
Creating block device nodes.
Loading dm-mod.ko module
device-mapper: ioctl: 4.7.0-ioctl (2006-06-24) initialised:
dm-devel(a)redhat.com
Loading dm-mirror.ko module
Loading dm-zero.ko module
Loading dm-snapshot.ko module
Making device-mapper control node
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
Scanning logical volumes
  Reading all physical volumes.  This may take a while...
  No volume groups found
Activating logical volumes
  Volume group "raid5" not found
Creating root device.
Mounting root filesystem.
EXT3-fs: INFO: recovery required on readonly filesystem.
EXT3-fs: write access will be enabled during recovery.
kjournald starting.  Commit interval 5 seconds
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
printk: 35012 messages suppressed.
4gb seg fixup, process init (pid 1), cs:ip 73:080f692d
Setting up other filesystems.
Setting up new root fs
no fstab.sys, mounting internal defaults
Switching to new root and running init.
unmounting old /dev
unmounting old /proc
unmounting old /sys
INIT: version 2.86 booting
                Welcome to Fedora Core
                Press 'I' to enter interactive startup.
modprobe: FATAL: Could not load
/lib/modules/2.6.18-1.2200.fc5xenU/modules.dep: No such file or directory

printk: 35210 messages suppressed.
4gb seg fixup, process hwclock (pid 202), cs:ip 73:00491036
Setting clock : Sun Oct 29 22:26:29 EST 2006 [  OK  ]
Starting udev: [  OK  ]
Setting hostname firewall.bradleyland.com:  [  OK  ]
Setting up Logical Volume Management:   No volume groups found
[  OK  ]
Checking filesystems
Checking all file systems.
[/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/sda1
/dev/sda1: clean, 32118/655360 files, 280414/1310720 blocks
[  OK  ]
Remounting root filesystem in read-write mode:  [  OK  ]
Mounting local filesystems:  [  OK  ]
Enabling local filesystem quotas:  [  OK  ]
rm: cannot remove `/var/run/heartbeat/ccm': Is a directory
rm: cannot remove `/var/run/heartbeat/crm': Is a directory
Enabling swap space:  [  OK  ]
INIT: Entering runlevel: 3
Entering non-interactive startup
Starting readahead_early:  [  OK  ]
Checking for hardware changes [  OK  ]
Bringing up loopback interface:  [  OK  ]
Bringing up interface eth0:  FATAL: Could not load
/lib/modules/2.6.18-1.2200.fc5xenU/modules.dep: No such file or directory
FATAL: Could not load /lib/modules/2.6.18-1.2200.fc5xenU/modules.dep: No
such file or directory
Device eth0 does not seem to be present, delaying initialization.
[FAILED]
Bringing up interface eth1:  FATAL: Could not load
/lib/modules/2.6.18-1.2200.fc5xenU/modules.dep: No such file or directory
FATAL: Could not load /lib/modules/2.6.18-1.2200.fc5xenU/modules.dep: No
such file or directory
xennet device eth1 does not seem to be present, delaying initialization.
[FAILED]
Bringing up interface eth2:  FATAL: Could not load
/lib/modules/2.6.18-1.2200.fc5xenU/modules.dep: No such file or directory
FATAL: Could not load /lib/modules/2.6.18-1.2200.fc5xenU/modules.dep: No
such file or directory
xennet device eth2 does not seem to be present, delaying initialization.
[FAILED]
Starting system logger: [  OK  ]
Starting kernel logger: [  OK  ]
Starting irqbalance: [  OK  ]
Starting portmap: [  OK  ]
Starting NFS statd: [  OK  ]
Starting RPC idmapd: FATAL: Could not load
/lib/modules/2.6.18-1.2200.fc5xenU/modules.dep: No such file or directory
Starting system message bus: [  OK  ]
Mounting other filesystems:  [  OK  ]
Starting hidd: Can't open HIDP control socket: Address family not
supported by protocol
[FAILED]
Starting automount: printk: 1224600 messages suppressed.
4gb seg fixup, process syslogd (pid 759), cs:ip 73:00a62036
[  OK  ]
Starting smartd: [  OK  ]
Starting sshd: [  OK  ]
Heartbeat not configured: /etc/ha.d/ha.cf not found.
Starting crond: [  OK  ]
Starting ldirectord Config file ldirectord.cf not found
[FAILED]
Starting anacron: [  OK  ]
Starting atd: [printk: 1082344 messages suppressed.
4gb seg fixup, process S95atd (pid 998), cs:ip 73:0056cc42
  OK  ]
Starting HAL daemon: [  OK  ]
printk: 108365 messages suppressed.
4gb seg fixup, process syslogd (pid 759), cs:ip 73:00ae9e3a
4gb seg fixup, process syslogd (pid 759), cs:ip 73:00ae9db0
4gb seg fixup, process klogd (pid 762), cs:ip 73:00b0b036
4gb seg fixup, process klogd (pid 762), cs:ip 73:00b8b000
printk: 18 messages suppressed.
4gb seg fixup, process syslogd (pid 759), cs:ip 73:00ae9e3a
4gb seg fixup, process klogd (pid 762), cs:ip 73:00b0b036
4gb seg fixup, process klogd (pid 762), cs:ip 73:00b8b000
4gb seg fixup, process klogd (pid 762), cs:ip 73:00b0b036
4gb seg fixup, process klogd (pid 762), cs:ip 73:00b8b000
4gb seg fixup, process klogd (pid 762), cs:ip 73:00b0b036
printk: 25 messages suppressed.
4gb seg fixup, process syslogd (pid 759), cs:ip 73:00ae9e3a
4gb seg fixup, process klogd (pid 762), cs:ip 73:00b0b036
4gb seg fixup, process klogd (pid 762), cs:ip 73:00b8b000
4gb seg fixup, process klogd (pid 762), cs:ip 73:00b0b036
4gb seg fixup, process klogd (pid 762), cs:ip 73:00b8b000
4gb seg fixup, process klogd (pid 762), cs:ip 73:00b0b036

and so on...

[Robert Story]

On Sun, 29 Oct 2006 23:27:03 +0000 Daniel wrote:
DPB> > Unfortunately, I'm seeing the same behavior (printk messages suppressed,
DPB> > 4GB fixup, etc.) on a completely "stock" FC5 install. No debian or
Ubuntu
DPB> > here...
DPB> 
DPB> Ok, so you're seeing these messages in the guest only - you're not seeing
DPB> any of these messages in the Dom0 host too ?

I've been seeing these in my Dom0 syslog, but figured I wanted to at least get
a guest booting before I complained about anything else..


DPB> Can you check to see whether ldconfig has correctly cached the nosegneg
DPB> versions of the libraries. For example, you should see something similar
DPB> to this:
DPB> 
DPB> # ldconfig -p | grep segneg
DPB>     libthread_db.so.1 (libc6, hwcap: 0x0018000000000000, OS ABI: Linux 2.6.9)
=> /lib/i686/nosegneg/libthread_db.so.1
DPB>     librt.so.1 (libc6, hwcap: 0x0018000000000000, OS ABI: Linux 2.6.9) =>
/lib/i686/nosegneg/librt.so.1
DPB>     libpthread.so.0 (libc6, hwcap: 0x0018000000000000, OS ABI: Linux 2.6.9) =>
/lib/i686/nosegneg/libpthread.so.0
DPB>     libm.so.6 (libc6, hwcap: 0x0018000000000000, OS ABI: Linux 2.6.9) =>
/lib/i686/nosegneg/libm.so.6
DPB>     libc.so.6 (libc6, hwcap: 0x0018000000000000, OS ABI: Linux 2.6.9) =>
/lib/i686/nosegneg/libc.so.6
DPB>  
DPB> There should be a file in /etc/ld.so.conf.d/kernelcaps-???? matching your
DPB> live kernel version which has the following setting:
DPB>  
DPB>   hwcap 0 nosegneg

I get that output from the grep, and have that file w/those contents.


One 'odd' thing about this config.. the initial install and testing was done
on a PIII, but the disc was moved to another machine and is now running on an
AMD Athalon....

[Daniel P. Berrange]

On Mon, Oct 30, 2006 at 10:26:02AM -0500, Robert Story wrote:
>  On Sun, 29 Oct 2006 23:27:03 +0000 Daniel wrote:
>  DPB> > Unfortunately, I'm seeing the same behavior (printk messages
> suppressed,
>  DPB> > 4GB fixup, etc.) on a completely "stock" FC5 install. No debian or
> Ubuntu
>  DPB> > here...
>  DPB> 
>  DPB> Ok, so you're seeing these messages in the guest only - you're not
> seeing
>  DPB> any of these messages in the Dom0 host too ?
>  
>  I've been seeing these in my Dom0 syslog, but figured I wanted to at least get
>  a guest booting before I complained about anything else..
>  
>  
>  DPB> Can you check to see whether ldconfig has correctly cached the nosegneg
>  DPB> versions of the libraries. For example, you should see something similar
>  DPB> to this:
>  DPB> 
>  DPB> # ldconfig -p | grep segneg
>  DPB>     libthread_db.so.1 (libc6, hwcap: 0x0018000000000000, OS ABI: Linux 2.6.9)
> => /lib/i686/nosegneg/libthread_db.so.1
>  DPB>     librt.so.1 (libc6, hwcap: 0x0018000000000000, OS ABI: Linux 2.6.9) =>
> /lib/i686/nosegneg/librt.so.1
>  DPB>     libpthread.so.0 (libc6, hwcap: 0x0018000000000000, OS ABI: Linux 2.6.9) =>
> /lib/i686/nosegneg/libpthread.so.0
>  DPB>     libm.so.6 (libc6, hwcap: 0x0018000000000000, OS ABI: Linux 2.6.9) =>
> /lib/i686/nosegneg/libm.so.6
>  DPB>     libc.so.6 (libc6, hwcap: 0x0018000000000000, OS ABI: Linux 2.6.9) =>
> /lib/i686/nosegneg/libc.so.6
>  DPB>  
>  DPB> There should be a file in /etc/ld.so.conf.d/kernelcaps-???? matching your
>  DPB> live kernel version which has the following setting:
>  DPB>  
>  DPB>   hwcap 0 nosegneg
>  
>  I get that output from the grep, and have that file w/those contents. 
Ok, I'm in the middle of re-installing one of my machines to the 32-bit FC5
instead of 64-bit, so with a little luck I should be able to reproduce the
problem shortly & which is one step closer to finding a fix :-)

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  [phone] -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|

[Robert Story]

On Sat, 28 Oct 2006 15:40:15 +0100 Daniel wrote:
DPB> > Glad to know you've figured it out. Please let us know when there is
something
DPB> > in updates-testing we can try! Thanks!
DPB> 
DPB> It'll take me a little while to get a formal build made & pushed through
the
DPB> errata process, because I need to do some further testing before committing
DPB> it to Fedora CVS. So if anyone is interested in testing in the meantime, I've
DPB> put my unofficial prototype build up here http://people.redhat.com/berrange/xen/
DPB> I'll remove RPMs once the formal build is in updates-testing.

I've just installed the i386 xen, and happily report that my guest domains are
now back up! Thanks!

What magic did you use to get it to work? I installed the test RPMs as
well. dom0 is now working, but the guests don't boot (see prior posts).

I sure hope there's a solution soon -- my main VM host has been down ever
since I installed FC5 months ago.

[Daniel P. Berrange]

On Sun, Oct 29, 2006 at 07:34:21PM -0800, master(a)bradleyland.com wrote:
>  
>  Correct -- it seems the messages only appear on the guest's console during
>  boot. No messages like that on dom0. The results of the commands you
>  suggested look as expected(?):
>  
>  [root@zeus ~]# ldconfig -p|grep segneg
>          libthread_db.so.1 (libc6, hwcap: 0x0018000000000000, OS ABI: Linux
>  2.6.9) => /lib/i686/nosegneg/libthread_db.so.1
>          librt.so.1 (libc6, hwcap: 0x0018000000000000, OS ABI: Linux 2.6.9)
>  => /lib/i686/nosegneg/librt.so.1
>          libpthread.so.0 (libc6, hwcap: 0x0018000000000000, OS ABI: Linux
>  2.6.9) => /lib/i686/nosegneg/libpthread.so.0
>          libm.so.6 (libc6, hwcap: 0x0018000000000000, OS ABI: Linux 2.6.9)
>  => /lib/i686/nosegneg/libm.so.6
>          libc.so.6 (libc6, hwcap: 0x0018000000000000, OS ABI: Linux 2.6.9)
>  => /lib/i686/nosegneg/libc.so.6
>  
>  [root@zeus ld.so.conf.d]# cat kernelcap-2.6.18-1.2200.fc5.conf
>  # This directive teaches ldconfig to search in nosegneg subdirectories
>  # and cache the DSOs there with extra bit 0 set in their hwcap match
>  # fields.  In Xen guest kernels, the vDSO tells the dynamic linker to
>  # search in nosegneg subdirectories and to match this extra hwcap bit
>  # in the ld.so.cache file.
>  hwcap 0 nosegneg
>  
>  [root@zeus ld.so.conf.d]# uname -a
>  Linux zeus.bradleyland.com 2.6.18-1.2200.fc5xen0 #1 SMP Sat Oct 14
>  17:49:47 EDT 2006 i686 athlon i386 GNU/Linux
>  
>  I tried running ldconfig without arguments in any case and then tried to
>  start a guest. I don't have the modules for the kernel in the guest yet,
>  but it should still boot. Here's the results:
>  
>  Starting hidd: Can't open HIDP control socket: Address family not
>  supported by protocol
>  [FAILED]
>  Starting automount: printk: 1224600 messages suppressed.
>  4gb seg fixup, process syslogd (pid 759), cs:ip 73:00a62036
>  [  OK  ]
>  Starting smartd: [  OK  ]
>  Starting sshd: [  OK  ]
>  Heartbeat not configured: /etc/ha.d/ha.cf not found.
>  Starting crond: [  OK  ]
>  Starting ldirectord Config file ldirectord.cf not found
>  [FAILED]
>  Starting anacron: [  OK  ]
>  Starting atd: [printk: 1082344 messages suppressed.
>  4gb seg fixup, process S95atd (pid 998), cs:ip 73:0056cc42
>    OK  ]
>  Starting HAL daemon: [  OK  ] 
Ok, I've figured out what's up here - HAL is actually one of the last, if
not the last, init script to be run. So it has actually completed booting
at this point & is running. The 2.6.18 kernel though now has the paravirt
framebuffer support integrated. This means the primary console is now the
graphical console, not the text console. If you happen to have a fixed (or
otherwise known) IP for the guest you should be able to login with SSH & 
see that its all running. The thing is the mingetty proesses are now 
spawning on the framebuffer instead, so as for fullyvirt, you need to 
turn on the VNC support int he guest config 'vnc=1' and one or more of
vncunused=1, vnclisten="0.0.0.0", vncpassword='123456'.

To make it launch a login prompt on the serial console too, it is neccessary
to have an extra line in /etc/inittab, ie

  co:2345:respawn:/sbin/agetty xvc0 9600 vt100-nav

Ordinaryily kudzu should make such an addition automatically upon bootup
if it detects a Xen guest, so I am co-ordinating to get an update for kudzu
too.

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  [phone] -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|

[Robert Story]

On Mon, 30 Oct 2006 11:51:42 -0800 (PST) master(a)bradleyland.com wrote:
MC> What magic did you use to get it to work? I installed the test RPMs as
MC> well. dom0 is now working, but the guests don't boot (see prior posts).
MC> 
MC> I sure hope there's a solution soon -- my main VM host has been down ever
MC> since I installed FC5 months ago.

No magic.. just 'rpm -Fvh xen-*', then 'service xend stop' and
'service xend
start'. After that, 'xm list' listed dom0, and I manually started my domU. I
didn't even have to reboot.

I was getting some errors related to 'log_success_msg' or some such, but I just
ignored them.

[Daniel P. Berrange]

On Mon, Oct 30, 2006 at 11:51:42AM -0800, master(a)bradleyland.com wrote:
>  What magic did you use to get it to work? I installed the test RPMs
> as
>  well. dom0 is now working, but the guests don't boot (see prior posts).
>  
>  I sure hope there's a solution soon -- my main VM host has been down ever
>  since I installed FC5 months ago. 
I just posted a response to your original messages - I think its very
likely to be the agetty process not being spawned on the serial console,
since the kernels now have a proper framebuffer console.

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  [phone] -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|

Thanks for the info Dan. Makes sense. On the road all this week; when I
get back, I'll give it a try.