Not seeing all RAM

[Ranbir]

Hello,

I'm running a FC6 dom0, with kernel version,
kernel-xen-2.6.19-1.2895.fc6 on a Dell PowerEdge 1600SC.  I just
upgraded it from 1 GB (2 x 512 MB modules) to 3 GB (2 x 1 GB + 2 x 512
MB).  The BIOS sees all of the ram, but kernel does not.  All I see when
I run free is this:

             total       used       free     shared    buffers
cached
Mem:       2454528     207876    2246652          0      11032
47380
-/+ buffers/cache:     149464    2305064
Swap:       524280          0     524280

It looks like 512 MB is missing.

Here's a bit more detail from dmesg:

Memory: 2874124k/2987900k available (2166k kernel code, 104532k
reserved, 1100k data, 184k init, 2242436k highmem)

So, what happened to the rest of the RAM?  I should add that the server
is updated with the latest firmware and BIOS.

BTW, I appear to be having problems with one of my domUs now too (after
the ram upgrade).  A PCI device I was handing to the domU is now causing
the domU kernel to panic, crashing the domU.  I'll try to get a copy of
the error message.

Any help appreciated.

Regards,

Ranbir
-- 
Kanwar Ranbir Sandhu
Linux 2.6.20-1.2944.fc6 i686 GNU/Linux 
00:17:27 up 10:25, 2 users, load average: 0.68, 0.43, 0.26

[itamar]

try a new xen kernel

run a yum update


--------------------

Itamar Reis Peixoto

e-mail: itamar(a)ispbrasil.com.br
msn: itamarjp(a)starmedia.com
skype: itamarjp 
icq: 81053601
[phone]
[phone]
----- Original Message ----- 
From: "Kanwar Ranbir Sandhu" <m3freak(a)thesandhufamily.ca&gt;
To: <fedora-xen(a)redhat.com&gt;
Sent: Tuesday, July 03, 2007 1:27 AM
Subject: [Fedora-xen] Not seeing all RAM


>  Hello,
>  
>  I'm running a FC6 dom0, with kernel version,
>  kernel-xen-2.6.19-1.2895.fc6 on a Dell PowerEdge 1600SC.  I just
>  upgraded it from 1 GB (2 x 512 MB modules) to 3 GB (2 x 1 GB + 2 x 512
>  MB).  The BIOS sees all of the ram, but kernel does not.  All I see when
>  I run free is this:
>  
>              total       used       free     shared    buffers
>  cached
>  Mem:       2454528     207876    2246652          0      11032
>  47380
>  -/+ buffers/cache:     149464    2305064
>  Swap:       524280          0     524280
>  
>  It looks like 512 MB is missing.
>  
>  Here's a bit more detail from dmesg:
>  
>  Memory: 2874124k/2987900k available (2166k kernel code, 104532k
>  reserved, 1100k data, 184k init, 2242436k highmem)
>  
>  So, what happened to the rest of the RAM?  I should add that the server
>  is updated with the latest firmware and BIOS.
>  
>  BTW, I appear to be having problems with one of my domUs now too (after
>  the ram upgrade).  A PCI device I was handing to the domU is now causing
>  the domU kernel to panic, crashing the domU.  I'll try to get a copy of
>  the error message.
>  
>  Any help appreciated.
>  
>  Regards,
>  
>  Ranbir
>  -- 
>  Kanwar Ranbir Sandhu
>  Linux 2.6.20-1.2944.fc6 i686 GNU/Linux 
>  00:17:27 up 10:25, 2 users, load average: 0.68, 0.43, 0.26 
>  
>  
>  --
>  Fedora-xen mailing list
>  Fedora-xen(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-xen
>

[Richard W.M. Jones]

Kanwar Ranbir Sandhu wrote:
>  Hello,
>  
>  I'm running a FC6 dom0, with kernel version,
>  kernel-xen-2.6.19-1.2895.fc6 on a Dell PowerEdge 1600SC.  I just
>  upgraded it from 1 GB (2 x 512 MB modules) to 3 GB (2 x 1 GB + 2 x 512
>  MB). 
If this still happens after you've updated to the latest FC6 (IIRC, 
kernel-xen 2.6.20-1.2952) then please file a bug here: 
https://bugzilla.redhat.com/bugzilla/index.cgi

There are various problems with Xen & memory, although they don't 
normally happen below the 4 GB boundary (and mostly happen for people 
with absolutely enormous amounts of memory, like > 60 GB).  We'll try 
and fix them.

Rich.

-- 
Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/
Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod
Street, Windsor, Berkshire, SL4 1TE, United Kingdom.  Registered in
England and Wales under Company Registration No. 03798903

[Ranbir]

On Tue, 2007-07-03 at 10:17 +0100, Richard W.M. Jones wrote:
>  If this still happens after you've updated to the latest FC6
> (IIRC, 
>  kernel-xen 2.6.20-1.2952) then please file a bug here: 
>  https://bugzilla.redhat.com/bugzilla/index.cgi 
Some improvement, but still missing RAM:

[root@xenmaster ~]# free
             total       used       free     shared    buffers
cached
Mem:       2716672     211452    2505220          0      11056
48608
-/+ buffers/cache:     151788    2564884
Swap:       524280          0     524280

[root@xenmaster ~]# grep -i memory /var/log/dmesg 
Memory: 2874636k/2987900k available (2135k kernel code, 104060k
reserved, 1130k data, 176k init, 2242436k highmem)

The BIOS reports 3072 MB.

I'll file that bug report.

Regards,

Ranbir
-- 
Kanwar Ranbir Sandhu
Linux 2.6.20-1.2944.fc6 i686 GNU/Linux 
10:06:21 up 20:14, 2 users, load average: 0.18, 0.20, 0.23

[Daniel P. Berrange]

On Tue, Jul 03, 2007 at 12:27:54AM -0400, Kanwar Ranbir Sandhu wrote:
>  Hello,
>  
>  I'm running a FC6 dom0, with kernel version,
>  kernel-xen-2.6.19-1.2895.fc6 on a Dell PowerEdge 1600SC.  I just
>  upgraded it from 1 GB (2 x 512 MB modules) to 3 GB (2 x 1 GB + 2 x 512
>  MB).  The BIOS sees all of the ram, but kernel does not.  All I see when
>  I run free is this:
>  
>               total       used       free     shared    buffers
>  cached
>  Mem:       2454528     207876    2246652          0      11032
>  47380
>  -/+ buffers/cache:     149464    2305064
>  Swap:       524280          0     524280
>  
>  It looks like 512 MB is missing. 
Let me guest - you are running a 32-bit kernel on the machine ? If so
the problem is likely that the BOIS is remapping a portion of memory
above the 4 GB mark, to make room for PCI device DMA zones. A 32-bit
kernel though can only access physical addrs upto 4 GB, so it can't
see the remapped memory.

>  BTW, I appear to be having problems with one of my domUs now too
> (after
>  the ram upgrade).  A PCI device I was handing to the domU is now causing
>  the domU kernel to panic, crashing the domU.  I'll try to get a copy of
>  the error message. 
PCI device mapping to DomUs is insanely fragile, whether it works or not
is mostly luck. In this case since you added more memory, the locaiton of
the PCI device's DMA zone probably changed and by (bad) luck is no longer
accessible to the DomU or something like that.

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  [phone] -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=|

[Ranbir]

On Tue, 2007-07-03 at 15:23 +0100, Daniel P. Berrange wrote:
>  Let me guest - you are running a 32-bit kernel on the machine ? If
> so
>  the problem is likely that the BOIS is remapping a portion of memory
>  above the 4 GB mark, to make room for PCI device DMA zones. A 32-bit
>  kernel though can only access physical addrs upto 4 GB, so it can't
>  see the remapped memory. 
I'm aware of that, but in my case it doesn't make sense: the server only
has 3 GB of RAM.  The kernel just doesn't see all of it.

I was using kernel-xen-2.6.19-1.2895.fc6 and about 512 MB were missing.
After upgrading to kernel-xen-2.6.20-1.2962.fc6, the kernel can see more
memory, but I'm still missing about 256 MB.

I'm going to open a new bug report for this.

>  PCI device mapping to DomUs is insanely fragile, whether it works or
> not
>  is mostly luck. In this case since you added more memory, the locaiton of
>  the PCI device's DMA zone probably changed and by (bad) luck is no longer
>  accessible to the DomU or something like that. 
That's what I figured, though I don't know all of the details.  However,
with the new kernel being able to see more of the 3 GB, the domU is
working again.  The pci device I'm hiding from dom0 is being properly
handed to the domU.

Regards,

Ranbir
-- 
Kanwar Ranbir Sandhu
Linux 2.6.20-1.2944.fc6 i686 GNU/Linux 
10:42:35 up 20:50, 2 users, load average: 2.15, 2.16, 1.45

[itamar]

with non xen kernel are you able to see all amount of memory ?

have you take a look on your bios to check if your video card is not using 
your memory ?


--------------------

Itamar Reis Peixoto

e-mail: itamar(a)ispbrasil.com.br
msn: itamarjp(a)starmedia.com
skype: itamarjp
icq: 81053601
[phone]
[phone]
----- Original Message ----- 
From: "Kanwar Ranbir Sandhu" <m3freak(a)thesandhufamily.ca&gt;
To: "Daniel P. Berrange" <berrange(a)redhat.com&gt;
Cc: <fedora-xen(a)redhat.com&gt;
Sent: Tuesday, July 03, 2007 11:42 AM
Subject: Re: [Fedora-xen] Not seeing all RAM


>  On Tue, 2007-07-03 at 15:23 +0100, Daniel P. Berrange wrote:
> > Let me guest - you are running a 32-bit kernel on the machine ? If so
> > the problem is likely that the BOIS is remapping a portion of memory
> > above the 4 GB mark, to make room for PCI device DMA zones. A 32-bit
> > kernel though can only access physical addrs upto 4 GB, so it can't
> > see the remapped memory.
>
>  I'm aware of that, but in my case it doesn't make sense: the server only
>  has 3 GB of RAM.  The kernel just doesn't see all of it.
>
>  I was using kernel-xen-2.6.19-1.2895.fc6 and about 512 MB were missing.
>  After upgrading to kernel-xen-2.6.20-1.2962.fc6, the kernel can see more
>  memory, but I'm still missing about 256 MB.
>
>  I'm going to open a new bug report for this.
>
> > PCI device mapping to DomUs is insanely fragile, whether it works or not
> > is mostly luck. In this case since you added more memory, the locaiton of
> > the PCI device's DMA zone probably changed and by (bad) luck is no longer
> > accessible to the DomU or something like that.
>
>  That's what I figured, though I don't know all of the details.  However,
>  with the new kernel being able to see more of the 3 GB, the domU is
>  working again.  The pci device I'm hiding from dom0 is being properly
>  handed to the domU.
>
>  Regards,
>
>  Ranbir
>  -- 
>  Kanwar Ranbir Sandhu
>  Linux 2.6.20-1.2944.fc6 i686 GNU/Linux
>  10:42:35 up 20:50, 2 users, load average: 2.15, 2.16, 1.45
>
>
>  --
>  Fedora-xen mailing list
>  Fedora-xen(a)redhat.com
>  https://www.redhat.com/mailman/listinfo/fedora-xen
>

[Ranbir]

On Tue, 2007-07-03 at 14:03 -0300, Itamar Reis Peixoto wrote:
>  with non xen kernel are you able to see all amount of memory ?
I haven't tried that, but I will tonight when I get home.  I'm expecting
the non-xen kernel to see all the RAM.

>  have you take a look on your bios to check if your video card is not
> using 
>  your memory ? 
It could be, but even if it was, it wouldn't be much.  I'm running the
dom0 on a real server, and the embedded graphics controller is very
simple.  At most, it's probably using 8 MB if it doesn't have it's own
dedicated memory.

I'm fairly certain the real problem is a bug in Fedora Core 6's xen
kernel.  When I upgraded to the lastest kernel version, the kernel
picked up more of the RAM.  There's still about 256 missing, but it's
better than the 512 MB that was missing with the older xen kernel.

Regards,

Ranbir
-- 
Kanwar Ranbir Sandhu
Linux 2.6.20-1.2962.fc6 i686 GNU/Linux 
16:54:39 up 1 day, 5:38, 3 users, load average: 0.20, 0.11, 0.09

[Mikael Bak]

On Wednesday 04 July 2007 22:55:01 Kanwar Ranbir Sandhu wrote:
>  On Tue, 2007-07-03 at 14:03 -0300, Itamar Reis Peixoto wrote:
>  > with non xen kernel are you able to see all amount of memory ?
>
>  I haven't tried that, but I will tonight when I get home.  I'm expecting
>  the non-xen kernel to see all the RAM.
>  
I expect that too!
I have exactly the same problem. The difference is that I'm on Debian 4.0. So 
this may not be a Fedora-specific problem at all.

If I boot the standard Debian 2.6.18-4-amd64 then I have 2061340k total 
using "top".
If I boot Xen kernel 2.6.18-4-xen-amd64 the I only get 1793024k total.

Something's eating memory.
I'm going to file a report at Debian too. Just thought I drop the note here as 
well.

HTH,
Mikael

[Ranbir]

On Thu, 2007-07-05 at 15:59 +0200, Mikael Bak wrote:
>  I expect that too!
>  I have exactly the same problem. The difference is that I'm on Debian 4.0. So 
>  this may not be a Fedora-specific problem at all.
>  
>  If I boot the standard Debian 2.6.18-4-amd64 then I have 2061340k total 
>  using "top".
>  If I boot Xen kernel 2.6.18-4-xen-amd64 the I only get 1793024k total. 
I booted the server with the non-xen kernel, and this is what free
reports:

[root@xenmaster ~]# free
             total       used       free     shared    buffers
cached
Mem:       3109504      82904    3026600          0       7084
33652
-/+ buffers/cache:      42168    3067336
Swap:       524280          0     524280

As you can see, all of the RAM is available.  So, it's most probably a
bug in the xen kernel.

Regards,

Ranbir
-- 
Kanwar Ranbir Sandhu
Linux 2.6.20-1.2962.fc6 i686 GNU/Linux 
12:52:10 up 6 days, 1:34, 3 users, load average: 0.50, 0.41, 0.31