FC4 x86_64 dies with heavy SATA RAID load

[Mark Miksis]

I have the 3Ware 9500S-4LP (running RAID5) installed in a Dell Poweredge
1800 (dual EM64T Xeon, 7520 chipset).  With Fedora Core 4 installed,
everything seems fine at first glance.  However, after a period (many
minutes) of heavy disk I/O operations, the machine begins to slow and
then becomes completely unresponsive.  I get "APIC error on CPUn:
00(40)" in dmesg which may or may not be related.  Nothing is logged to
syslog when the failures occur.  I have tried booting with noapic and
using the uP kernel.  Some of these configurations make the APIC error
go away, but the hangs still occur.  This machine will run fine for days
or more at high CPU loads as long as the I/O load to the RAID card is low.

Any thoughts?  Anyone running this configuration?

[Alexander Dalloz]

Am Do, den 30.06.2005 schrieb Mark Miksis um 1:38:

>  I have the 3Ware 9500S-4LP (running RAID5) installed in a Dell
> Poweredge
>  1800 (dual EM64T Xeon, 7520 chipset).  With Fedora Core 4 installed,
>  everything seems fine at first glance.  However, after a period (many
>  minutes) of heavy disk I/O operations, the machine begins to slow and
>  then becomes completely unresponsive.  I get "APIC error on CPUn:
>  00(40)" in dmesg which may or may not be related.  Nothing is logged to
>  syslog when the failures occur.  I have tried booting with noapic and
>  using the uP kernel.  Some of these configurations make the APIC error
>  go away, but the hangs still occur.  This machine will run fine for days
>  or more at high CPU loads as long as the I/O load to the RAID card is low. 
A known issue. Some weeks ago discussed on the CentOS list:

http://lists.centos.org/pipermail/centos/2005-May/005824.html

Don't know why the bugzilla ticket is restricted

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=121434

Alexander


-- 
Alexander Dalloz | Enger, Germany | GPG http://pgp.mit.edu 0xB366A773
legal statement: http://www.uni-x.org/legal.html
Fedora Core 2 GNU/Linux on Athlon with kernel 2.6.11-1.35_FC2smp 
Serendipity 03:36:28 up 4 days, 10:28, load average: 0.07, 0.10, 0.09

[Mark Miksis]

Alexander Dalloz wrote:
>  Am Do, den 30.06.2005 schrieb Mark Miksis um 1:38:
>  
>  
> > I have the 3Ware 9500S-4LP (running RAID5) installed in a Dell
> > Poweredge 1800 (dual EM64T Xeon, 7520 chipset).  With Fedora Core 4
> > installed, everything seems fine at first glance.  However, after a
> > period (many minutes) of heavy disk I/O operations, the machine
> > begins to slow and then becomes completely unresponsive.  I get
> > "APIC error on CPUn: 00(40)" in dmesg which may or may not be
> > related.  Nothing is logged to syslog when the failures occur.  I
> > have tried booting with noapic and using the uP kernel.  Some of
> > these configurations make the APIC error go away, but the hangs
> > still occur.  This machine will run fine for days or more at high
> > CPU loads as long as the I/O load to the RAID card is low.
>  
>  
>  A known issue. Some weeks ago discussed on the CentOS list:
>  
>  http://lists.centos.org/pipermail/centos/2005-May/005824.html
>  
>  Don't know why the bugzilla ticket is restricted
>  
>  https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=121434 

Very interesting reading - thanks for the link.  Some of those posts
suggest that the newer 3Ware firmware may "improve" the issue.
Otherwise, I guess I'm in the market for a different 4 channel RAID 5
SATA card.  Any recommendations?

[Peter Arremann]

On Wednesday 29 June 2005 21:58, Mark Miksis wrote:
>  Very interesting reading - thanks for the link.  Some of those posts
>  suggest that the newer 3Ware firmware may "improve" the issue.
>  Otherwise, I guess I'm in the market for a different 4 channel RAID 5
>  SATA card.  Any recommendations? 
Actually, I have some 9500s that are finally getting stable. Each new driver 
revision and each new firmware is better than the last and by now, I have no 
issues (also running the x86_64 but on amd hardware). 

Anyone who asks me, usually gets a 8500 controller as recommendation because 
unlike the 9500s, those are actually rock solid. You should try the firmware 
updates and maybe the driver (if there is one newer than the one in the 
kernel you're running) and see if you can't get your box stable that way - as 
I said before, it worked for me.

Also, you might want to get an AMD system next time if you have heavy IO and 
more than 4GB ram - the AMD iommu requires no bounce buffers...

Peter.

[Mark Miksis]

Peter Arremann wrote:
>  Actually, I have some 9500s that are finally getting stable. Each
> new
>  driver revision and each new firmware is better than the last and by
>  now, I have no issues (also running the x86_64 but on amd hardware).
>   
Thanks for the encouraging feedback.  It appears that the driver in FC4
is a bit old (and doesn't seem to correspond to a 3Ware release).  Are
you running SMP?
>  
>  Anyone who asks me, usually gets a 8500 controller as recommendation
>  because unlike the 9500s, those are actually rock solid. You should
>  try the firmware updates and maybe the driver (if there is one newer
>  than the one in the kernel you're running) and see if you can't get
>  your box stable that way - as I said before, it worked for me. 
I plan to compile the latest driver tomorrow...
>  
>  Also, you might want to get an AMD system next time if you have heavy
>  IO and more than 4GB ram - the AMD iommu requires no bounce
>  buffers...
>   
Interesting.  Actually, my IO needs are not *that* heavy, but I first
encountered the problem when trying to restore a bunch of backups from
my previous machine.

[Peter Arremann]

On Wednesday 29 June 2005 23:59, Mark Miksis wrote:
>  Peter Arremann wrote:
>  > Actually, I have some 9500s that are finally getting stable. Each new
>  > driver revision and each new firmware is better than the last and by
>  > now, I have no issues (also running the x86_64 but on amd hardware).
>
>  Thanks for the encouraging feedback.  It appears that the driver in FC4
>  is a bit old (and doesn't seem to correspond to a 3Ware release).  Are
>  you running SMP? 
Both - running an Athlon64, 32bit PCI slot, 2 mirrored drives and
a quad 
opteron, pci-x, 4 raid 0 drives. 

>  > Anyone who asks me, usually gets a 8500 controller as
> recommendation
>  > because unlike the 9500s, those are actually rock solid. You should
>  > try the firmware updates and maybe the driver (if there is one newer
>  > than the one in the kernel you're running) and see if you can't get
>  > your box stable that way - as I said before, it worked for me.
>
>  I plan to compile the latest driver tomorrow... 
The driver in the FC4 kernel is
2.26.02.002 (grep -i 
version /lib/modules/2.6.11-1.1369_FC4/kernel/drivers/scsi/3w-9xxx.ko) 
while the driver that amcc has is 2.26.03.015fw (grep TW_DRIVER_VERSION 
3w-9xxx.c) Its not that great a difference but it might be worth it for you. 
Don't forget to check the firmware upgrades as well. 

>  > Also, you might want to get an AMD system next time if you have
> heavy
>  > IO and more than 4GB ram - the AMD iommu requires no bounce
>  > buffers...
>
>  Interesting.  Actually, my IO needs are not *that* heavy, but I first
>  encountered the problem when trying to restore a bunch of backups from
>  my previous machine. 
High is relative... Sometimes a simple script can create a ton
of IO for just 
a few seconds and that can be enough. :-) 

Peter.

[Mark Miksis]

Peter Arremann wrote:
>  On Wednesday 29 June 2005 23:59, Mark Miksis wrote: 
> >I plan to compile the latest driver tomorrow...
>  
>  The driver in the FC4 kernel is 2.26.02.002 (grep -i 
>  version /lib/modules/2.6.11-1.1369_FC4/kernel/drivers/scsi/3w-9xxx.ko) 
>  while the driver that amcc has is 2.26.03.015fw (grep TW_DRIVER_VERSION 
>  3w-9xxx.c) Its not that great a difference but it might be worth it for you. 
>  Don't forget to check the firmware upgrades as well. 
>   
I thought that when you update the driver, it automatically updates the
firmware?

[Peter Arremann]

On Thursday 30 June 2005 00:45, Mark Miksis wrote:
>  I thought that when you update the driver, it automatically updates
> the
>  firmware? 
Hmmm... Honestly, I don't really know. The 7000 series manuals
somewhere lists 
the installation of driver and firmware separately and I've done it that way 
ever since... Check your firmware version, put in the new driver and check 
again :-) Let me know what you find out... 

Peter.

[Mark Miksis]

Peter Arremann wrote:
>  On Wednesday 29 June 2005 21:58, Mark Miksis wrote:
>  
> > Very interesting reading - thanks for the link.  Some of those
> > posts suggest that the newer 3Ware firmware may "improve" the
> > issue. Otherwise, I guess I'm in the market for a different 4
> > channel RAID 5 SATA card.  Any recommendations?
>  
>  
>  Actually, I have some 9500s that are finally getting stable. Each new
>  driver revision and each new firmware is better than the last and by
>  now, I have no issues (also running the x86_64 but on amd hardware).
>  
>  
>  Anyone who asks me, usually gets a 8500 controller as recommendation
>  because unlike the 9500s, those are actually rock solid. You should
>  try the firmware updates and maybe the driver (if there is one newer
>  than the one in the kernel you're running) and see if you can't get
>  your box stable that way - as I said before, it worked for me.
>  
>  Also, you might want to get an AMD system next time if you have heavy
>  IO and more than 4GB ram - the AMD iommu requires no bounce
>  buffers...
>  
>  Peter.
>   
Well, I upgraded to the latest driver today and the problem persists.
(By the way, upgrading via the driver source does also upgrade the
firmware and BIOS.)  I spoke to 3Ware support.  They seemed familiar
with the problem but blamed it on Red Hat's Kernels.  They then referred
me to some performance tuning appnotes which I tried with no success.

I guess I'm off to buy a new card this afternoon - probably LSI or Adaptec.

FC4 x86_64 dies with heavy SATA RAID load - SOLVED

[Mark Miksis]

Mark Miksis wrote:
>  Well, I upgraded to the latest driver today and the problem
> persists.
>  (By the way, upgrading via the driver source does also upgrade the
>  firmware and BIOS.)  I spoke to 3Ware support.  They seemed familiar
>  with the problem but blamed it on Red Hat's Kernels.  They then referred
>  me to some performance tuning appnotes which I tried with no success.
>  
>  I guess I'm off to buy a new card this afternoon - probably LSI or Adaptec.
>   
I've been running the 3Ware card with the write cache turned off because
I was nervous about not having any battery backup on the card itself.
Turning the cache on seems to have completely solved my problem!  I've
been running heavy traffic (both reads and writes) for the last few
hours and it's been rock solid and very fast.  I guess I learned
something today.  I'm still not sure I'm comfortable running in this
configuration, but I do have a UPS on the box.

BTW, this means my 3Ware card is not available;)

FC4 x86_64 dies with heavy SATA RAID load - SOLVED

[Peter Arremann]

On Thursday 30 June 2005 21:35, Mark Miksis wrote:
>  I've been running the 3Ware card with the write cache turned off
> because
>  I was nervous about not having any battery backup on the card itself.
>  Turning the cache on seems to have completely solved my problem!  I've
>  been running heavy traffic (both reads and writes) for the last few
>  hours and it's been rock solid and very fast.  I guess I learned
>  something today.  I'm still not sure I'm comfortable running in this
>  configuration, but I do have a UPS on the box. 
Very interesting - what sodimm are
you using? Maybe its simply a defective 
cache module... ?


>  BTW, this means my 3Ware card is not available;) 
Oh darn :-)
Was wanting one for home to play around with but not willing to 
spend so much money just for yet another toy... :-D

Peter.

FC4 x86_64 dies with heavy SATA RAID load - SOLVED

[Mark Miksis]

Peter Arremann wrote:
>  On Thursday 30 June 2005 21:35, Mark Miksis wrote:
>  
> >I've been running the 3Ware card with the write cache turned off because
> >I was nervous about not having any battery backup on the card itself.
> >Turning the cache on seems to have completely solved my problem!  I've
> >been running heavy traffic (both reads and writes) for the last few
> >hours and it's been rock solid and very fast.  I guess I learned
> >something today.  I'm still not sure I'm comfortable running in this
> >configuration, but I do have a UPS on the box.
>  
>  Very interesting - what sodimm are you using? Maybe its simply a defective 
>  cache module... ?
>   
I never looked at the sodimm - it's whatever the card came with.  If it
was bad, I'd expect the card to fail with the cache enabled, not the
other way around.
>  
>  
> >BTW, this means my 3Ware card is not available;)
>  
>  Oh darn :-) Was wanting one for home to play around with but not willing to 
>  spend so much money just for yet another toy... :-D
>  
>  Peter.
>

FC4 x86_64 dies with heavy SATA RAID load - SOLVED

[Peter Arremann]

On Thursday 30 June 2005 23:04, Mark Miksis wrote:
>  I never looked at the sodimm - it's whatever the card came with. 
> If it
>  was bad, I'd expect the card to fail with the cache enabled, not the
>  other way around. 
Oops - sorry, I misread your post. But at least that gives me
something to try 
out - I'll disable the cache on one of the boxes we've got and see if it 
fails as well...

Peter.

[Randy]

Mark Miksis wrote:

> Alexander Dalloz wrote:
>   
>
> >Am Do, den 30.06.2005 schrieb Mark Miksis um 1:38:
> >
> >
> >    
> >
> >>I have the 3Ware 9500S-4LP (running RAID5) installed in a Dell
> >>Poweredge 1800 (dual EM64T Xeon, 7520 chipset).  With Fedora Core 4
> >>installed, everything seems fine at first glance.  However, after a
> >>period (many minutes) of heavy disk I/O operations, the machine
> >>begins to slow and then becomes completely unresponsive.  I get
> >>"APIC error on CPUn: 00(40)" in dmesg which may or may not be
> >>related.  Nothing is logged to syslog when the failures occur.  I
> >>have tried booting with noapic and using the uP kernel.  Some of
> >>these configurations make the APIC error go away, but the hangs
> >>still occur.  This machine will run fine for days or more at high
> >>CPU loads as long as the I/O load to the RAID card is low.
> >>      
> >>
> >A known issue. Some weeks ago discussed on the CentOS list:
> >
> >http://lists.centos.org/pipermail/centos/2005-May/005824.html
> >
> >Don't know why the bugzilla ticket is restricted
> >
> >https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=121434
> >    
> >
>
>
> Very interesting reading - thanks for the link.  Some of those posts
> suggest that the newer 3Ware firmware may "improve" the issue.
> Otherwise, I guess I'm in the market for a different 4 channel RAID 5
> SATA card.  Any recommendations?
>
>   
>  
I am running an Adaptec 2410SA 4 port SATA raid card on x86_64 in FC4 
and have had no problems since the third or fourth kernel update of 
FC3.  I recently copied my entire array to another network server, 
upgraded my firmware on the card and rebuilt the array, and copied 
everything back... about 200GB.  My only trouble is that I can't seem to 
get the raid tools working. But I haven't really been working on that 
since about mid FC3.