Hello, Can anybody explain these error message to me? I have Fedora Core 6 (updated till its EOL) on IBM server running fine since last two years and suddenly I got following messages and after reboot my RAID1 mirror status changed from OPTIMAL to DEGRADED. I could not understand whether this is a software BUG or hardware (HDD/SCSI controller) failure. Please explain.
Sep 27 09:27:58 matrix kernel: sd 1:0:2:0: [sdc] Attempting to queue an ABORT message:CDB: 0x28 0x0 0x8 0x34 0x5b 0x8b 0x0 0x0 0x8 0x0 Sep 27 09:27:58 matrix kernel: scsi1: At time of recovery, card was not paused Sep 27 09:27:58 matrix kernel: >>>>>>>>>>>>>>>>>> Dump Card State Begins <<<<<<<<<<<<<<<<< Sep 27 09:27:58 matrix kernel: scsi1: Dumping Card State at program address 0xd Mode 0x33 Sep 27 09:27:58 matrix kernel: Card was paused Sep 27 09:27:58 matrix kernel: INTSTAT[0x0] SELOID[0x1] SELID[0x10] HS_MAILBOX[0x0] Sep 27 09:27:58 matrix kernel: INTCTL[0x80] SEQINTSTAT[0x0] SAVED_MODE[0x11] DFFSTAT[0x33]
Sep 27 09:27:58 matrix kernel: SCSISIGI[0x0] SCSIPHASE[0x0] SCSIBUS[0x0] LASTPHASE[0x1] Sep 27 09:27:58 matrix kernel: SCSISEQ0[0x0] SCSISEQ1[0x12] SEQCTL0[0x0] SEQINTCTL[0x0] Sep 27 09:27:58 matrix kernel: SEQ_FLAGS[0xc0] SEQ_FLAGS2[0x4] QFREEZE_COUNT[0x4] Sep 27 09:27:58 matrix kernel: KERNEL_QFREEZE_COUNT[0x4] MK_MESSAGE_SCB[0xff00] Sep 27 09:27:58 matrix kernel: MK_MESSAGE_SCSIID[0xff] SSTAT0[0x0] SSTAT1[0x8] Sep 27 09:27:58 matrix kernel: SSTAT2[0x0] SSTAT3[0x0] PERRDIAG[0x0] SIMODE1[0xa4] Sep 27 09:27:58 matrix kernel: LQISTAT0[0x0] LQISTAT1[0x0] LQISTAT2[0x0] LQOSTAT0[0x0] Sep 27 09:27:58 matrix kernel: LQOSTAT1[0x0] LQOSTAT2[0xe1] Sep 27 09:27:58 matrix kernel: Sep 27 09:27:58 matrix kernel: SCB Count = 12 CMDS_PENDING = 4 LASTSCB 0x0 CURRSCB 0x5 NEX TSCB 0xff40 Sep 27 09:27:58 matrix kernel: qinstart = 59273 qinfifonext = 59273 Sep 27 09:27:58 matrix kernel: QINFIFO: Sep 27 09:27:58 matrix kernel: WAITING_TID_QUEUES: Sep 27 09:27:58 matrix kernel: Pending list: Sep 27 09:27:58 matrix kernel: 2 FIFO_USE[0x0] SCB_CONTROL[0x60] SCB_SCSIID[0x27] Sep 27 09:27:58 matrix kernel: 7 FIFO_USE[0x0] SCB_CONTROL[0x60] SCB_SCSIID[0x27] Sep 27 09:27:58 matrix kernel: 1 FIFO_USE[0x0] SCB_CONTROL[0x60] SCB_SCSIID[0x27] Sep 27 09:27:58 matrix kernel: 6 FIFO_USE[0x0] SCB_CONTROL[0x60] SCB_SCSIID[0x27] Sep 27 09:27:58 matrix kernel: Total 4 Sep 27 09:27:58 matrix kernel: Kernel Free SCB list: 5 0 3 9 10 11 4 8 Sep 27 09:27:58 matrix kernel: Sequencer Complete DMA-inprog list: Sep 27 09:27:58 matrix kernel: Sequencer Complete list: Sep 27 09:27:58 matrix kernel: Sequencer DMA-Up and Complete list: Sep 27 09:27:58 matrix kernel: Sequencer On QFreeze and Complete list: Sep 27 09:27:58 matrix kernel: Sep 27 09:27:58 matrix kernel: Sep 27 09:27:58 matrix kernel: scsi1: FIFO0 Free, LONGJMP == 0x8259, SCB 0x5 Sep 27 09:27:58 matrix kernel: SEQIMODE[0x3f] SEQINTSRC[0x0] DFCNTRL[0x0] DFSTATUS[0x89] Sep 27 09:27:58 matrix kernel: SG_CACHE_SHADOW[0x2] SG_STATE[0x0] DFFSXFRCTL[0x0] Sep 27 09:27:58 matrix kernel: SOFFCNT[0x0] MDFFSTAT[0x5] SHADDR = 0x00, SHCNT = 0x0 Sep 27 09:27:58 matrix kernel: HADDR = 0x00, HCNT = 0x0 CCSGCTL[0x10] Sep 27 09:27:58 matrix kernel: Sep 27 09:27:58 matrix kernel: scsi1: FIFO1 Free, LONGJMP == 0x8271, SCB 0x2 Sep 27 09:27:58 matrix kernel: SEQIMODE[0x3f] SEQINTSRC[0x0] DFCNTRL[0x4] DFSTATUS[0x89] Sep 27 09:27:58 matrix kernel: SG_CACHE_SHADOW[0x2] SG_STATE[0x0] DFFSXFRCTL[0x0] Sep 27 09:27:58 matrix kernel: SOFFCNT[0x0] MDFFSTAT[0x5] SHADDR = 0x00, SHCNT = 0x0 Sep 27 09:27:58 matrix kernel: HADDR = 0x00, HCNT = 0x0 CCSGCTL[0x10] Sep 27 09:27:58 matrix kernel: LQIN: 0x8 0x0 0x0 0x5 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0 x0 0x0 0x0 0x0 0x0 0x0 0x0 Sep 27 09:27:58 matrix kernel: scsi1: LQISTATE = 0x0, LQOSTATE = 0x0, OPTIONMODE = 0x52 Sep 27 09:27:58 matrix kernel: scsi1: OS_SPACE_CNT = 0x20 MAXCMDCNT = 0x1 Sep 27 09:27:58 matrix kernel: scsi1: SAVED_SCSIID = 0x0 SAVED_LUN = 0x0 Sep 27 09:27:58 matrix kernel: SIMODE0[0xc] Sep 27 09:27:58 matrix kernel: CCSCBCTL[0x0] Sep 27 09:27:58 matrix kernel: scsi1: REG0 == 0x9, SINDEX = 0x182, DINDEX = 0x104 Sep 27 09:27:58 matrix kernel: scsi1: SCBPTR == 0x5, SCB_NEXT == 0xff40, SCB_NEXT2 == 0xff eb Sep 27 09:27:58 matrix kernel: CDB 28 0 4 80 80 94 Sep 27 09:27:58 matrix kernel: STACK: 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 Sep 27 09:27:58 matrix kernel: <<<<<<<<<<<<<<<<< Dump Card State Ends
>>>>>>>>>>>>
Sep 27 09:30:05 matrix kernel: sd 1:0:2:0: rejecting I/O to offline device Sep 27 09:30:05 matrix kernel: EXT3-fs error (device dm-2): ext3_get_inode_loc: unable to read inode block - inode=6848607, block=6848516 Sep 27 09:30:05 matrix kernel: sd 1:0:2:0: rejecting I/O to offline device Sep 27 09:30:05 matrix kernel: inode_doinit_with_dentry: getxattr returned 5 for dev=dm-2 ino=6848607 Sep 27 09:30:05 matrix kernel: sd 1:0:2:0: rejecting I/O to offline device Sep 27 09:30:05 matrix kernel: EXT3-fs error (device dm-2): ext3_get_inode_loc: unable to read inode block - inode=6848562, block=6848515 Sep 27 09:30:05 matrix kernel: sd 1:0:2:0: rejecting I/O to offline device Sep 27 09:30:05 matrix kernel: inode_doinit_with_dentry: getxattr returned 5 for dev=dm-2 ino=6848562 Sep 27 09:30:05 matrix kernel: sd 1:0:2:0: rejecting I/O to offline device Sep 27 09:30:05 matrix kernel: EXT3-fs error (device dm-2): ext3_get_inode_loc: unable to read inode block - inode=6848561, block=6848515 Sep 27 09:30:05 matrix kernel: sd 1:0:2:0: rejecting I/O to offline device Sep 27 09:30:05 matrix kernel: inode_doinit_with_dentry: getxattr returned 5 for dev=dm-2 ino=6848561 Sep 27 09:30:05 matrix kernel: sd 1:0:2:0: rejecting I/O to offline device Sep 27 09:30:05 matrix kernel: EXT3-fs error (device dm-2): ext3_get_inode_loc: unable to read inode block - inode=6848565, block=6848515 Sep 27 09:30:05 matrix kernel: sd 1:0:2:0: rejecting I/O to offline device Sep 27 09:30:05 matrix kernel: inode_doinit_with_dentry: getxattr returned 5 for dev=dm-2 ino=6848565 Sep 27 09:30:05 matrix kernel: sd 1:0:2:0: rejecting I/O to offline device Sep 27 09:30:05 matrix kernel: EXT3-fs error (device dm-2): ext3_get_inode_loc: unable to read inode block - inode=6848566, block=6848515 Sep 27 09:30:05 matrix kernel: sd 1:0:2:0: rejecting I/O to offline device Sep 27 09:30:05 matrix kernel: inode_doinit_with_dentry: getxattr returned 5 for dev=dm-2 ino=6848566 Sep 27 09:30:05 matrix kernel: sd 1:0:2:0: rejecting I/O to offline device Sep 27 09:30:05 matrix kernel: EXT3-fs error (device dm-2): ext3_get_inode_loc: unable to read inode block - inode=6029408, block=6029316 Sep 27 09:30:05 matrix kernel: sd 1:0:2:0: rejecting I/O to offline device Sep 27 09:30:05 matrix kernel: inode_doinit_with_dentry: getxattr returned 5 for dev=dm-2 ino=6029408 Sep 27 09:30:05 matrix kernel: sd 1:0:2:0: rejecting I/O to offline device Sep 27 09:30:05 matrix kernel: EXT3-fs error (device dm-2): ext3_get_inode_loc: unable to read inode block - inode=12255233, block=12255234 Sep 27 09:30:05 matrix kernel: sd 1:0:2:0: rejecting I/O to offline device Sep 27 09:30:05 matrix kernel: inode_doinit_with_dentry: getxattr returned 5 for dev=dm-2 ino=12255233 Sep 27 09:30:05 matrix kernel: sd 1:0:2:0: rejecting I/O to offline device Sep 27 09:30:05 matrix kernel: EXT3-fs error (device dm-2): ext3_get_inode_loc: unable to read inode block - inode=12124161, block=12124162 Sep 27 09:30:05 matrix kernel: sd 1:0:2:0: rejecting I/O to offline device Sep 27 09:30:05 matrix kernel: inode_doinit_with_dentry: getxattr returned 5 for dev=dm-2 ino=12124161 Sep 27 09:30:05 matrix kernel: sd 1:0:2:0: rejecting I/O to offline device Sep 27 09:30:05 matrix kernel: EXT3-fs error (device dm-2): ext3_get_inode_loc: unable to read inode block - inode=8355841, block=8355842 Sep 27 09:31:54 matrix last message repeated 111 times Sep 27 09:31:54 matrix last message repeated 13 times Sep 27 09:33:41 matrix smartd[3466]: Device: /dev/sdc, No such device or address, open() f ailed Sep 27 09:33:41 matrix smartd[3466]: Sending warning via mail to root ... Sep 27 09:33:42 matrix smartd[3466]: Warning via mail to root: successful Sep 27 09:34:11 matrix smartd[3466]: smartd received signal 15: Terminated Sep 27 09:34:11 matrix smartd[3466]: smartd is exiting (exit status 0) Sep 27 09:34:11 matrix avahi-daemon[3332]: Got SIGTERM, quitting.
Thank you.
Without Wax !
Rahul. Technical Support
On 10/1/2010 3:38 AM, Rahul Tidke wrote:
Sep 27 09:30:05 matrix kernel: sd 1:0:2:0: rejecting I/O to offline device
I had the same thing this week with a Dell SAS 5/iR controller. I didn't see any hard drive i/o errors but of the 'sd' device itself. I rebooted and went into the card's bios, and in there it crashed with a parity error. Bought a used card off ebay, drove there and back to replace it, and it resync'd the array (raid-1 that is..) and we're up and running.
So, do you have another raid card?
-eric wood
-
Eric Wood -
Rahul Tidke