mdadm: cannot open device /dev/sda1: Device or resource busy \ mdadm: /dev/sda1 has no superblock - assembly aborted

[James Groleau]

Hey Folks,

I've got a problem that I cannot seem to make go away.

I've got a machine running FC8 
]# uname -a
Linux <hostname> 2.6.26.6-49.fc8 #1 SMP Fri Oct 17 15:33:32 EDT 2008 x86_64 
x86_64 x86_64 GNU/Linux
with an external eSATA Hard Drive RAID 5 array. 

I installed a replacement drive for the main system and performed a fresh 
install of FC10 with the RAID 5 array disconnected. Then after ensuring all 
updates were completed (as of Jan 03/09) I reattached the array. 

As root I issued the mdadm command and received the following error:
]# mdadm --assemble /dev/md1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sda1
mdadm: cannot open device /dev/sda1: Device or resource busy
mdadm: /dev/sda1 has no superblock - assembly aborted

I cannot seem to determine why I cannot access the sda1 drive. I have replaced 
the FC8 system drive and all is well. I can start and mount the RAID array 
normally. 

There is no mdadm.conf file. I have always mounted the array manually. 

I've beat Google into a frenzy over the last couple days trying find a 
solution to no avail.

Any clues/hints/suggestion? 
Thanks much.

Additional information:
======================================================
]# mdadm --assemble --run /dev/md1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1
This will work to bring up the array under FC10 in a degraded state. 

I did remove sda1 from the array, then delete and recreate the partition on 
sda, and attempt to add it back into the array, but continue to receive the 
same error messages from mdadm. 

I had to rebuild the array under FC8.
====================================================

]# lsof | grep -i sda
<nothing listed>
======================================================
]#  mdadm --examine /dev/sda1 
/dev/sda1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : c03bad6f:28dae689:a100065d:2118bad4
  Creation Time : Sat Nov  8 17:08:48 2008
     Raid Level : raid5
  Used Dev Size : 488383936 (465.76 GiB 500.11 GB)
     Array Size : 1953535744 (1863.04 GiB 2000.42 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 1

    Update Time : Sat Jan  3 16:11:42 2009
          State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 3fb6543 - correct
         Events : 445816

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     4       8        1        4      active sync   /dev/sda1

   0     0       8       33        0      active sync   /dev/sdc1
   1     1       8       49        1      active sync   /dev/sdd1
   2     2       8       65        2      active sync   /dev/sde1
   3     3       8       81        3      active sync   /dev/sdf1
   4     4       8        1        4      active sync   /dev/sda1
======================================================

]#  mdadm --examine /dev/sdb1 
mdadm: No md superblock detected on /dev/sdb1.
]#  mdadm --examine /dev/sdc1 
/dev/sdc1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : c03bad6f:28dae689:a100065d:2118bad4
  Creation Time : Sat Nov  8 17:08:48 2008
     Raid Level : raid5
  Used Dev Size : 488383936 (465.76 GiB 500.11 GB)
     Array Size : 1953535744 (1863.04 GiB 2000.42 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 1

    Update Time : Sat Jan  3 16:11:42 2009
          State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 3fb655b - correct
         Events : 445816

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     0       8       33        0      active sync   /dev/sdc1

   0     0       8       33        0      active sync   /dev/sdc1
   1     1       8       49        1      active sync   /dev/sdd1
   2     2       8       65        2      active sync   /dev/sde1
   3     3       8       81        3      active sync   /dev/sdf1
   4     4       8        1        4      active sync   /dev/sda1
=================================================================
]#  mdadm --examine /dev/sdc1 
/dev/sdc1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : c03bad6f:28dae689:a100065d:2118bad4
  Creation Time : Sat Nov  8 17:08:48 2008
     Raid Level : raid5
  Used Dev Size : 488383936 (465.76 GiB 500.11 GB)
     Array Size : 1953535744 (1863.04 GiB 2000.42 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 1

    Update Time : Sat Jan  3 16:11:42 2009
          State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 3fb655b - correct
         Events : 445816

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     0       8       33        0      active sync   /dev/sdc1

   0     0       8       33        0      active sync   /dev/sdc1
   1     1       8       49        1      active sync   /dev/sdd1
   2     2       8       65        2      active sync   /dev/sde1
   3     3       8       81        3      active sync   /dev/sdf1
   4     4       8        1        4      active sync   /dev/sda1
================================================================

]#  mdadm --examine /dev/sde1 
/dev/sde1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : c03bad6f:28dae689:a100065d:2118bad4
  Creation Time : Sat Nov  8 17:08:48 2008
     Raid Level : raid5
  Used Dev Size : 488383936 (465.76 GiB 500.11 GB)
     Array Size : 1953535744 (1863.04 GiB 2000.42 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 1

    Update Time : Sat Jan  3 16:11:42 2009
          State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 3fb657f - correct
         Events : 445816

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     2       8       65        2      active sync   /dev/sde1

   0     0       8       33        0      active sync   /dev/sdc1
   1     1       8       49        1      active sync   /dev/sdd1
   2     2       8       65        2      active sync   /dev/sde1
   3     3       8       81        3      active sync   /dev/sdf1
   4     4       8        1        4      active sync   /dev/sda1

===============================================================

]#  mdadm --examine /dev/sdf1 
/dev/sdf1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : c03bad6f:28dae689:a100065d:2118bad4
  Creation Time : Sat Nov  8 17:08:48 2008
     Raid Level : raid5
  Used Dev Size : 488383936 (465.76 GiB 500.11 GB)
     Array Size : 1953535744 (1863.04 GiB 2000.42 GB)
   Raid Devices : 5
  Total Devices : 5
Preferred Minor : 1

    Update Time : Sat Jan  3 16:11:42 2009
          State : clean
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 3fb6591 - correct
         Events : 445816

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     3       8       81        3      active sync   /dev/sdf1

   0     0       8       33        0      active sync   /dev/sdc1
   1     1       8       49        1      active sync   /dev/sdd1
   2     2       8       65        2      active sync   /dev/sde1
   3     3       8       81        3      active sync   /dev/sdf1
   4     4       8        1        4      active sync   /dev/sda1
============================================================

]# service --status-all | grep -i md
mdadm is stopped
======================================================
]# service --status-all 
acpid (pid 2176) is running...
anacron (pid 2649) is running...
atd (pid  2569) is running...
auditd (pid  2057) is running...
Avahi daemon is running
BackupPC is stopped
bluetoothd is stopped
btseed is stopped
bttrack is stopped
Frequency scaling enabled using ondemand governor
crond (pid  2518) is running...
cupsd (pid  2616) is running...
cups-config-daemon is obsolete
dc_client is stopped
dc_server is stopped
dnsmasq (pid 2657) is running...
Usage: /etc/init.d/firstboot {start|stop}
Checking fuse filesystem ok.
gpm (pid  2507) is running...
hald (pid 2184) is running...
httpd is stopped
innd is stopped

irattach is stopped
irqbalance (pid  2085) is running...
iscsid is stopped
kerneloops (pid 2527) is running...
libvirtd (pid  2590) is running...
k8temp-pci-00c3
Adapter: PCI adapter
Core0 Temp:  +15.0°C                                    
Core0 Temp:   +5.0°C                                    
Core1 Temp:   +0.0°C                                    
Core1 Temp:   -3.0°C                                    

mdadm is stopped
dbus-daemon (pid 3149 2152) is running...
multipathd is stopped
netconsole module not loaded
netplugd is stopped
Configured devices:
lo eth0 eth1
Currently active devices:
lo eth0 eth1 virbr0
NetworkManager (pid  2402) is running...
rpc.mountd is stopped
nfsd is stopped
rpc.rquotad is stopped
rpc.statd (pid  2105) is running...
nmbd (pid  2540) is running...
nscd is stopped
ntpd (pid  2453) is running...
nvidia.ko for kernel 2.6.27.9-159.fc10.x86_64 was found.
The driver is enabled.
openvpn: service not started
pcscd (pid  2389) is running...
portreserve is stopped
Process accounting is disabled.
racoon is stopped
rdisc is stopped
restorecond is stopped
rpcbind (pid 2092) is running...
rpc.idmapd (pid 2139) is running...
rsyslogd (pid 2073) is running...
saslauthd is stopped
sendmail (pid  2488) is running...
sm-client (pid  2497) is running...
setroubleshootd (pid  2435) is running...
smartd is stopped
smbd (pid  2548) is running...
Monthly smolt check-in is enabled.
snmpd is stopped
snmptrapd is stopped
squid is stopped
openssh-daemon (pid  2445) is running...
Xvnc is stopped
vsftpd (pid 2471) is running...
winbindd (pid  2415) is running...
wpa_supplicant (pid 2412) is running...
ypbind is stopped