Fedora 23 Install Issues on PPC64LE Server

[Adi Gangidi]

Hello All

As I am trying to install Fedora 23 on our PPC64LE server. I am running into an issue with install. It has to be noted that Fedora 22/21 fail with similar errors on this configuration:

Any pointers on narrowing this issue down would be greatly appreciated.

To give details on the box:
A) Stoarage: Avago RAID controller 9361-8i (with 15 solid state drives on RAID0 config) Firmware version we used is 4.300.00-4429.
B) Network: Mellanox Connectx 3 pro card in the box.

Here are the details. Install log is attached:

1.  Install Process gets stuck at the following point during installation:  (Full install logs are attached)

[   35.253098] mlx4_en: 0002:01:00.0: Port 1: Initializing port

[   35.263979] mlx4_en 0002:01:00.0: Activating port:2

[   35.271034] mlx4_en: eth1: Link Up

[   35.281464] mlx4_en: 0002:01:00.0: Port 2: Using 256 TX rings

[   35.285573] mlx4_en: 0002:01:00.0: Port 2: Using 8 RX rings

[   35.291084] mlx4_en: 0002:01:00.0: Port 2:   frag:0 - size:1518 prefix:0 stride:1536

[   35.298236] mlx4_en: 0002:01:00.0: Port 2: Initializing port

[   35.309191] <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v2.2-1 (Feb 2014)

[   35.327918] <mlx4_ib> mlx4_ib_add: counter index 2 for port 1 allocated 1

[   35.333372] <mlx4_ib> mlx4_ib_add: counter index 3 for port 2 allocated 1

[   35.354383] <mlx4_ib> mlx4_ib_add: counter index 2 for port 1 allocated 1

[   35.358463] <mlx4_ib> mlx4_ib_add: counter index 3 for port 2 allocated 1

[   35.363710] mlx4_en: eth3: Link Up

[   35.376411] mlx4_core 0000:01:00.0 enp1s0: renamed from eth0

[   35.432571] mlx4_core 0000:01:00.0 enp1s0d1: renamed from eth1

[   35.603694] mlx4_core 0002:01:00.0 enP2p1s0d1: renamed from eth3

[   35.692421] mlx4_core 0002:01:00.0 enP2p1s0: renamed from eth2

[   37.691861] sd 5:2:10:0: [sdl] Attached SCSI disk

[   37.692054] sd 5:2:6:0: [sdh] Attached SCSI disk

[   37.692520] sd 5:2:13:0: [sdo] Attached SCSI disk

[   37.692800] sd 5:2:14:0: [sdp] Attached SCSI disk

[   43.690839] sd 5:2:12:0: [sdn] Attached SCSI disk

[   43.691142] sd 5:2:8:0: [sdj] Attached SCSI disk

[   43.691335] sd 5:2:11:0: [sdm] Attached SCSI disk

[   43.691633] sd 5:2:9:0: [sdk] Attached SCSI disk

[   49.685080] sd 5:2:5:0: [sdg] Attached SCSI disk

[   49.685256] sd 5:2:4:0: [sdf] Attached SCSI disk

[   49.685755] sd 5:2:7:0: [sdi] Attached SCSI disk

2.  After getting stuck for about 3-5 minutes, rescue shell kicks in as below. However  as soon as it loads, serial console  doesn't respond anymore, resulting in not being able to collect logs from rescue shell.
[  235.757258] ]dracut-initqueue[1776]: Warning: Could not boot.
  235.758170] dracut-initqueue[1776]: Warning: /dev/root does not exist
         Starting Setup Virtual Console...
[  OK  ] Started Setup Virtual Console.
[  236.203610] audit: type=1130 audit(235.750:12): pid=1 uid=0 auid=4294967295 s
es=4294967295 subj=kernel msg='unit=systemd-vconsole-setup comm="systemd" exe="/
usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
         Starting Dracut Emergency Shell...
Warning: /dev/root does not exist
Generating "/run/initramfs/rdsosreport.txt"
Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.

I can add details here as necessary.

Best
Adi Gangidi
Rackspace

[Jakub Cajka]

----- Original Message -----
> From: "Adi Gangidi" 
> To: [e-mail]
> Cc: "John Hincher" , "Aaron Sullivan" , "Antony Messerli"
> , "Major Hayden" 
> Sent: Thursday, September 10, 2015 1:40:08 AM
> Subject: Fedora 23 Install Issues on PPC64LE Server
>
> Hello All

Hello,
> As I am trying to install Fedora 23 on our PPC64LE server. I am running into
> an issue with install. It has to be noted that Fedora 22/21 fail with
> similar errors on this configuration:
>
> Any pointers on narrowing this issue down would be greatly appreciated.
>
> To give details on the box:
>
> A ) Stoarage: Avago RAID controller 9361-8i (with 15 solid state drives on
> RAID0 config) Firmware version we used is 4.300.00-4429 .
>
> B) Network: Mellanox Connectx 3 pro card in the box.
>
> Here are the details. Install log is attached:
> 1. Install Process gets stuck at the following point during installation:
> (Full install logs are attached)
>
> [ 35.253098] mlx4_en: 0002:01:00.0: Port 1: Initializing port
>
> [ 35.263979] mlx4_en 0002:01:00.0: Activating port:2
>
> [ 35.271034] mlx4_en: eth1: Link Up
>
> [ 35.281464] mlx4_en: 0002:01:00.0: Port 2: Using 256 TX rings
>
> [ 35.285573] mlx4_en: 0002:01:00.0: Port 2: Using 8 RX rings
>
> [ 35.291084] mlx4_en: 0002:01:00.0: Port 2: frag:0 - size:1518 prefix:0
> stride:1536
>
> [ 35.298236] mlx4_en: 0002:01:00.0: Port 2: Initializing port
>
> [ 35.309191] <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand
> driver v2.2-1 (Feb 2014)
>
> [ 35.327918] <mlx4_ib> mlx4_ib_add: counter index 2 for port 1 allocated 1
>
> [ 35.333372] <mlx4_ib> mlx4_ib_add: counter index 3 for port 2 allocated 1
>
> [ 35.354383] <mlx4_ib> mlx4_ib_add: counter index 2 for port 1 allocated 1
>
> [ 35.358463] <mlx4_ib> mlx4_ib_add: counter index 3 for port 2 allocated 1
>
> [ 35.363710] mlx4_en: eth3: Link Up
>
> [ 35.376411] mlx4_core 0000:01:00.0 enp1s0: renamed from eth0
>
> [ 35.432571] mlx4_core 0000:01:00.0 enp1s0d1: renamed from eth1
>
> [ 35.603694] mlx4_core 0002:01:00.0 enP2p1s0d1: renamed from eth3
>
> [ 35.692421] mlx4_core 0002:01:00.0 enP2p1s0: renamed from eth2
>
> [ 37.691861] sd 5:2:10:0: [sdl] Attached SCSI disk
>
> [ 37.692054] sd 5:2:6:0: [sdh] Attached SCSI disk
>
> [ 37.692520] sd 5:2:13:0: [sdo] Attached SCSI disk
>
> [ 37.692800] sd 5:2:14:0: [sdp] Attached SCSI disk
>
> [ 43.690839] sd 5:2:12:0: [sdn] Attached SCSI disk
>
> [ 43.691142] sd 5:2:8:0: [sdj] Attached SCSI disk
>
> [ 43.691335] sd 5:2:11:0: [sdm] Attached SCSI disk
>
> [ 43.691633] sd 5:2:9:0: [sdk] Attached SCSI disk
>
> [ 49.685080] sd 5:2:5:0: [sdg] Attached SCSI disk
>
> [ 49.685256] sd 5:2:4:0: [sdf] Attached SCSI disk
>
> [ 49.685755] sd 5:2:7:0: [sdi] Attached SCSI disk
>     2. After getting stuck for about 3-5 minutes, rescue shell kicks in as
>     below. However as soon as it loads, serial console doesn’t respond
>     anymore, resulting in not being able to collect logs from rescue shell.
> [ 235.757258] ]dracut-initqueue[1776]: Warning: Could not boot. 235.758170]
> dracut-initqueue[1776]: Warning: /dev/root does not exist Starting Setup
> Virtual Console... [ OK ] Started Setup Virtual Console. [ 236.203610]
> audit: type=1130 audit(235.750:12): pid=1 uid=0 auid=4294967295 s
> es=4294967295 subj=kernel msg='unit=systemd-vconsole-setup comm="systemd"
> exe="/ usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
> Starting Dracut Emergency Shell... Warning: /dev/root does not exist
> Generating "/run/initramfs/rdsosreport.txt" Entering emergency mode. Exit
> the shell to continue. Type "journalctl" to view system logs. You might want
> to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot after
> mounting them and attach it to a bug report.

The freezing console might be related to the https://bugzilla.redhat.com/show_bug.cgi?id=1255074 . Which makes debugging really annoying and painful especially combined with https://bugzilla.redhat.com/show_bug.cgi?id=1255066 (it should get fixed in next image iteration).

Do you observe same behavior of console with f21/f22?

What do you use to obtain console(ipmi sol, physical connection,...)?

What are you using as installation source(DVD, PXE, USB, remote repo,....) and what image are you using(netinst/dvd)?

Are disks already partitioned or are empty?

Have you been successful with booting/installing any other linux distribution(either LE/BE)?

Thanks for report and testing,

Jakub
> I can add details here as necessary.
>
> Best
> Adi Gangidi
> Rackspace
>
> ppc mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/ppc

[Adi Gangidi]

Adding some more info that I had written down for one of our community members:

Hello Jakub

Here is the response to your questions.

Q: Do you observe same behavior of console with f21/f22?

Install gets stuck at the same place: (Attached SCSI) in f21/f22 but
rescue shell does NOT freeze in f21/f22.

Q: What do you use to obtain console(ipmi sol, physical connection,Š)?

‹Physical connection: Serial USB cable attached to debug port on server.

Q: What are you using as installation source(DVD, PXE, USB, remote
repo,....) and what image are you using(netinst/dvd)?

‹ Installation source is a Fedora 23 DVD content put on a USB stick.

Q: Are disks already partitioned or are empty?

‹ Disks where previously partitioned (I had powerkvm installed on there)
but this error occurs before I can get to the partition screen.

Q: Have you been successful with booting/installing any other linux
distribution(either LE/BE)?

‹ Yes Ubuntu 14.04 / 14.10 (PPC64LE) installed smoothly. With Ubuntu 15.04
install I ran into this install error on partitioning Avago raid:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1475166

However we are working around that bug by setting max_sectors of Megaraid
module manually and thus able to finish the install.

We are also able to install powerKVM (IBM¹s PPC64LE KVM)

Best
Adi

From: Rackspace Hosting <[e-mail]>
Date: Wednesday, September 9, 2015 at 6:40 PM
To: "[e-mail]" <[e-mail]>
Cc: Aaron Sullivan <[e-mail]>, Major Hayden <[e-mail]>, John Hincher <[e-mail]>, Antony Messerli <[e-mail]>
Subject: Fedora 23 Install Issues on PPC64LE Server

onnectx 3 pro card in the box.

[Adi Gangidi]

Hello All

I wanted to update on how we overcame below install issue and successfully installed Fedora 23 on our power server. This install error was caused by Driver bug (Avago RAID card, our storage controller) for which the fix is process.

This bug manifests itself to the end user who is installing Fedora 23/22 on Avago RAID based as: Partitioning / mounting of file system as “ext4” failed. Hence the install process fails.

More details about bug can be found in last few comments of this ubuntu bug (We found this bug while installing ubuntu 15.04  for the first time):

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1475166

For now following work around can be used to install Fedora 23 on power systems with Avago raid card:

1) Power on and wait system boots into Petitboot:
    1a) Press "e" on "Install Fedora 23 (64-bit kernel)" option and append "megaraid_sas.max_sectors=2048" in the end of "boot argument"
2) Select modified option and follow the installation process.
3) After install is complete and you are back to Petitboot: Edit booting option by appending megaraid_sas.max_sectors=2048, and boot with completed install.
***IMPORTANT NOTE***  Step 3 is needed for every boot time, or you will get system going to failure due to system can't mount file system.
Alternatively we can add this to Grub config to avoid repeating this process.

Please give it a try as possible.

Also any advise on how to best track this on corresponding Fedora board would be appreciated.

If what I am doing currently is enough (sharing on this mailing list), that’s good to know also.

Best
Adi

From: Rackspace Hosting <[e-mail]>
Date: Friday, September 11, 2015 at 1:06 PM
To: "[e-mail]" <[e-mail]>
Cc: Aaron Sullivan <[e-mail]>, Major Hayden <[e-mail]>, John Hincher <[e-mail]>, Antony Messerli <[e-mail]>
Subject: Re: Fedora 23 Install Issues on PPC64LE Server

Adding some more info that I had written down for one of our community members:

Hello Jakub

Here is the response to your questions.

Q: Do you observe same behavior of console with f21/f22?

Install gets stuck at the same place: (Attached SCSI) in f21/f22 but
rescue shell does NOT freeze in f21/f22.

Q: What do you use to obtain console(ipmi sol, physical connection,Š)?

‹Physical connection: Serial USB cable attached to debug port on server.

Q: What are you using as installation source(DVD, PXE, USB, remote
repo,....) and what image are you using(netinst/dvd)?

‹ Installation source is a Fedora 23 DVD content put on a USB stick.

Q: Are disks already partitioned or are empty?

‹ Disks where previously partitioned (I had powerkvm installed on there)
but this error occurs before I can get to the partition screen.

Q: Have you been successful with booting/installing any other linux
distribution(either LE/BE)?

‹ Yes Ubuntu 14.04 / 14.10 (PPC64LE) installed smoothly. With Ubuntu 15.04
install I ran into this install error on partitioning Avago raid:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1475166

However we are working around that bug by setting max_sectors of Megaraid
module manually and thus able to finish the install.

We are also able to install powerKVM (IBM¹s PPC64LE KVM)

Best
Adi

From: Rackspace Hosting <[e-mail]>
Date: Wednesday, September 9, 2015 at 6:40 PM
To: "[e-mail]" <[e-mail]>
Cc: Aaron Sullivan <[e-mail]>, Major Hayden <[e-mail]>, John Hincher <[e-mail]>, Antony Messerli <[e-mail]>
Subject: Fedora 23 Install Issues on PPC64LE Server

onnectx 3 pro card in the box.

[Adi Gangidi]

Sending this update to the list again. Since my last message is still stuck in review with Admin.

Hello All

I wanted to update on how we overcame below install issue and successfully installed Fedora 23 on our power server. This install error was caused by Driver bug (Avago RAID card, our storage controller) for which the fix is process.

This bug manifests itself to the end user who is installing Fedora 23/22 on Avago RAID based as: Partitioning / mounting of file system as “ext4” failed. Hence the install process fails.

More details about bug can be found in last few comments of this ubuntu bug (We found this bug while installing ubuntu 15.04  for the first time):

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1475166

For now following work around can be used to install Fedora 23 on power systems with Avago raid card:

1) Power on and wait system boots into Petitboot:
    1a) Press "e" on "Install Fedora 23 (64-bit kernel)" option and append "megaraid_sas.max_sectors=2048" in the end of "boot argument"
2) Select modified option and follow the installation process.
3) After install is complete and you are back to Petitboot: Edit booting option by appending megaraid_sas.max_sectors=2048, and boot with completed install.
***IMPORTANT NOTE***  Step 3 is needed for every boot time, or you will get system going to failure due to system can't mount file system.
Alternatively we can add this to Grub config to avoid repeating this process.

Please give it a try as possible.

Also any advise on how to best track this on corresponding Fedora board would be appreciated.

If what I am doing currently is enough (sharing on this mailing list), that’s good to know also.

Best
Adi

From: Rackspace Hosting <[e-mail]>
Date: Friday, September 11, 2015 at 1:06 PM
To: "[e-mail]" <[e-mail]>
Cc: Aaron Sullivan <[e-mail]>, Major Hayden <[e-mail]>, John Hincher <[e-mail]>, Antony Messerli <[e-mail]>
Subject: Re: Fedora 23 Install Issues on PPC64LE Server

Adding some more info that I had written down for one of our community members:

Hello Jakub

Here is the response to your questions.

Q: Do you observe same behavior of console with f21/f22?

Install gets stuck at the same place: (Attached SCSI) in f21/f22 but
rescue shell does NOT freeze in f21/f22.

Q: What do you use to obtain console(ipmi sol, physical connection,Š)?

‹Physical connection: Serial USB cable attached to debug port on server.

Q: What are you using as installation source(DVD, PXE, USB, remote
repo,....) and what image are you using(netinst/dvd)?

‹ Installation source is a Fedora 23 DVD content put on a USB stick.

Q: Are disks already partitioned or are empty?

‹ Disks where previously partitioned (I had powerkvm installed on there)
but this error occurs before I can get to the partition screen.

Q: Have you been successful with booting/installing any other linux
distribution(either LE/BE)?

‹ Yes Ubuntu 14.04 / 14.10 (PPC64LE) installed smoothly. With Ubuntu 15.04
install I ran into this install error on partitioning Avago raid:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1475166

However we are working around that bug by setting max_sectors of Megaraid
module manually and thus able to finish the install.

We are also able to install powerKVM (IBM¹s PPC64LE KVM)

Best
Adi

From: Rackspace Hosting <[e-mail]>
Date: Wednesday, September 9, 2015 at 6:40 PM
To: "[e-mail]" <[e-mail]>
Cc: Aaron Sullivan <[e-mail]>, Major Hayden <[e-mail]>, John Hincher <[e-mail]>, Antony Messerli <[e-mail]>
Subject: Fedora 23 Install Issues on PPC64LE Server

onnectx 3 pro card in the box.

[Major Hayden]

On 10/01/2015 08:23 PM, Adi Gangidi wrote:
> Also any advise on how to best track this on corresponding Fedora board would be appreciated.
>
> If what I am doing currently is enough (sharing on this mailing list), that’s good to know also.

It might be worthwhile to add a bug to Red Hat's Bugzilla:

https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora

--
Major Hayden

[Adi Gangidi]

Will do that.

Thanks
‹Adi

On 10/2/15, 8:23 AM, "Major Hayden"  wrote:
> On 10/01/2015 08:23 PM, Adi Gangidi wrote:
> > Also any advise on how to best track this on corresponding Fedora board
> > would be appreciated.
> >
> > If what I am doing currently is enough (sharing on this mailing list),
> > that¹s good to know also.
>
> It might be worthwhile to add a bug to Red Hat's Bugzilla:
>
> https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora
>
> --
> Major Hayden

[Dan Horák]

On Fri, 2 Oct 2015 13:24:56 +0000
Adi Gangidi  wrote:
> Will do that.

thanks, if I understand correctly the real culprit is in the kernel
Avago driver and that way we (and the Fedora kernel team) can track if
the necessary change went into the mainline kernel or LKML.

Dan
> Thanks
> ‹Adi
>
> On 10/2/15, 8:23 AM, "Major Hayden" 
> wrote:
> > On 10/01/2015 08:23 PM, Adi Gangidi wrote:
> > > Also any advise on how to best track this on corresponding Fedora
> > > board
> > > would be appreciated.
> > >
> > > If what I am doing currently is enough (sharing on this mailing
> > > list),
> > > that¹s good to know also.
> >
> > It might be worthwhile to add a bug to Red Hat's Bugzilla:
> >
> > https://bugzilla.redhat.com/enter_bug.cgi?product=Fedora
> >
> > --
> > Major Hayden
>
> ppc mailing list
> [e-mail]
> https://admin.fedoraproject.org/mailman/listinfo/ppc

[Adi Gangidi]

Here is the link to the bug:

https://bugzilla.redhat.com/show_bug.cgi?id=1269300

Will make sure any progress on merging these changes is reflected on the
filed bug and here on the list.

Thanks
Adi

On 10/2/15, 8:44 AM, "Dan Horák"  wrote:
> the necessary change went into the mainline kernel or LKML.