(devel) bonding test results

[Vreman, Peter]

Hi,

I'm testing the bonding patches of the current devel tree and found some issues.

The executed provisioning commands are:

cobbler system edit --name=bcsia74 --interface=bond0 --bonding=master \
 --bonding-opts="mode=active-backup miimon=100" --ip=10.227.128.204 \
 --subnet=255.255.255.0 --gateway=10.227.128.254 --static=1
cobbler system edit --name=bcsia74 --interface=eth0 \
 --mac=00:21:5A:AB:29:22 --bonding=slave --bonding-master=bond0
cobbler system edit --name=bcsia74 --interface=eth1 \
 --mac=00:21:5a:ab:29:12 --bonding=slave --bonding-master=bond0

I found the following issues:

- The WebUI can't be used. Bonding configurations are not saved.
- In the WebUI can some parts be made dynamic? E.g. when Static is disabled that the IP address/netmask/gateway are hidden/disabled. And for bonding=slave that only enables bonding master field maybe even with a drop down box.
- The new network code expects a MAC address for all NICs. I have about 100 nodes with each 6 NICs, that means 600 MAC addresses need to be added. Is there a way for auto-discovery? E.g. booting a live cd image and provising all detected NICs with a MAC address. Can the cloner live-cd be extended for such 'bare-metal' discovery?
- The physical slave nics don't get an IP address assigned in dhcpd.conf

host generic2 {
        hardware ethernet 00:21:5A:AB:29:22;
    }
    host generic1 {
        hardware ethernet 00:21:5a:ab:29:12;
    }

- I found that eth1 is configured and not the expected eth0 that was used for the PXE boot. See below for the generated pre_install code. This might give problems with DHCP where the dhpcd is configured to send the PXE boot only for the first NIC. Maybe this can be solved with configuring a primary NIC?

#  Start bond0
#  Found a slave for this interface: eth1 (00:21:5a:ab:29:12)
# Configuring bond0 (00:21:5a:ab:29:12)
if ifconfig | grep -i 00:21:5a:ab:29:12
then
  IFNAME=$(ifconfig | grep -i '00:21:5a:ab:29:12' | cut -d " " -f 1)
  echo "network --device=$IFNAME --bootproto=static --ip=10.227.128.204 --netmask=255.255.255.0 --gateway=10.227.128.254 --hostname=bcsia74.intinfra.com" >> /tmp/pre_install_network_config
fi
#  Start eth1
#  Skipping (slave-interface)
#  Start eth0
#  Skipping (slave-interface)
# Start pre_install_network_config generated code

- In the postinstall there is a replacement done in modprobe.conf. This replacement might go wrong in case the eth0 and eth1 are swapped. The first sed will e.g. replace eth0->eth1. And the second sed will replace eth1->eth0. The end result is that both are eth0. The solution is to process the modprobe.conf in a single run.

#  Start eth1
IFNAME=$(ifconfig | grep -i '00:21:5a:ab:29:12' | cut -d ' ' -f 1)
sed -i "s/$IFNAME/eth1/" /etc/modprobe.conf
rm -f /etc/sysconfig/network-scripts/ifcfg-$IFNAME
...
#  End eth1
#  Start eth0
IFNAME=$(ifconfig | grep -i '00:21:5A:AB:29:22' | cut -d ' ' -f 1)
sed -i "s/$IFNAME/eth0/" /etc/modprobe.conf
rm -f /etc/sysconfig/network-scripts/ifcfg-$IFNAME
...
#  End eth0
# End post_install_network_config generated code

Regards,
Peter

This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.

[Michael DeHaan]

Vreman, Peter - Acision wrote:
> Hi,
>
> I'm testing the bonding patches of the current devel tree and found some issues.
>
> The executed provisioning commands are:
>
> cobbler system edit --name=bcsia74 --interface=bond0 --bonding=master \
>  --bonding-opts="mode=active-backup miimon=100" --ip=10.227.128.204 \
>  --subnet=255.255.255.0 --gateway=10.227.128.254 --static=1
> cobbler system edit --name=bcsia74 --interface=eth0 \
>  --mac=00:21:5A:AB:29:22 --bonding=slave --bonding-master=bond0
> cobbler system edit --name=bcsia74 --interface=eth1 \
>  --mac=00:21:5a:ab:29:12 --bonding=slave --bonding-master=bond0
>
> I found the following issues:
> The WebUI can't be used. Bonding configurations are not saved.

Fixing the webapp NIC editor (overhauled for 1.3) is on my list for 
today/tomorrow, so I'll handle this one.
> In the WebUI can some parts be made dynamic? E.g. when Static is disabled that the IP address/netmask/gateway are hidden/disabled. And for bonding=slave that only enables bonding master field maybe even with a drop down box.

Good ideas, I'll see about these.
> The new network code expects a MAC address for all NICs. I have about 100 nodes with each 6 NICs, that means 600 MAC addresses need to be added. Is there a way for auto-discovery? E.g. booting a live cd image and provising all detected NICs with a MAC address. Can the cloner live-cd be extended for such 'bare-metal' discovery?

Specifically what do you mean happens if they don't have a MAC?

(I am still reviewing the network feature changes myself... just to make 
sure we didn't miss anything...)

No reason to use the cloner image specifically -- Any general live image 
PXE'd out to the nodes could work, yes. Func with auto-signing enabled 
would be ideally suited to create those reports and update things via 
the Cobbler API.

More simply, cobbler also features to auto register systems (it's 
disabled by default in /etc/cobbler/settings) -- though this will only 
register the first requesting interface.
> The physical slave nics don't get an IP address assigned in dhcpd.conf
>
> host generic2 {
>       hardware ethernet 00:21:5A:AB:29:22;
>   }
>   host generic1 {
>       hardware ethernet 00:21:5a:ab:29:12;
>   }

Jasper, want to handle/comment on this one?
> I found that eth1 is configured and not the expected eth0 that was used for the PXE boot. See below for the generated pre_install code. This might give problems with DHCP where the dhpcd is configured to send the PXE boot only for the first NIC. Maybe this can be solved with configuring a primary NIC?

Are you using ksdevice=bootif ? (Also, Jasper, any comments on this?)
> #  Start bond0
> #  Found a slave for this interface: eth1 (00:21:5a:ab:29:12)
> # Configuring bond0 (00:21:5a:ab:29:12)
> if ifconfig | grep -i 00:21:5a:ab:29:12
> then
>   IFNAME=$(ifconfig | grep -i '00:21:5a:ab:29:12' | cut -d " " -f 1)
>   echo "network --device=$IFNAME --bootproto=static --ip=10.227.128.204 --netmask=255.255.255.0 --gateway=10.227.128.254 --hostname=bcsia74.intinfra.com" >> /tmp/pre_install_network_config
> fi
> #  Start eth1
> #  Skipping (slave-interface)
> #  Start eth0
> #  Skipping (slave-interface)
> # Start pre_install_network_config generated code
> In the postinstall there is a replacement done in modprobe.conf. This replacement might go wrong in case the eth0 and eth1 are swapped. The first sed will e.g. replace eth0->eth1. And the second sed will replace eth1->eth0. The end result is that both are eth0. The solution is to process the modprobe.conf in a single run.
>
> #  Start eth1
> IFNAME=$(ifconfig | grep -i '00:21:5a:ab:29:12' | cut -d ' ' -f 1)
> sed -i "s/$IFNAME/eth1/" /etc/modprobe.conf
> rm -f /etc/sysconfig/network-scripts/ifcfg-$IFNAME
> ...
> #  End eth1
> #  Start eth0
> IFNAME=$(ifconfig | grep -i '00:21:5A:AB:29:22' | cut -d ' ' -f 1)
> sed -i "s/$IFNAME/eth0/" /etc/modprobe.conf
> rm -f /etc/sysconfig/network-scripts/ifcfg-$IFNAME
> ...
> #  End eth0
> # End post_install_network_config generated code
>
> Regards,
> Peter
>
> This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.
>
> cobbler mailing list
> [e-mail]
> https://fedorahosted.org/mailman/listinfo/cobbler

[Michael DeHaan]

Michael DeHaan wrote:
> Vreman, Peter - Acision wrote:
> > Hi,
> >
> > I'm testing the bonding patches of the current devel tree and found 
> > some issues.

I've fixed the problem where the webapp isn't saving settings just now, 
working on making the master/slave thing a radio button and so forth 
with greyed out boxes next.

[Michael DeHaan]

Michael DeHaan wrote:
> Michael DeHaan wrote:
> > Vreman, Peter - Acision wrote:
> > > Hi,
> > >
> > > I'm testing the bonding patches of the current devel tree and found 
> > > some issues.
>
> I've fixed the problem where the webapp isn't saving settings just 
> now, working on making the master/slave thing a radio button and so 
> forth with greyed out boxes next.

This is now also done for those that want to do some testing on the web 
app NIC editor. I think it's much nicer than in previous versions, 
though I would welcome any comments.

Randomized macs are still not specific to the virt type -- though that 
will likely come for 1.5/1.6 and not now.

--Michael

user tracking question?

[Joseph Boyer Jr.]

Question,

I know that you have enable timestamps of parameters enter into cobbler; is it possible to track what user added or updated these entries as well?

I only ask because that would make my auditing of the cobbler system that much easier, as I need to track who, what and when.  I am primarily interested in what is entered through the WebGUI,  as I only give the install team access to that.

Thanks
joe

Joseph Boyer Jr.
Liquidnet Holdings, Inc.
E [e-mail]
T +1 646.660.8352
C +1 646.284.8394

user tracking question?

[Michael DeHaan]

Joseph Boyer Jr. wrote:
> Question,
>
> I know that you have enable timestamps of parameters enter into cobbler; is it possible to track what user added or updated these entries as well?
>
> I only ask because that would make my auditing of the cobbler system that much easier, as I need to track who, what and when.  I am primarily interested in what is entered through the WebGUI,  as I only give the install team access to that.
>
> Thanks
> joe
>
> Joseph Boyer Jr.
> Liquidnet Holdings, Inc.
> E [e-mail]
> T +1 646.660.8352
> C +1 646.284.8394

That's a good request. You could also use that to do audits by backing 
up the metadata. If you do this though I think you also want to keep 
track of the local users so the mtimes are not misleading. (Don't want 
webuser1 blamed for a change made by root this week when webuser1 edited 
it last month). If you did this too, you could also possibly but 
/var/lib/cobbler/config under version control and git-commit it to get 
RSS feeds of deltas.

Can you file an RFE in Trac so we can remember this idea? 
(fedorahosted.org/cobbler ?)

--Michael
> cobbler mailing list
> [e-mail]
> https://fedorahosted.org/mailman/listinfo/cobbler

[Vreman, Peter]

Thanks for the quick replies and the fixes in the WebUI.
> > The new network code expects a MAC address for all NICs. I have about
>
> 100 nodes with each 6 NICs, that means 600 MAC addresses need to be added.
> Is there a way for auto-discovery? E.g. booting a live cd image and
> provising all detected NICs with a MAC address. Can the cloner live-cd be
> extended for such 'bare-metal' discovery?
> >
>
> Specifically what do you mean happens if they don't have a MAC?
>
> (I am still reviewing the network feature changes myself... just to make
> sure we didn't miss anything...)
>
> No reason to use the cloner image specifically -- Any general live image
> PXE'd out to the nodes could work, yes. Func with auto-signing enabled
> would be ideally suited to create those reports and update things via
> the Cobbler API.
>
> More simply, cobbler also features to auto register systems (it's
> disabled by default in /etc/cobbler/settings) -- though this will only
> register the first requesting interface.

I know the auto register feature for new kickstarts. The discovery profile can also send the MAC address like Kickstart does and after that update the record with the additional info. It can for example also send additional info to cobbler like cpus, memory and disks and maybe also things like asset tags.

I made the remark about using the cloner live image, because that allows to have a single sourcecode for a cobbler-specific live CD that supports multiple features. E.g.
 - rescue/maintenance mode
 - image creation and restoring
 - asset discovery

For the auto-discovery it might also be good to mark the NICs that are PXE-bootable for a node. And only these will be added to dhpcd.conf. In the lab we also have blades with 12 NICs and I don't like to have all 11 MAC's in the dhcpd.conf that are not used.

Regards,
Peter

This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.

[Michael DeHaan]

Vreman, Peter - Acision wrote:
> Thanks for the quick replies and the fixes in the WebUI.
> > > The new network code expects a MAC address for all NICs. I have about
> >
> > 100 nodes with each 6 NICs, that means 600 MAC addresses need to be added.
> > Is there a way for auto-discovery? E.g. booting a live cd image and
> > provising all detected NICs with a MAC address. Can the cloner live-cd be
> > extended for such 'bare-metal' discovery?
> >
> > Specifically what do you mean happens if they don't have a MAC?
> >
> > (I am still reviewing the network feature changes myself... just to make
> > sure we didn't miss anything...)
> >
> > No reason to use the cloner image specifically -- Any general live image
> > PXE'd out to the nodes could work, yes. Func with auto-signing enabled
> > would be ideally suited to create those reports and update things via
> > the Cobbler API.
> >
> > More simply, cobbler also features to auto register systems (it's
> > disabled by default in /etc/cobbler/settings) -- though this will only
> > register the first requesting interface.
>
> I know the auto register feature for new kickstarts. The discovery profile can also send the MAC address like Kickstart does and after that update the record with the additional info. It can for example also send additional info to cobbler like cpus, memory and disks and maybe also things like asset tags.
>
> I made the remark about using the cloner live image, because that allows to have a single sourcecode for a cobbler-specific live CD that supports multiple features. E.g.
> rescue/maintenance mode
> image creation and restoring
> asset discovery

I'll just be clear here -- unless someone comes on the list and says 
"the cloner image really works great", I'm not planning on supporting it 
as there's too much to go wrong -- and in fact, I've never gotten it 
/fully/ working. It's included in the cobbler tree as an example only. 
For cloning systems, it's much better to use the virtual image tools and 
virtualize on common minimal hypervisors if you think that's important. 
virtp2v may also be helpful.

Constructing a live image that sends data to cobbler for registration 
purposes sounds nice at first -- but then we have to worry about 
security aspects.

A better route would be to build a live image that contains Func that 
generates all of this inventory data, saves it to a file as needed, and 
then write a seperate script using Cobbler's Python API to plug that 
information into Cobbler.

This way it could work for any live image, or also existing systems that 
had func installed -- and Cobbler doesn't need to start becoming a 
"hardware inventory of everything" system.

I think it's fair to explore that but it leads down a path to where 
Cobbler looses focus too much and can no longer be the best thing at 
what it does -- because it also has to maintain a mediocre inventory system.

The Func script to populate cobbler approach, however, would be pretty good.
> For the auto-discovery it might also be good to mark the NICs that are PXE-bootable for a node. And only these will be added to dhpcd.conf. In the lab we also have blades with 12 NICs and I don't like to have all 11 MAC's in the dhcpd.conf that are not used.

DHCP.conf generation is not specific to PXE -- it's there to pin a MAC 
address to a specific IP (and also enable pinning an IP to a hostname 
via the transitive property if DNS management is also used).

If the length of the file is causing problems for the software we can 
re-address this, though I think it's ok.
> Regards,
> Peter
>
> This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.
>
> cobbler mailing list
> [e-mail]
> https://fedorahosted.org/mailman/listinfo/cobbler

[Jasper Capel]

Michael DeHaan wrote:
> Vreman, Peter - Acision wrote:
> > Hi,
> >
> > I'm testing the bonding patches of the current devel tree and found 
> > some issues.

Thank you for testing. :)
*cut*
> > The physical slave nics don't get an IP address assigned in dhcpd.conf
> >
> > host generic2 {
> >       hardware ethernet 00:21:5A:AB:29:22;
> >   }
> >   host generic1 {
> >       hardware ethernet 00:21:5a:ab:29:12;
> >   }
>
> Jasper, want to handle/comment on this one?

I totally overlooked this, I'll implement this. I imagine it won't be 
too difficult.
I think I can just take the bonding configuration and add it to each 
slave interface here. It wouldn't matter at all, as you're not booting 
from both interfaces simultaneously. Does anyone know if ISC-dhcpd 
allows this?
> > I found that eth1 is configured and not the expected eth0 that was
> >
> > used for the PXE boot. See below for the generated pre_install code. 
> > This might give problems with DHCP where the dhpcd is configured to 
> > send the PXE boot only for the first NIC. Maybe this can be solved 
> > with configuring a primary NIC?
>
> Are you using ksdevice=bootif ? (Also, Jasper, any comments on this?)
> > #  Start bond0
> > #  Found a slave for this interface: eth1 (00:21:5a:ab:29:12)
> > # Configuring bond0 (00:21:5a:ab:29:12)
> > if ifconfig | grep -i 00:21:5a:ab:29:12
> > then
> >   IFNAME=$(ifconfig | grep -i '00:21:5a:ab:29:12' | cut -d " " -f 1)
> >   echo "network --device=$IFNAME --bootproto=static 
> > --ip=10.227.128.204 --netmask=255.255.255.0 --gateway=10.227.128.254 
> > --hostname=bcsia74.intinfra.com" >> /tmp/pre_install_network_config
> > fi
> > #  Start eth1
> > #  Skipping (slave-interface)
> > #  Start eth0
> > #  Skipping (slave-interface)
> > # Start pre_install_network_config generated code

I assumed it would not matter at all which interface got the bond-IP 
during installation, as those interfaces would have to be connected to 
the same ethernet segment anyway. Is this really something that needs 
fixing? Right now, it just iterates through the list of interfaces, 
until it's found an interface with "bond0", that list is not sorted. I'd 
actualy prefer that the "default interface" code as it's in the tree now 
would be removed, as it's confusing. So if it's not necessary I'd really 
prefer not to add complexity here.
> > In the postinstall there is a replacement done in modprobe.conf.
> >
> > This replacement might go wrong in case the eth0 and eth1 are swapped. 
> > The first sed will e.g. replace eth0->eth1. And the second sed will 
> > replace eth1->eth0. The end result is that both are eth0. The solution 
> > is to process the modprobe.conf in a single run.
> >
> > #  Start eth1
> > IFNAME=$(ifconfig | grep -i '00:21:5a:ab:29:12' | cut -d ' ' -f 1)
> > sed -i "s/$IFNAME/eth1/" /etc/modprobe.conf
> > rm -f /etc/sysconfig/network-scripts/ifcfg-$IFNAME
> > ...
> > #  End eth1
> > #  Start eth0
> > IFNAME=$(ifconfig | grep -i '00:21:5A:AB:29:22' | cut -d ' ' -f 1)
> > sed -i "s/$IFNAME/eth0/" /etc/modprobe.conf
> > rm -f /etc/sysconfig/network-scripts/ifcfg-$IFNAME
> > ...
> > #  End eth0
> > # End post_install_network_config generated code

This is a problem indeed. I'll make it two-stage. For the modprobe.conf, 
copy it to a "staging" location, replace all interfaces with generated 
unique identifiers, and then replace go through the file again and put 
the final configuration there, unless you have any suggestions on how to 
replace all relevant values at once; does sed support this?

I'll also change the code that generates ifcfg-files, so it places those 
files in a temp-directory first, and copies the files to their new 
locations after it's processed all interfaces. That way we'll avoid any 
conflicts we might have there.

Jasper