Ipmitool and ipmilan woes and power resetting and PXE Mode issues.

[Jennifer Patrick]

Just having some woes with making fence_ipmilan work with some Supermicro
servers (Don't say use the supermicro IPMI Tool, we're trying to get around
that.)

The problem is, using fence_ipmilan, we can initiate a boot (off then on)
from the web GUI, or from the CLI via...

*# cobbler system reboot --name=test123*
*task started: 2015-08-05_092815_power*
*task started (id=Power management (reboot), time=Wed Aug  5 09:28:15 2015)*
*cobbler power configuration is:*
*      type   : ipmilan*
*      address: 192.168.1.5*
*      user   : user*
*      id     :*
*running: /usr/sbin/fence_ipmilan*
*running: /usr/sbin/fence_ipmilan*
*received on stdout: Powering off machine @ IPMI:**192.168.1.5**...Failed*

*received on stderr:*
*running: /usr/sbin/fence_ipmilan*
*received on stdout: Powering off machine @ IPMI:**192.168.1.5**...Done*

*received on stderr:*
*cobbler power configuration is:*
*      type   : ipmilan*
*      address: **192.168.1.5*
*      user   : user*
*      id     :*
*running: /usr/sbin/fence_ipmilan*
*received on stdout: Powering on machine @ IPMI:**192.168.1.5**...Failed*

*received on stderr: ipmilan: Power still off*

*running: /usr/sbin/fence_ipmilan*
*received on stdout: Powering on machine @ IPMI:192.168.1.5...Done*

*received on stderr:*
**** TASK COMPLETE ****

So that's great.  My real question comes from the fact that I can't make
cobbler set these machines to reboot via the Web GUI to Netboot/PXE.  It
just seems to never issue the right command via fence_ipmilan (which
actually uses IPMITOOL)

I can do it fine in IPMITOOL:

*ipmitool -H 192.168.1.5 -U user  -I lanplus -P password chassis bootdev
pxeipmitool -H 192.168.1.5 -U user  -I lanplus -P password chassis power
reset*

And the system reboots into PXE boot mode.

So is there something I have to mangle in
/etc/cobbler/fence_ipmilan.template?

Something Else?

All help is appreciated.

Thanks.

JLP

[Alvin Starr]

It was my impression that fence is just to shut a machine down as part 
of the HA/clustering toolset.
All the other features are just added extras that are not needed as part 
of the primary function.

The features from man include on,off,reboot,status,diag,list,monitor.

There is no "set to boot from PXE".
I believe that you will need to do this manually.

On 08/05/2015 05:36 AM, Jennifer Patrick wrote:
> Just having some woes with making fence_ipmilan work with some 
> Supermicro servers (Don't say use the supermicro IPMI Tool, we're 
> trying to get around that.)
>
> The problem is, using fence_ipmilan, we can initiate a boot (off then 
> on) from the web GUI, or from the CLI via...
>
> /# cobbler system reboot --name=test123/
> /task started: 2015-08-05_092815_power/
> /task started (id=Power management (reboot), time=Wed Aug  5 09:28:15 
> 2015)/
> /cobbler power configuration is:/
> /      type   : ipmilan/
> /      address: 192.168.1.5/
> /      user   : user/
> /      id     :/
> /running: /usr/sbin/fence_ipmilan/
> /running: /usr/sbin/fence_ipmilan/
> /received on stdout: Powering off machine @ IPMI://192.168.1.5//...Failed/
> /
> /
> /received on stderr:/
> /running: /usr/sbin/fence_ipmilan/
> /received on stdout: Powering off machine @ IPMI://192.168.1.5//...Done/
> /
> /
> /received on stderr:/
> /cobbler power configuration is:/
> /      type   : ipmilan/
> /      address: //192.168.1.5/
> /      user   : user/
> /      id     :/
> /running: /usr/sbin/fence_ipmilan/
> /received on stdout: Powering on machine @ IPMI://192.168.1.5//...Failed/
> /
> /
> /received on stderr: ipmilan: Power still off/
> /
> /
> /running: /usr/sbin/fence_ipmilan/
> /received on stdout: Powering on machine @ IPMI:192.168.1.5...Done/
> /
> /
> /received on stderr:/
> /*** TASK COMPLETE ***/
> /
> /
> So that's great.  My real question comes from the fact that I can't 
> make cobbler set these machines to reboot via the Web GUI to 
> Netboot/PXE.  It just seems to never issue the right command via 
> fence_ipmilan (which actually uses IPMITOOL)
>
> I can do it fine in IPMITOOL:
> /
> /
> /
> ipmitool -H 192.168.1.5 -U user  -I lanplus -P password chassis 
> bootdev pxe
> ipmitool -H 192.168.1.5 -U user  -I lanplus -P password chassis power 
> reset
> /
> /
> /
> And the system reboots into PXE boot mode.
>
> So is there something I have to mangle in 
> /etc/cobbler/fence_ipmilan.template?
>
> Something Else?
>
> All help is appreciated.
>
> Thanks.
>
> JLP
>
> cobbler mailing list
> [e-mail]
> https://lists.fedorahosted.org/mailman/listinfo/cobbler

[Jennifer Patrick]

Howow would you suggest going about that...

Aside from somehow inserting some manual bit into the python code to do
this?  And even then, what and where?

And why doesn't the CLI work for things like this...?

cobbler system edit --name=test123 --netboot-enabled=true

cobbler system reboot --name=test123

Does not produce a netboot.  But ipmitool wiil.

When I do try it that way, I don't even get anything valuable in the logs
if it's failing to try to set a netboot.

Seems like this has been solved or documented by someone somewhere.

On Wed, Aug 5, 2015 at 7:39 AM, Alvin Starr  wrote:
> It was my impression that fence is just to shut a machine down as part of
> the HA/clustering toolset.
> All the other features are just added extras that are not needed as part
> of the primary function.
>
> The features from man include on,off,reboot,status,diag,list,monitor.
>
> There is no "set to boot from PXE".
> I believe that you will need to do this manually.
>
> On 08/05/2015 05:36 AM, Jennifer Patrick wrote:
>
> Just having some woes with making fence_ipmilan work with some Supermicro
> servers (Don't say use the supermicro IPMI Tool, we're trying to get around
> that.)
>
> The problem is, using fence_ipmilan, we can initiate a boot (off then on)
> from the web GUI, or from the CLI via...
>
> *# cobbler system reboot --name=test123*
> *task started: 2015-08-05_092815_power*
> *task started (id=Power management (reboot), time=Wed Aug  5 09:28:15
> 2015)*
> *cobbler power configuration is:*
>  type   : ipmilan*
>  address: 192.168.1.5*
>  user   : user*
>  id     :*
>
> *running: /usr/sbin/fence_ipmilan*
> *running: /usr/sbin/fence_ipmilan*
> *received on stdout: Powering off machine @ IPMI:**192.168.1.5**...Failed*
>
> *received on stderr:*
> *running: /usr/sbin/fence_ipmilan*
> *received on stdout: Powering off machine @ IPMI:**192.168.1.5**...Done*
>
> *received on stderr:*
> *cobbler power configuration is:*
>  type   : ipmilan*
>  address: **192.168.1.5*
>  user   : user*
>  id     :*
>
> *running: /usr/sbin/fence_ipmilan*
> *received on stdout: Powering on machine @ IPMI:**192.168.1.5**...Failed*
>
> *received on stderr: ipmilan: Power still off*
>
> *running: /usr/sbin/fence_ipmilan*
> *received on stdout: Powering on machine @ IPMI:192.168.1.5...Done*
>
> *received on stderr:*
> **** TASK COMPLETE ****
>
> So that's great.  My real question comes from the fact that I can't make
> cobbler set these machines to reboot via the Web GUI to Netboot/PXE.  It
> just seems to never issue the right command via fence_ipmilan (which
> actually uses IPMITOOL)
>
> I can do it fine in IPMITOOL:
> ipmitool -H 192.168.1.5 -U user  -I lanplus -P password chassis bootdev
>
> pxe ipmitool -H 192.168.1.5 -U user  -I lanplus -P password chassis power
> reset *
>
> And the system reboots into PXE boot mode.
>
> So is there something I have to mangle in
> /etc/cobbler/fence_ipmilan.template?
>
> Something Else?
>
> All help is appreciated.
>
> Thanks.
>
> JLP
>
> cobbler mailing [e-mail]://lists.fedorahosted.org/mailman/listinfo/cobbler
>
> --
> Alvin Starr                   ||   voice: [phone]
> Netvel Inc.                   ||   Cell:  (416)[e-mail]              ||
>
> cobbler mailing list
> [e-mail]
> https://lists.fedorahosted.org/mailman/listinfo/cobbler

[Scott Mattan]

Jennifer,

You are specifying that on a system already registered in cobbler that
you want to be able to switch the BIOS setting for boot source?  As Alvin
mentioned, at the moment this is not possible, as an alternative you could
script it yourself... I am however, curious to understand the reason for
your needing this functionality provided that you can tell me.

I am not a developer working on cobbler... not yet anyway,  however,
since I am a heavy user of cobbler and follow this list very closely I was
curious about your post...

The reason why I am curious is mainly due to the fact that once your
server is set to boot from PXE you shouldn't need to have it changed.
Unless you are pre-defining your servers in cobbler and then using it to
deploy each one?

My set up uses the default system settings to install a base operating
system, automatically register the system to cobbler, determine the system
type (virtual or physical) , reinstall the correct OS/Profile, and then
configure it automatically.  This allows me to get my servers from the
factory preset to PXE, press the power button and go.

I would like to see various functionality like this integrated into the
cobbler framework however... of particular interest is turning on and off
CPU components i.e. hyperthreading or virtualization at the BIOS level.

Scott

On Wed, Aug 5, 2015 at 8:44 PM, Jennifer Patrick  wrote:
> Howow would you suggest going about that...
>
> Aside from somehow inserting some manual bit into the python code to do
> this?  And even then, what and where?
>
> And why doesn't the CLI work for things like this...?
>
> cobbler system edit --name=test123 --netboot-enabled=true
>
> cobbler system reboot --name=test123
>
> Does not produce a netboot.  But ipmitool wiil.
>
> When I do try it that way, I don't even get anything valuable in the logs
> if it's failing to try to set a netboot.
>
> Seems like this has been solved or documented by someone somewhere.
>
> On Wed, Aug 5, 2015 at 7:39 AM, Alvin Starr  wrote:
> > It was my impression that fence is just to shut a machine down as part of
> > the HA/clustering toolset.
> > All the other features are just added extras that are not needed as part
> > of the primary function.
> >
> > The features from man include on,off,reboot,status,diag,list,monitor.
> >
> > There is no "set to boot from PXE".
> > I believe that you will need to do this manually.
> >
> > On 08/05/2015 05:36 AM, Jennifer Patrick wrote:
> >
> > Just having some woes with making fence_ipmilan work with some Supermicro
> > servers (Don't say use the supermicro IPMI Tool, we're trying to get around
> > that.)
> >
> > The problem is, using fence_ipmilan, we can initiate a boot (off then on)
> > from the web GUI, or from the CLI via...
> >
> > *# cobbler system reboot --name=test123*
> > *task started: 2015-08-05_092815_power*
> > *task started (id=Power management (reboot), time=Wed Aug  5 09:28:15
> > 2015)*
> > *cobbler power configuration is:*
> >  type   : ipmilan*
> >  address: 192.168.1.5*
> >  user   : user*
> >  id     :*
> >
> > *running: /usr/sbin/fence_ipmilan*
> > *running: /usr/sbin/fence_ipmilan*
> > *received on stdout: Powering off machine @ IPMI:**192.168.1.5*
> > *...Failed*
> >
> > *received on stderr:*
> > *running: /usr/sbin/fence_ipmilan*
> > *received on stdout: Powering off machine @ IPMI:**192.168.1.5**...Done*
> >
> > *received on stderr:*
> > *cobbler power configuration is:*
> >  type   : ipmilan*
> >  address: **192.168.1.5*
> >  user   : user*
> >  id     :*
> >
> > *running: /usr/sbin/fence_ipmilan*
> > *received on stdout: Powering on machine @ IPMI:**192.168.1.5**...Failed*
> >
> > *received on stderr: ipmilan: Power still off*
> >
> > *running: /usr/sbin/fence_ipmilan*
> > *received on stdout: Powering on machine @ IPMI:192.168.1.5...Done*
> >
> > *received on stderr:*
> > **** TASK COMPLETE ****
> >
> > So that's great.  My real question comes from the fact that I can't make
> > cobbler set these machines to reboot via the Web GUI to Netboot/PXE.  It
> > just seems to never issue the right command via fence_ipmilan (which
> > actually uses IPMITOOL)
> >
> > I can do it fine in IPMITOOL:
> > ipmitool -H 192.168.1.5 -U user  -I lanplus -P password chassis bootdev
> >
> > pxe ipmitool -H 192.168.1.5 -U user  -I lanplus -P password chassis power
> > reset *
> >
> > And the system reboots into PXE boot mode.
> >
> > So is there something I have to mangle in
> > /etc/cobbler/fence_ipmilan.template?
> >
> > Something Else?
> >
> > All help is appreciated.
> >
> > Thanks.
> >
> > JLP
> >
> > cobbler mailing [e-mail]://lists.fedorahosted.org/mailman/listinfo/cobbler
> >
> > --
> > Alvin Starr                   ||   voice: [phone]
> > Netvel Inc.                   ||   Cell:  (416)[e-mail]              ||
> >
> > cobbler mailing list
> > [e-mail]
> > https://lists.fedorahosted.org/mailman/listinfo/cobbler
>
> --
>
> [image: Dyn logo, Dyn.com] http://dyn.com/
> http://twitter.com/dyn    http://twitter.com/dyninc[image: Dyn
> facebook account] http://facebook.com/dyn    http://twitter.com/dyninc[image:
> Dyn LinkedIn account] http://linkedin.com/company/dyn
>
> Jennifer Patrick  / Sr. Systems Engineer
>  617-335-6284
>
> cobbler mailing list
> [e-mail]
> https://lists.fedorahosted.org/mailman/listinfo/cobbler

[Nishanth Aravamudan]

On 06.08.2015 [14:32:47 +0900], Scott Mattan wrote:
> Jennifer,
> You are specifying that on a system already registered in cobbler
>
> that you want to be able to switch the BIOS setting for boot source?
> As Alvin mentioned, at the moment this is not possible, as an
> alternative you could script it yourself... I am however, curious to
> understand the reason for your needing this functionality provided
> that you can tell me.

I assumed, based upon the context in the first e-mail, that Jennifer's
SuperMicro systems (note also mentioned working around the SuperMicro
tool) are somehow "broken" at the BIOS level and don't maintain this
setting.
> The reason why I am curious is mainly due to the fact that once
>
> your server is set to boot from PXE you shouldn't need to have it
> changed.  Unless you are pre-defining your servers in cobbler and then
> using it to deploy each one?

I agree with you -- it's unclear why that's a necessary step.
> My set up uses the default system settings to install a base
>
> operating system, automatically register the system to cobbler,
> determine the system type (virtual or physical) , reinstall the
> correct OS/Profile, and then configure it automatically.  This allows
> me to get my servers from the factory preset to PXE, press the power
> button and go.

It would be great to see an article about your setup (presuming your
allowed by your employer to publish it) or maybe contrib some of your
tooling around cobbler!
> I would like to see various functionality like this integrated
>
> into the cobbler framework however... of particular interest is
> turning on and off CPU components i.e. hyperthreading or
> virtualization at the BIOS level.

I'd be pretty impressed if BIOS level settings are so
standardized/exposed that it could really be integrated -- and would
actually work -- in Cobbler!

I'm not sure how much makes sense to integrate into cobbler proper
(given that cobbler's sole goal should be installation of clients, not
the configuration of the clients in some sense) but maybe in snippets
(if the BIOS is exposed to the client from within the client).

-Nish

[Scott Mattan]

Nish,

Sorry for the late response, I would love to write an article on my setup,
unfortunately, I am bound by company restrictions.

That being said, it isn't a customer facing setup so the process does have
some security issues that I need to eventually work out.
(although being on an isolated network, I am not too worried)

I would also love to commit to the project, although at the moment my
python skills are non-existant so everything that my setup is running on is
either BASH, AWK, SED, or Ruby scripts.

It may be possible to order your servers with SeaBios(also known as
coreboot or openbios), which is open source legacy BIOS implementation and
therefore you should be able to write code against it... I haven't taken a
look into it much myself yet. (although I have heard that there are still a
good many issues with some of the more advanced features)

My reasoning for wanting to see the ability to configure the BIOS is that
once you have the Server set to PXE boot (preferably factory set) I would
rather not have to jump into the BIOS of each individual server to change a
single setting here and or there based on some SLA that is in place.
Although I realize that this is more of a far off dream at the moment for
cobbler, I know that using proprietary tools that this likely currently
being done by industry giants such as Google and Amazon.

In anycase, I will keep my eye open, and if I have the chance to contribute
something, even if its a simple bash script (albeit slightly edited as my
current scripts have a couple of issues related to security as mentioned
earlier) I will do so.

Scott Mattan

On Fri, Aug 7, 2015 at 12:42 AM, Nishanth Aravamudan <
[e-mail]> wrote:
> On 06.08.2015 [14:32:47 +0900], Scott Mattan wrote:
> > Jennifer,
> > You are specifying that on a system already registered in cobbler
> >
> > that you want to be able to switch the BIOS setting for boot source?
> > As Alvin mentioned, at the moment this is not possible, as an
> > alternative you could script it yourself... I am however, curious to
> > understand the reason for your needing this functionality provided
> > that you can tell me.
>
> I assumed, based upon the context in the first e-mail, that Jennifer's
> SuperMicro systems (note also mentioned working around the SuperMicro
> tool) are somehow "broken" at the BIOS level and don't maintain this
> setting.
> > The reason why I am curious is mainly due to the fact that once
> >
> > your server is set to boot from PXE you shouldn't need to have it
> > changed.  Unless you are pre-defining your servers in cobbler and then
> > using it to deploy each one?
>
> I agree with you -- it's unclear why that's a necessary step.
> > My set up uses the default system settings to install a base
> >
> > operating system, automatically register the system to cobbler,
> > determine the system type (virtual or physical) , reinstall the
> > correct OS/Profile, and then configure it automatically.  This allows
> > me to get my servers from the factory preset to PXE, press the power
> > button and go.
>
> It would be great to see an article about your setup (presuming your
> allowed by your employer to publish it) or maybe contrib some of your
> tooling around cobbler!
> > I would like to see various functionality like this integrated
> >
> > into the cobbler framework however... of particular interest is
> > turning on and off CPU components i.e. hyperthreading or
> > virtualization at the BIOS level.
>
> I'd be pretty impressed if BIOS level settings are so
> standardized/exposed that it could really be integrated -- and would
> actually work -- in Cobbler!
>
> I'm not sure how much makes sense to integrate into cobbler proper
> (given that cobbler's sole goal should be installation of clients, not
> the configuration of the clients in some sense) but maybe in snippets
> (if the BIOS is exposed to the client from within the client).
>
> -Nish

[Nishanth Aravamudan]

On 05.08.2015 [05:36:29 -0400], Jennifer Patrick wrote:
> Just having some woes with making fence_ipmilan work with some Supermicro
> servers (Don't say use the supermicro IPMI Tool, we're trying to get around
> that.)
>
> The problem is, using fence_ipmilan, we can initiate a boot (off then on)
> from the web GUI, or from the CLI via...
>
> *# cobbler system reboot --name=test123*
> *task started: 2015-08-05_092815_power*
> *task started (id=Power management (reboot), time=Wed Aug  5 09:28:15 2015)*
> *cobbler power configuration is:*
>  type   : ipmilan*
>  address: 192.168.1.5*
>  user   : user*
>  id     :*
>
> *running: /usr/sbin/fence_ipmilan*
> *running: /usr/sbin/fence_ipmilan*
> *received on stdout: Powering off machine @ IPMI:**192.168.1.5**...Failed*
>
> *received on stderr:*
> *running: /usr/sbin/fence_ipmilan*
> *received on stdout: Powering off machine @ IPMI:**192.168.1.5**...Done*
>
> *received on stderr:*
> *cobbler power configuration is:*
>  type   : ipmilan*
>  address: **192.168.1.5*
>  user   : user*
>  id     :*
>
> *running: /usr/sbin/fence_ipmilan*
> *received on stdout: Powering on machine @ IPMI:**192.168.1.5**...Failed*
>
> *received on stderr: ipmilan: Power still off*
>
> *running: /usr/sbin/fence_ipmilan*
> *received on stdout: Powering on machine @ IPMI:192.168.1.5...Done*
>
> *received on stderr:*
> **** TASK COMPLETE ****
>
> So that's great.  My real question comes from the fact that I can't make
> cobbler set these machines to reboot via the Web GUI to Netboot/PXE.  It
> just seems to never issue the right command via fence_ipmilan (which
> actually uses IPMITOOL)
>
> I can do it fine in IPMITOOL:
>
> *ipmitool -H 192.168.1.5 -U user  -I lanplus -P password chassis bootdev
> pxe

Does this imply that you have to set the bootdev to pxe *every* time?
> ipmitool -H 192.168.1.5 -U user  -I lanplus -P password chassis power
> reset*

This is essentially the only command that Cobbler ends up invoking, I
believe.

I think you will have to write a wrapper around fence_ipmilan (say
fence_supermicro) that basically does everything the same as
fence_ipmilan (so it could just source and inherit) except that reboot
sets the pxe boot mode first, then falls through to what fence_ipmilan
does.

-Nish