Adding support for manually configured fence_kdump

Hi,

we want to integrate support for fence_kdump into oVirt project. Marek Grac pointed me
to kexec-tools-2.0.4-23 which contains automatic configuration of fence_kdump from Pacemaker config.
Unfortunately for us, this support is tightly bound to Pacemaker, so I created attached patch,
which adds support for manual configuration of fence_kdump:

1) Current status without this patch (kexec-tools-2.0.4-23)
     - when running 'kdumpctl restart' kdump detects if fence_kdump_send command is installed
       and if so, it tries to:

a) Create /etc/sysconfig/fence_kdump file with command line parameters for fence_kdump_send.
            These parameters are extracted from cluster configuration using 'pcs cluster' command
         b) Create /etc/fence_kdump_nodes file with list of hosts to send fence_kdump notification to.
            These hosts are also extracted from cluster configuration using 'pcs cluster' command
         c) If these two files are successfully created, kdump tries to configure network to be able
            to reach all hosts defined in /etc/fence_kdump_nodes
         d) Install these two config files into kdump's kernel initramfs

2) Behavior with this patch
     - when running 'kdumpctl restart' kdump detect if fence_kdump_send binary is installed
       and if so, it tries to:

a) It tries to detect is cluster is configured ('pcs' command is available). If so, it creates
            needed config files as described above in steps 1a) and 1b)
         b) If cluster is not configured, it tries to detect if /etc/sysconfig/fence_kdump and
            /etc/sysconfig/fence_kdump_nodes exist. If so, fence_kdump is marked as successfully
            configured and kdump continues with steps 1c) and 1d)

Is it possible to integrate such patch into kexec-tools RPM?

We would like to integrate fence_kdump support into oVirt 3.5 which should be released in May,
so before this to happen, we need kexec-tools RPM with this patch to be available for RHEL 6 and
Fedora 19/20.

Thanks

Martin Perina

Patch in attachment was filtered so I resent it inline:

----- Original Message -----
> From: "Martin Perina" 
> To: [e-mail]
> Sent: Thursday, March 13, 2014 7:47:08 AM
> Subject: Adding support for manually configured fence_kdump
>
> Hi,
>
> we want to integrate support for fence_kdump into oVirt project. Marek Grac
> pointed me
> to kexec-tools-2.0.4-23 which contains automatic configuration of fence_kdump
> from Pacemaker config.
> Unfortunately for us, this support is tightly bound to Pacemaker, so I
> created attached patch,
> which adds support for manual configuration of fence_kdump:
>
> Current status without this patch (kexec-tools-2.0.4-23)
> when running 'kdumpctl restart' kdump detects if fence_kdump_send
>
> command is installed
>     and if so, it tries to:
>   a) Create /etc/sysconfig/fence_kdump file with command line
>   parameters for fence_kdump_send.
>      These parameters are extracted from cluster configuration using
>      'pcs cluster' command
>   b) Create /etc/fence_kdump_nodes file with list of hosts to send
>   fence_kdump notification to.
>      These hosts are also extracted from cluster configuration using
>      'pcs cluster' command
>   c) If these two files are successfully created, kdump tries to
>   configure network to be able
>      to reach all hosts defined in /etc/fence_kdump_nodes
>   d) Install these two config files into kdump's kernel initramfs
>
>
> Behavior with this patch
> when running 'kdumpctl restart' kdump detect if fence_kdump_send
>
> binary is installed
>     and if so, it tries to:
>   a) It tries to detect is cluster is configured ('pcs' command is
>   available). If so, it creates
>      needed config files as described above in steps 1a) and 1b)
>   b) If cluster is not configured, it tries to detect if
>   /etc/sysconfig/fence_kdump and
>      /etc/sysconfig/fence_kdump_nodes exist. If so, fence_kdump is
>      marked as successfully
>      configured and kdump continues with steps 1c) and 1d)
>
>
> Is it possible to integrate such patch into kexec-tools RPM?
>
> We would like to integrate fence_kdump support into oVirt 3.5 which should be
> released in May,
> so before this to happen, we need kexec-tools RPM with this patch to be
> available for RHEL 6 and
> Fedora 19/20.
>
> Thanks
>
> Martin Perina
> _______________________________________________
> kexec mailing list
> [e-mail]
> https://lists.fedoraproject.org/mailman/listinfo/kexec

diff -rupN a/dracut-module-setup.sh b/dracut-module-setup.sh
--- a/dracut-module-setup.sh	2014-02-17 06:50:46.000000000 +0100
+++ b/dracut-module-setup.sh	2014-03-12 08:38:03.000000000 +0100
@@ -423,26 +423,38 @@ kdump_check_fence_kdump () {
     local nodes
     is_fence_kdump || return 1

-    # get cluster nodes from cluster cib, get interface and ip address
-    nodelist=`pcs cluster cib | xmllint --xpath "/cib/status/node_state/@uname" -`
+    if is_cluster; then
+        # get cluster nodes from cluster cib, get interface and ip address
+        nodelist=`pcs cluster cib | xmllint --xpath "/cib/status/node_state/@uname" -`

-    # nodelist is formed as 'uname="node1" uname="node2" ... uname="nodeX"'
-    # we need to convert each to node1, node2 ... nodeX in each iteration
-    for node in ${nodelist}; do
-        # convert $node from 'uname="nodeX"' to 'nodeX'
-        eval $node
-        nodename=$uname
-        # Skip its own node name
-        if [ "$nodename" = `hostname` ]; then
-            continue
-        fi
-        nodes="$nodes $nodename"
+        # nodelist is formed as 'uname="node1" uname="node2" ... uname="nodeX"'
+        # we need to convert each to node1, node2 ... nodeX in each iteration
+        for node in ${nodelist}; do
+            # convert $node from 'uname="nodeX"' to 'nodeX'
+            eval $node
+            nodename=$uname
+            # Skip its own node name
+            if [ "$nodename" = `hostname` ]; then
+                continue
+            fi
+            nodes="$nodes $nodename"
+        done
+    else
+        # fence_kdump configured manually
+        read nodes < $FENCE_KDUMP_NODES
+    fi

-        kdump_install_net $nodename
+    # install network for each node
+    for node in ${nodes}; do
+        kdump_install_net $node
     done
+
     echo

-    echo "$nodes" > ${initdir}/$FENCE_KDUMP_NODES
+    mkdir -p ${initdir}$FENCE_KDUMP_NODES
+    rmdir ${initdir}$FENCE_KDUMP_NODES
+    echo "$nodes" > ${initdir}$FENCE_KDUMP_NODES
+
     dracut_install $FENCE_KDUMP_SEND
     dracut_install -o $FENCE_KDUMP_CONFIG
 }
diff -rupN a/kdumpctl b/kdumpctl
--- a/kdumpctl	2014-02-17 06:50:46.000000000 +0100
+++ b/kdumpctl	2014-03-12 10:36:26.829734090 +0100
@@ -133,7 +133,8 @@ function check_config()
 }

# check_fence_kdump <image timestamp>
-# return 0 if fence_kdump is configured and kdump initrd needs to be rebuilt
+# return 0 if fence_kdump is configured in cluster and kdump initrd needs
+# to be rebuilt
 function check_fence_kdump()
 {
    local image_time=$1
@@ -141,6 +142,8 @@ function check_fence_kdump()

is_fence_kdump || return 1

+	is_cluster || return 1
+
    cib_time=`pcs cluster cib | xmllint --xpath 'string(/cib/@cib-last-written)' - | \
    	  xargs -0 date +%s --date`

@@ -199,6 +202,10 @@ function check_rebuild()
    if [ -f $FENCE_KDUMP_CONFIG ]; then
    	files="$files $FENCE_KDUMP_CONFIG"
    fi
+	
+	if [ -f $FENCE_KDUMP_NODES ]; then
+		files="$files $FENCE_KDUMP_NODES"
+	fi

check_exist "$files" && check_executable "$EXTRA_BINS"
    [ $? -ne 0 ] && return 1
diff -rupN a/kdump-lib.sh b/kdump-lib.sh
--- a/kdump-lib.sh	2014-02-17 06:50:46.000000000 +0100
+++ b/kdump-lib.sh	2014-03-12 10:38:29.108892093 +0100
@@ -5,7 +5,7 @@

FENCE_KDUMP_CONFIG="/etc/sysconfig/fence_kdump"
 FENCE_KDUMP_SEND="/usr/libexec/fence_kdump_send"
-FENCE_KDUMP_NODES="/etc/fence_kdump_nodes"
+FENCE_KDUMP_NODES="/etc/sysconfig/fence_kdump_nodes"

is_ssh_dump_target()
 {
@@ -27,13 +27,25 @@ strip_comments()
     echo $@ | sed -e 's/(.*)#.*/\1/'
 }

+# Check if cluster environment is present
+is_cluster()
+{
+    # no pcs executable installed
+    type -P pcs > /dev/null || return 1
+}
+
 # Check if fence kdump is configured in cluster
 is_fence_kdump()
 {
-    # no pcs or fence_kdump_send executables installed?
-    type -P pcs > /dev/null || return 1
+    # no fence_kdump_send executables installed?
     [ -x $FENCE_KDUMP_SEND ] || return 1

-    # fence kdump not configured?
-    (pcs cluster cib | grep -q 'type="fence_kdump"') &> /dev/null || return 1
+    if is_cluster; then
+        # fence kdump configured in cluster?
+        (pcs cluster cib | grep -q 'type="fence_kdump"') &> /dev/null || return 1
+    else
+        # fence kdump configured manually?
+        [ -f $FENCE_KDUMP_CONFIG ] || return 1
+        [ -f $FENCE_KDUMP_NODES ] || return 1
+    fi
 }

[WANG Chao]

On 03/13/14 at 02:47am, Martin Perina wrote:
> Hi,
>
> we want to integrate support for fence_kdump into oVirt project. Marek Grac pointed me
> to kexec-tools-2.0.4-23 which contains automatic configuration of fence_kdump from Pacemaker config.
> Unfortunately for us, this support is tightly bound to Pacemaker, so I created attached patch,
> which adds support for manual configuration of fence_kdump:
>
> Current status without this patch (kexec-tools-2.0.4-23)
>
> when running 'kdumpctl restart' kdump detects if fence_kdump_send command is installed
> and if so, it tries to:
>
> a) Create /etc/sysconfig/fence_kdump file with command line parameters for fence_kdump_send.
>      These parameters are extracted from cluster configuration using 'pcs cluster' command

We don't create /etc/sysconfig/fence_kdump. Instead it's created by
fence kdump end user who want to configure fence_kdump_send to run with
options rather than default. And we install it to 2nd kernel and source
it in our kdump.sh if it exists in 1st kernel.

In 2nd kernel, we run fence_kdump_send as the following:

$FENCE_KDUMP_SEND $FENCE_KDUMP_OPTS host1 host2 ... hostN

As you probably know, $FENCE_KDUMP_SEND stands for
/usr/libexec/fence_kdump_send. So the command is pretty much as

/usr/libexec/fence_kdump_send $FENCE_KDUMP_OPTS host1...hostN

n /etc/sysconfig/fence_kdump doesn't exist, $FENCE_KDUMP_SEND is
empty, fence_kdump_send is called with no options, but hosts list:

/usr/lib/exec/fence_kdump_send host1...hostX

When user creates /etc/sysconfig/fence_kdump and configures
FENCE_KDUMP_OPTS to something like "-c 2 -i 10 -v" in it,
fence_kdump_send is called as:

/usr/libexec/fence_kdump_send -c 2 -i 10 -v host1...hostN
>      b) Create /etc/fence_kdump_nodes file with list of hosts to send fence_kdump notification to.
>         These hosts are also extracted from cluster configuration using 'pcs cluster' command
>      c) If these two files are successfully created, kdump tries to configure network to be able
>         to reach all hosts defined in /etc/fence_kdump_nodes

Not exactly. We configure network accordingly if is_fence_kdump().

is_fence_kdump() checks if cluster and if cluster is configured fence
kdump in cluster cib.
>      d) Install these two config files into kdump's kernel initramfs

/etc/fence_kdump_nodes is mandatory while /etc/sysconfig/kdump_fence is
optional depending on if it exists.
> Behavior with this patch
>
> when running 'kdumpctl restart' kdump detect if fence_kdump_send binary is installed
> and if so, it tries to:
>
> a) It tries to detect is cluster is configured ('pcs' command is available). If so, it creates
>      needed config files as described above in steps 1a) and 1b)

Please don't create /etc/sysconfig/fence_kdump. If it doesn't in 1st,
there's no need to create one for the 2nd kernel. We only source it when
it exists in 2nd kernel.
>      b) If cluster is not configured, it tries to detect if /etc/sysconfig/fence_kdump and
>         /etc/sysconfig/fence_kdump_nodes exist. If so, fence_kdump is marked as successfully
>         configured and kdump continues with steps 1c) and 1d)

Is /etc/sysconfig/fence_kdump_nodes created by the user who want to use
fence_kdump facility? Even without being in a cluster environment? Does
oVirt work with fence_kdump_send?

Thanks
WANG Chao
> Is it possible to integrate such patch into kexec-tools RPM?
>
> We would like to integrate fence_kdump support into oVirt 3.5 which should be released in May,
> so before this to happen, we need kexec-tools RPM with this patch to be available for RHEL 6 and
> Fedora 19/20.
>
> Thanks
>
> Martin Perina
> _______________________________________________
> kexec mailing list
> [e-mail]
> https://lists.fedoraproject.org/mailman/listinfo/kexec

----- Original Message -----
> From: "WANG Chao" 
> To: "Martin Perina" 
> Cc: [e-mail]
> Sent: Thursday, March 13, 2014 8:50:26 AM
> Subject: Re: Adding support for manually configured fence_kdump
>
> On 03/13/14 at 02:47am, Martin Perina wrote:
> > Hi,
> >
> > we want to integrate support for fence_kdump into oVirt project. Marek Grac
> > pointed me
> > to kexec-tools-2.0.4-23 which contains automatic configuration of
> > fence_kdump from Pacemaker config.
> > Unfortunately for us, this support is tightly bound to Pacemaker, so I
> > created attached patch,
> > which adds support for manual configuration of fence_kdump:
> >
> > Current status without this patch (kexec-tools-2.0.4-23)
> > when running 'kdumpctl restart' kdump detects if fence_kdump_send
> >
> > command is installed
> >     and if so, it tries to:
> >   a) Create /etc/sysconfig/fence_kdump file with command line
> >   parameters for fence_kdump_send.
> >      These parameters are extracted from cluster configuration using
> >      'pcs cluster' command
>
> We don't create /etc/sysconfig/fence_kdump. Instead it's created by
> fence kdump end user who want to configure fence_kdump_send to run with
> options rather than default. And we install it to 2nd kernel and source
> it in our kdump.sh if it exists in 1st kernel.

That's true, my mistake.
> In 2nd kernel, we run fence_kdump_send as the following:
>
> $FENCE_KDUMP_SEND $FENCE_KDUMP_OPTS host1 host2 ... hostN
>
> As you probably know, $FENCE_KDUMP_SEND stands for
> /usr/libexec/fence_kdump_send. So the command is pretty much as
>
> /usr/libexec/fence_kdump_send $FENCE_KDUMP_OPTS host1...hostN
>
> n /etc/sysconfig/fence_kdump doesn't exist, $FENCE_KDUMP_SEND is
> empty, fence_kdump_send is called with no options, but hosts list:

OK, so I can modify my patch, that for manual configuration only
/usr/libexec/fence_kdump_send and /etc/sysconfig/fence_kdump_hosts
will be mandatory
> /usr/lib/exec/fence_kdump_send host1...hostX
>
> When user creates /etc/sysconfig/fence_kdump and configures
> FENCE_KDUMP_OPTS to something like "-c 2 -i 10 -v" in it,
> fence_kdump_send is called as:
>
> /usr/libexec/fence_kdump_send -c 2 -i 10 -v host1...hostN
> >      b) Create /etc/fence_kdump_nodes file with list of hosts to send
> >      fence_kdump notification to.
> >         These hosts are also extracted from cluster configuration using
> >         'pcs cluster' command
> >      c) If these two files are successfully created, kdump tries to
> >      configure network to be able
> >         to reach all hosts defined in /etc/fence_kdump_nodes
>
> Not exactly. We configure network accordingly if is_fence_kdump().

But you run kdump_install_net function for each node
> is_fence_kdump() checks if cluster and if cluster is configured fence
> kdump in cluster cib.

I modified this, is_fence_kdump() now checks:

1) if cluster and if cluster is configured fence kdump in cluster cib
2) or if manual configuration exists
> >      d) Install these two config files into kdump's kernel initramfs
>
> /etc/fence_kdump_nodes is mandatory while /etc/sysconfig/kdump_fence is
> optional depending on if it exists.

True
> > Behavior with this patch
> > when running 'kdumpctl restart' kdump detect if fence_kdump_send
> >
> > binary is installed
> >     and if so, it tries to:
> >   a) It tries to detect is cluster is configured ('pcs' command is
> >   available). If so, it creates
> >      needed config files as described above in steps 1a) and 1b)
>
> Please don't create /etc/sysconfig/fence_kdump. If it doesn't in 1st,
> there's no need to create one for the 2nd kernel. We only source it when
> it exists in 2nd kernel.

This file will be created manually (oVirt will most probably alter defaults
of fence_kdump_send) and if it exists it will be installed into initramfs.
This behavior exists before my patch
> >      b) If cluster is not configured, it tries to detect if
> >      /etc/sysconfig/fence_kdump and
> >         /etc/sysconfig/fence_kdump_nodes exist. If so, fence_kdump is
> >         marked as successfully
> >         configured and kdump continues with steps 1c) and 1d)
>
> Is /etc/sysconfig/fence_kdump_nodes created by the user who want to use
> fence_kdump facility? Even without being in a cluster environment? Does
> oVirt work with fence_kdump_send?

That's the reason why I wrote this patch. We want to support fence_kdump
in oVirt, but we don't want to install and configure Pacemaker on oVirt hosts,
so this patch tries to add 2nd scenario:

1) Host is part of Pacemaker cluster - use pcs command to get list of hosts

2) Host is not part of Pacemake cluster - /etc/sysconfig/fence_kdump_nodes
     will be created manually by user or automatically by oVirt
> Thanks
> WANG Chao
> > Is it possible to integrate such patch into kexec-tools RPM?
> >
> > We would like to integrate fence_kdump support into oVirt 3.5 which should
> > be released in May,
> > so before this to happen, we need kexec-tools RPM with this patch to be
> > available for RHEL 6 and
> > Fedora 19/20.
> >
> > Thanks
> >
> > Martin Perina
> > _______________________________________________
> > kexec mailing list
> > [e-mail]
> > https://lists.fedoraproject.org/mailman/listinfo/kexec

[WANG Chao]

On 03/13/14 at 04:09am, Martin Perina wrote:
> ----- Original Message -----
> > From: "WANG Chao" 
> > To: "Martin Perina" 
> > Cc: [e-mail]
> > Sent: Thursday, March 13, 2014 8:50:26 AM
> > Subject: Re: Adding support for manually configured fence_kdump
> >
> > On 03/13/14 at 02:47am, Martin Perina wrote:
> > > Hi,
> > >
> > > we want to integrate support for fence_kdump into oVirt project. Marek Grac
> > > pointed me
> > > to kexec-tools-2.0.4-23 which contains automatic configuration of
> > > fence_kdump from Pacemaker config.
> > > Unfortunately for us, this support is tightly bound to Pacemaker, so I
> > > created attached patch,
> > > which adds support for manual configuration of fence_kdump:
> > >
> > > Current status without this patch (kexec-tools-2.0.4-23)
> > > when running 'kdumpctl restart' kdump detects if fence_kdump_send
> > >
> > > command is installed
> > >     and if so, it tries to:
> > >   a) Create /etc/sysconfig/fence_kdump file with command line
> > >   parameters for fence_kdump_send.
> > >      These parameters are extracted from cluster configuration using
> > >      'pcs cluster' command
> >
> > We don't create /etc/sysconfig/fence_kdump. Instead it's created by
> > fence kdump end user who want to configure fence_kdump_send to run with
> > options rather than default. And we install it to 2nd kernel and source
> > it in our kdump.sh if it exists in 1st kernel.
>
> That's true, my mistake.
> > In 2nd kernel, we run fence_kdump_send as the following:
> >
> > $FENCE_KDUMP_SEND $FENCE_KDUMP_OPTS host1 host2 ... hostN
> >
> > As you probably know, $FENCE_KDUMP_SEND stands for
> > /usr/libexec/fence_kdump_send. So the command is pretty much as
> >
> > /usr/libexec/fence_kdump_send $FENCE_KDUMP_OPTS host1...hostN
> >
> > n /etc/sysconfig/fence_kdump doesn't exist, $FENCE_KDUMP_SEND is
> > empty, fence_kdump_send is called with no options, but hosts list:
>
> OK, so I can modify my patch, that for manual configuration only
> /usr/libexec/fence_kdump_send and /etc/sysconfig/fence_kdump_hosts
> will be mandatory
> > /usr/lib/exec/fence_kdump_send host1...hostX
> >
> > When user creates /etc/sysconfig/fence_kdump and configures
> > FENCE_KDUMP_OPTS to something like "-c 2 -i 10 -v" in it,
> > fence_kdump_send is called as:
> >
> > /usr/libexec/fence_kdump_send -c 2 -i 10 -v host1...hostN
> > >      b) Create /etc/fence_kdump_nodes file with list of hosts to send
> > >      fence_kdump notification to.
> > >         These hosts are also extracted from cluster configuration using
> > >         'pcs cluster' command
> > >      c) If these two files are successfully created, kdump tries to
> > >      configure network to be able
> > >         to reach all hosts defined in /etc/fence_kdump_nodes
> >
> > Not exactly. We configure network accordingly if is_fence_kdump().
>
> But you run kdump_install_net function for each node

I was talking about "if these two files are successfully created ..."
> > is_fence_kdump() checks if cluster and if cluster is configured fence
> > kdump in cluster cib.
>
> I modified this, is_fence_kdump() now checks:
> if cluster and if cluster is configured fence kdump in cluster cib
> or if manual configuration exists

I'm fine with this change. It makes sense in your case.
> > >      d) Install these two config files into kdump's kernel initramfs
> >
> > /etc/fence_kdump_nodes is mandatory while /etc/sysconfig/kdump_fence is
> > optional depending on if it exists.
>
> True
> > > Behavior with this patch
> > > when running 'kdumpctl restart' kdump detect if fence_kdump_send
> > >
> > > binary is installed
> > >     and if so, it tries to:
> > >   a) It tries to detect is cluster is configured ('pcs' command is
> > >   available). If so, it creates
> > >      needed config files as described above in steps 1a) and 1b)
> >
> > Please don't create /etc/sysconfig/fence_kdump. If it doesn't in 1st,
> > there's no need to create one for the 2nd kernel. We only source it when
> > it exists in 2nd kernel.
>
> This file will be created manually (oVirt will most probably alter defaults
> of fence_kdump_send) and if it exists it will be installed into initramfs.
> This behavior exists before my patch
> > >      b) If cluster is not configured, it tries to detect if
> > >      /etc/sysconfig/fence_kdump and
> > >         /etc/sysconfig/fence_kdump_nodes exist. If so, fence_kdump is
> > >         marked as successfully
> > >         configured and kdump continues with steps 1c) and 1d)
> >
> > Is /etc/sysconfig/fence_kdump_nodes created by the user who want to use
> > fence_kdump facility? Even without being in a cluster environment? Does
> > oVirt work with fence_kdump_send?
>
> That's the reason why I wrote this patch. We want to support fence_kdump
> in oVirt, but we don't want to install and configure Pacemaker on oVirt hosts,
> so this patch tries to add 2nd scenario:
>
> Host is part of Pacemaker cluster - use pcs command to get list of hosts
>
> Host is not part of Pacemake cluster - /etc/sysconfig/fence_kdump_nodes
> will be created manually by user or automatically by oVirt

OK. I understand your case.

Thanks
WANG Chao
> > Thanks
> > WANG Chao
> > > Is it possible to integrate such patch into kexec-tools RPM?
> > >
> > > We would like to integrate fence_kdump support into oVirt 3.5 which should
> > > be released in May,
> > > so before this to happen, we need kexec-tools RPM with this patch to be
> > > available for RHEL 6 and
> > > Fedora 19/20.
> > >
> > > Thanks
> > >
> > > Martin Perina
> > > _______________________________________________
> > > kexec mailing list
> > > [e-mail]
> > > https://lists.fedoraproject.org/mailman/listinfo/kexec

----- Original Message -----
> From: "WANG Chao" 
> To: "Martin Perina" 
> Cc: [e-mail], "Itamar Heim" 
> Sent: Thursday, March 13, 2014 12:05:51 PM
> Subject: Re: Adding support for manually configured fence_kdump
>
> On 03/13/14 at 04:09am, Martin Perina wrote:
> > ----- Original Message -----
> > > From: "WANG Chao" 
> > > To: "Martin Perina" 
> > > Cc: [e-mail]
> > > Sent: Thursday, March 13, 2014 8:50:26 AM
> > > Subject: Re: Adding support for manually configured fence_kdump
> > >
> > > On 03/13/14 at 02:47am, Martin Perina wrote:
> > > > Hi,
> > > >
> > > > we want to integrate support for fence_kdump into oVirt project. Marek
> > > > Grac
> > > > pointed me
> > > > to kexec-tools-2.0.4-23 which contains automatic configuration of
> > > > fence_kdump from Pacemaker config.
> > > > Unfortunately for us, this support is tightly bound to Pacemaker, so I
> > > > created attached patch,
> > > > which adds support for manual configuration of fence_kdump:
> > > >
> > > > Current status without this patch (kexec-tools-2.0.4-23)
> > > > when running 'kdumpctl restart' kdump detects if
> > > >
> > > > fence_kdump_send
> > > >   command is installed
> > > >     and if so, it tries to:
> > > >   a) Create /etc/sysconfig/fence_kdump file with command line
> > > >   parameters for fence_kdump_send.
> > > >      These parameters are extracted from cluster configuration
> > > >      using
> > > >      'pcs cluster' command
> > >
> > > We don't create /etc/sysconfig/fence_kdump. Instead it's created by
> > > fence kdump end user who want to configure fence_kdump_send to run with
> > > options rather than default. And we install it to 2nd kernel and source
> > > it in our kdump.sh if it exists in 1st kernel.
> >
> > That's true, my mistake.
> > > In 2nd kernel, we run fence_kdump_send as the following:
> > >
> > > $FENCE_KDUMP_SEND $FENCE_KDUMP_OPTS host1 host2 ... hostN
> > >
> > > As you probably know, $FENCE_KDUMP_SEND stands for
> > > /usr/libexec/fence_kdump_send. So the command is pretty much as
> > >
> > > /usr/libexec/fence_kdump_send $FENCE_KDUMP_OPTS host1...hostN
> > >
> > > n /etc/sysconfig/fence_kdump doesn't exist, $FENCE_KDUMP_SEND is
> > > empty, fence_kdump_send is called with no options, but hosts list:
> >
> > OK, so I can modify my patch, that for manual configuration only
> > /usr/libexec/fence_kdump_send and /etc/sysconfig/fence_kdump_hosts
> > will be mandatory
> > > /usr/lib/exec/fence_kdump_send host1...hostX
> > >
> > > When user creates /etc/sysconfig/fence_kdump and configures
> > > FENCE_KDUMP_OPTS to something like "-c 2 -i 10 -v" in it,
> > > fence_kdump_send is called as:
> > >
> > > /usr/libexec/fence_kdump_send -c 2 -i 10 -v host1...hostN
> > > >      b) Create /etc/fence_kdump_nodes file with list of hosts to
> > > >      send
> > > >      fence_kdump notification to.
> > > >         These hosts are also extracted from cluster configuration
> > > >         using
> > > >         'pcs cluster' command
> > > >      c) If these two files are successfully created, kdump tries to
> > > >      configure network to be able
> > > >         to reach all hosts defined in /etc/fence_kdump_nodes
> > >
> > > Not exactly. We configure network accordingly if is_fence_kdump().
> >
> > But you run kdump_install_net function for each node
>
> I was talking about "if these two files are successfully created ..."
> > > is_fence_kdump() checks if cluster and if cluster is configured fence
> > > kdump in cluster cib.
> >
> > I modified this, is_fence_kdump() now checks:
> > if cluster and if cluster is configured fence kdump in cluster cib
> > or if manual configuration exists
>
> I'm fine with this change. It makes sense in your case.
> > > >      d) Install these two config files into kdump's kernel
> > > >      initramfs
> > >
> > > /etc/fence_kdump_nodes is mandatory while /etc/sysconfig/kdump_fence is
> > > optional depending on if it exists.
> >
> > True
> > > > Behavior with this patch
> > > > when running 'kdumpctl restart' kdump detect if fence_kdump_send
> > > >
> > > > binary is installed
> > > >     and if so, it tries to:
> > > >   a) It tries to detect is cluster is configured ('pcs' command
> > > >   is
> > > >   available). If so, it creates
> > > >      needed config files as described above in steps 1a) and 1b)
> > >
> > > Please don't create /etc/sysconfig/fence_kdump. If it doesn't in 1st,
> > > there's no need to create one for the 2nd kernel. We only source it when
> > > it exists in 2nd kernel.
> >
> > This file will be created manually (oVirt will most probably alter defaults
> > of fence_kdump_send) and if it exists it will be installed into initramfs.
> > This behavior exists before my patch
> > > >      b) If cluster is not configured, it tries to detect if
> > > >      /etc/sysconfig/fence_kdump and
> > > >         /etc/sysconfig/fence_kdump_nodes exist. If so, fence_kdump
> > > >         is
> > > >         marked as successfully
> > > >         configured and kdump continues with steps 1c) and 1d)
> > >
> > > Is /etc/sysconfig/fence_kdump_nodes created by the user who want to use
> > > fence_kdump facility? Even without being in a cluster environment? Does
> > > oVirt work with fence_kdump_send?
> >
> > That's the reason why I wrote this patch. We want to support fence_kdump
> > in oVirt, but we don't want to install and configure Pacemaker on oVirt
> > hosts,
> > so this patch tries to add 2nd scenario:
> > Host is part of Pacemaker cluster - use pcs command to get list of
> >
> > hosts
> > Host is not part of Pacemake cluster -
> >
> > /etc/sysconfig/fence_kdump_nodes
> >      will be created manually by user or automatically by oVirt
>
> OK. I understand your case.

OK, so what else should I do to make the patch merged?

Thanks

Martin Perina
> Thanks
> WANG Chao
> > > Thanks
> > > WANG Chao
> > > > Is it possible to integrate such patch into kexec-tools RPM?
> > > >
> > > > We would like to integrate fence_kdump support into oVirt 3.5 which
> > > > should
> > > > be released in May,
> > > > so before this to happen, we need kexec-tools RPM with this patch to be
> > > > available for RHEL 6 and
> > > > Fedora 19/20.
> > > >
> > > > Thanks
> > > >
> > > > Martin Perina
> > > > _______________________________________________
> > > > kexec mailing list
> > > > [e-mail]
> > > > https://lists.fedoraproject.org/mailman/listinfo/kexec

[WANG Chao]

On 03/13/14 at 08:07am, Martin Perina wrote:
> ----- Original Message -----
> > From: "WANG Chao" 
> > To: "Martin Perina" 
> > Cc: [e-mail], "Itamar Heim" 
> > Sent: Thursday, March 13, 2014 12:05:51 PM
> > Subject: Re: Adding support for manually configured fence_kdump
> >
> > On 03/13/14 at 04:09am, Martin Perina wrote:
> > > ----- Original Message -----
> > > > From: "WANG Chao" 
> > > > To: "Martin Perina" 
> > > > Cc: [e-mail]
> > > > Sent: Thursday, March 13, 2014 8:50:26 AM
> > > > Subject: Re: Adding support for manually configured fence_kdump
> > > >
> > > > On 03/13/14 at 02:47am, Martin Perina wrote:
> > > > > Hi,
> > > > >
> > > > > we want to integrate support for fence_kdump into oVirt project. Marek
> > > > > Grac
> > > > > pointed me
> > > > > to kexec-tools-2.0.4-23 which contains automatic configuration of
> > > > > fence_kdump from Pacemaker config.
> > > > > Unfortunately for us, this support is tightly bound to Pacemaker, so I
> > > > > created attached patch,
> > > > > which adds support for manual configuration of fence_kdump:
> > > > >
> > > > > Current status without this patch (kexec-tools-2.0.4-23)
> > > > > when running 'kdumpctl restart' kdump detects if
> > > > >
> > > > > fence_kdump_send
> > > > >   command is installed
> > > > >     and if so, it tries to:
> > > > >   a) Create /etc/sysconfig/fence_kdump file with command line
> > > > >   parameters for fence_kdump_send.
> > > > >      These parameters are extracted from cluster configuration
> > > > >      using
> > > > >      'pcs cluster' command
> > > >
> > > > We don't create /etc/sysconfig/fence_kdump. Instead it's created by
> > > > fence kdump end user who want to configure fence_kdump_send to run with
> > > > options rather than default. And we install it to 2nd kernel and source
> > > > it in our kdump.sh if it exists in 1st kernel.
> > >
> > > That's true, my mistake.
> > > > In 2nd kernel, we run fence_kdump_send as the following:
> > > >
> > > > $FENCE_KDUMP_SEND $FENCE_KDUMP_OPTS host1 host2 ... hostN
> > > >
> > > > As you probably know, $FENCE_KDUMP_SEND stands for
> > > > /usr/libexec/fence_kdump_send. So the command is pretty much as
> > > >
> > > > /usr/libexec/fence_kdump_send $FENCE_KDUMP_OPTS host1...hostN
> > > >
> > > > n /etc/sysconfig/fence_kdump doesn't exist, $FENCE_KDUMP_SEND is
> > > > empty, fence_kdump_send is called with no options, but hosts list:
> > >
> > > OK, so I can modify my patch, that for manual configuration only
> > > /usr/libexec/fence_kdump_send and /etc/sysconfig/fence_kdump_hosts
> > > will be mandatory
> > > > /usr/lib/exec/fence_kdump_send host1...hostX
> > > >
> > > > When user creates /etc/sysconfig/fence_kdump and configures
> > > > FENCE_KDUMP_OPTS to something like "-c 2 -i 10 -v" in it,
> > > > fence_kdump_send is called as:
> > > >
> > > > /usr/libexec/fence_kdump_send -c 2 -i 10 -v host1...hostN
> > > > >      b) Create /etc/fence_kdump_nodes file with list of hosts to
> > > > >      send
> > > > >      fence_kdump notification to.
> > > > >         These hosts are also extracted from cluster configuration
> > > > >         using
> > > > >         'pcs cluster' command
> > > > >      c) If these two files are successfully created, kdump tries to
> > > > >      configure network to be able
> > > > >         to reach all hosts defined in /etc/fence_kdump_nodes
> > > >
> > > > Not exactly. We configure network accordingly if is_fence_kdump().
> > >
> > > But you run kdump_install_net function for each node
> >
> > I was talking about "if these two files are successfully created ..."
> > > > is_fence_kdump() checks if cluster and if cluster is configured fence
> > > > kdump in cluster cib.
> > >
> > > I modified this, is_fence_kdump() now checks:
> > > if cluster and if cluster is configured fence kdump in cluster cib
> > > or if manual configuration exists
> >
> > I'm fine with this change. It makes sense in your case.
> > > > >      d) Install these two config files into kdump's kernel
> > > > >      initramfs
> > > >
> > > > /etc/fence_kdump_nodes is mandatory while /etc/sysconfig/kdump_fence is
> > > > optional depending on if it exists.
> > >
> > > True
> > > > > Behavior with this patch
> > > > > when running 'kdumpctl restart' kdump detect if fence_kdump_send
> > > > >
> > > > > binary is installed
> > > > >     and if so, it tries to:
> > > > >   a) It tries to detect is cluster is configured ('pcs' command
> > > > >   is
> > > > >   available). If so, it creates
> > > > >      needed config files as described above in steps 1a) and 1b)
> > > >
> > > > Please don't create /etc/sysconfig/fence_kdump. If it doesn't in 1st,
> > > > there's no need to create one for the 2nd kernel. We only source it when
> > > > it exists in 2nd kernel.
> > >
> > > This file will be created manually (oVirt will most probably alter defaults
> > > of fence_kdump_send) and if it exists it will be installed into initramfs.
> > > This behavior exists before my patch
> > > > >      b) If cluster is not configured, it tries to detect if
> > > > >      /etc/sysconfig/fence_kdump and
> > > > >         /etc/sysconfig/fence_kdump_nodes exist. If so, fence_kdump
> > > > >         is
> > > > >         marked as successfully
> > > > >         configured and kdump continues with steps 1c) and 1d)
> > > >
> > > > Is /etc/sysconfig/fence_kdump_nodes created by the user who want to use
> > > > fence_kdump facility? Even without being in a cluster environment? Does
> > > > oVirt work with fence_kdump_send?
> > >
> > > That's the reason why I wrote this patch. We want to support fence_kdump
> > > in oVirt, but we don't want to install and configure Pacemaker on oVirt
> > > hosts,
> > > so this patch tries to add 2nd scenario:
> > > Host is part of Pacemaker cluster - use pcs command to get list of
> > >
> > > hosts
> > > Host is not part of Pacemake cluster -
> > >
> > > /etc/sysconfig/fence_kdump_nodes
> > >      will be created manually by user or automatically by oVirt
> >
> > OK. I understand your case.
>
> OK, so what else should I do to make the patch merged?

Hold on for several days for giving people enough time to review. Once
Vivek, me or others has ACKed your patch, it's good to go.

If there's need for sending v2, it's preferred by git format-patch with
proper changelog and a S-O-B line.

BTW I'm starting to review your patch.

Thanks
WANG Chao
> Thanks
>
> Martin Perina
> > Thanks
> > WANG Chao
> > > > Thanks
> > > > WANG Chao
> > > > > Is it possible to integrate such patch into kexec-tools RPM?
> > > > >
> > > > > We would like to integrate fence_kdump support into oVirt 3.5 which
> > > > > should
> > > > > be released in May,
> > > > > so before this to happen, we need kexec-tools RPM with this patch to be
> > > > > available for RHEL 6 and
> > > > > Fedora 19/20.
> > > > >
> > > > > Thanks
> > > > >
> > > > > Martin Perina
> > > > > _______________________________________________
> > > > > kexec mailing list
> > > > > [e-mail]
> > > > > https://lists.fedoraproject.org/mailman/listinfo/kexec

[WANG Chao]

On 03/13/14 at 03:36am, Martin Perina wrote:
> Patch in attachment was filtered so I resent it inline:
>
> ----- Original Message -----
> > From: "Martin Perina" 
> > To: [e-mail]
> > Sent: Thursday, March 13, 2014 7:47:08 AM
> > Subject: Adding support for manually configured fence_kdump
> >
> > Hi,
> >
> > we want to integrate support for fence_kdump into oVirt project. Marek Grac
> > pointed me
> > to kexec-tools-2.0.4-23 which contains automatic configuration of fence_kdump
> > from Pacemaker config.
> > Unfortunately for us, this support is tightly bound to Pacemaker, so I
> > created attached patch,
> > which adds support for manual configuration of fence_kdump:
> >
> > Current status without this patch (kexec-tools-2.0.4-23)
> > when running 'kdumpctl restart' kdump detects if fence_kdump_send
> >
> > command is installed
> >     and if so, it tries to:
> >   a) Create /etc/sysconfig/fence_kdump file with command line
> >   parameters for fence_kdump_send.
> >      These parameters are extracted from cluster configuration using
> >      'pcs cluster' command
> >   b) Create /etc/fence_kdump_nodes file with list of hosts to send
> >   fence_kdump notification to.
> >      These hosts are also extracted from cluster configuration using
> >      'pcs cluster' command
> >   c) If these two files are successfully created, kdump tries to
> >   configure network to be able
> >      to reach all hosts defined in /etc/fence_kdump_nodes
> >   d) Install these two config files into kdump's kernel initramfs
> >
> >
> > Behavior with this patch
> > when running 'kdumpctl restart' kdump detect if fence_kdump_send
> >
> > binary is installed
> >     and if so, it tries to:
> >   a) It tries to detect is cluster is configured ('pcs' command is
> >   available). If so, it creates
> >      needed config files as described above in steps 1a) and 1b)
> >   b) If cluster is not configured, it tries to detect if
> >   /etc/sysconfig/fence_kdump and
> >      /etc/sysconfig/fence_kdump_nodes exist. If so, fence_kdump is
> >      marked as successfully
> >      configured and kdump continues with steps 1c) and 1d)
> >
> >
> > Is it possible to integrate such patch into kexec-tools RPM?
> >
> > We would like to integrate fence_kdump support into oVirt 3.5 which should be
> > released in May,
> > so before this to happen, we need kexec-tools RPM with this patch to be
> > available for RHEL 6 and
> > Fedora 19/20.
> >
> > Thanks
> >
> > Martin Perina
> > _______________________________________________
> > kexec mailing list
> > [e-mail]
> > https://lists.fedoraproject.org/mailman/listinfo/kexec
>
> diff -rupN a/dracut-module-setup.sh b/dracut-module-setup.sh
> --- a/dracut-module-setup.sh	2014-02-17 06:50:46.000000000 +0100
> +++ b/dracut-module-setup.sh	2014-03-12 08:38:03.000000000 +0100
> @@ -423,26 +423,38 @@ kdump_check_fence_kdump () {
>      local nodes
>      is_fence_kdump || return 1
> # get cluster nodes from cluster cib, get interface and ip address
> nodelist=`pcs cluster cib | xmllint --xpath "/cib/status/node_state/@uname" -`
> if is_cluster; then
>    # get cluster nodes from cluster cib, get interface and ip address
>    nodelist=`pcs cluster cib | xmllint --xpath "/cib/status/node_state/@uname" -`
>
> # nodelist is formed as 'uname="node1" uname="node2" ... uname="nodeX"'
> # we need to convert each to node1, node2 ... nodeX in each iteration
> for node in ${nodelist}; do
>    # convert $node from 'uname="nodeX"' to 'nodeX'
>    eval $node
>    nodename=$uname
>    # Skip its own node name
>    if [ "$nodename" = `hostname` ]; then
>        continue
>    fi
>    nodes="$nodes $nodename"
>    # nodelist is formed as 'uname="node1" uname="node2" ... uname="nodeX"'
>    # we need to convert each to node1, node2 ... nodeX in each iteration
>    for node in ${nodelist}; do
>        # convert $node from 'uname="nodeX"' to 'nodeX'
>        eval $node
>        nodename=$uname
>        # Skip its own node name
>        if [ "$nodename" = `hostname` ]; then
>            continue
>        fi
>        nodes="$nodes $nodename"
>    done
> else
>    # fence_kdump configured manually
>    read nodes < $FENCE_KDUMP_NODES

You may want to skip the case $FENCE_KDUMP_NODES contains the own hostname.
Something like this:

# Skip its own node name
            if [ "$nodename" = `hostname` ]; then
                continue
            fi

W/o skipping such case, kdump could be failing setup network.
>    kdump_install_net $nodename
> # install network for each node
> for node in ${nodes}; do
>    kdump_install_net $node
> done
> echo
> echo "$nodes" > ${initdir}/$FENCE_KDUMP_NODES
> mkdir -p ${initdir}$FENCE_KDUMP_NODES
> rmdir ${initdir}$FENCE_KDUMP_NODES
> echo "$nodes" > ${initdir}$FENCE_KDUMP_NODES

So $initdir has "/" suffix at the end?
> dracut_install $FENCE_KDUMP_SEND
>    dracut_install -o $FENCE_KDUMP_CONFIG
>
> }
> diff -rupN a/kdumpctl b/kdumpctl
> --- a/kdumpctl	2014-02-17 06:50:46.000000000 +0100
> +++ b/kdumpctl	2014-03-12 10:36:26.829734090 +0100
> @@ -133,7 +133,8 @@ function check_config()
>  }
>
> # check_fence_kdump <image timestamp>
> -# return 0 if fence_kdump is configured and kdump initrd needs to be rebuilt
> +# return 0 if fence_kdump is configured in cluster and kdump initrd needs
> +# to be rebuilt
>  function check_fence_kdump()
>  {
>    local image_time=$1
> @@ -141,6 +142,8 @@ function check_fence_kdump()
>
> is_fence_kdump || return 1
> is_cluster || return 1
> cib_time=`pcs cluster cib | xmllint --xpath 'string(/cib/@cib-last-written)' - | \
>     xargs -0 date +%s --date`
>
> @@ -199,6 +202,10 @@ function check_rebuild()
>    if [ -f $FENCE_KDUMP_CONFIG ]; then
>    	files="$files $FENCE_KDUMP_CONFIG"
>    fi
>
> if [ -f $FENCE_KDUMP_NODES ]; then
> files="$files $FENCE_KDUMP_NODES"
>
> fi
>
> check_exist "$files" && check_executable "$EXTRA_BINS"
> [ $? -ne 0 ] && return 1
>
> diff -rupN a/kdump-lib.sh b/kdump-lib.sh
> --- a/kdump-lib.sh	2014-02-17 06:50:46.000000000 +0100
> +++ b/kdump-lib.sh	2014-03-12 10:38:29.108892093 +0100
> @@ -5,7 +5,7 @@
>
> FENCE_KDUMP_CONFIG="/etc/sysconfig/fence_kdump"
>  FENCE_KDUMP_SEND="/usr/libexec/fence_kdump_send"
> -FENCE_KDUMP_NODES="/etc/fence_kdump_nodes"
> +FENCE_KDUMP_NODES="/etc/sysconfig/fence_kdump_nodes"
>
> is_ssh_dump_target()
>  {
> @@ -27,13 +27,25 @@ strip_comments()
>      echo $@ | sed -e 's/(.*)#.*/\1/'
>  }
>
> +# Check if cluster environment is present
> +is_cluster()
> +{
> # no pcs executable installed
> type -P pcs > /dev/null || return 1
>
> +}
>
> # Check if fence kdump is configured in cluster
>  is_fence_kdump()
>  {
> # no pcs or fence_kdump_send executables installed?
> type -P pcs > /dev/null || return 1
> # no fence_kdump_send executables installed?
> [ -x $FENCE_KDUMP_SEND ] || return 1
> # fence kdump not configured?
> (pcs cluster cib | grep -q 'type="fence_kdump"') &> /dev/null || return 1
> if is_cluster; then
>    # fence kdump configured in cluster?
>    (pcs cluster cib | grep -q 'type="fence_kdump"') &> /dev/null || return 1
> else
>    # fence kdump configured manually?
>    [ -f $FENCE_KDUMP_CONFIG ] || return 1

Does $FENCE_KDUMP_CONFIG has to exist in oVirt case?
>    [ -f $FENCE_KDUMP_NODES ] || return 1
> fi
>
> }

I have a question in general, when both pcs and
/etc/sysconfig/fence_kdump_nodes are present in the system, which one is
taking precedence? pcs configured nodes or nodes listed in
fence_kdump_nodes?

Thanks
WANG Chao
> kexec mailing list
> [e-mail]
> https://lists.fedoraproject.org/mailman/listinfo/kexec

----- Original Message -----
> From: "WANG Chao" 
> To: "Martin Perina" 
> Cc: [e-mail]
> Sent: Thursday, March 13, 2014 1:32:27 PM
> Subject: Re: Adding support for manually configured fence_kdump
>
> On 03/13/14 at 03:36am, Martin Perina wrote:
> > Patch in attachment was filtered so I resent it inline:
> >
> > ----- Original Message -----
> > > From: "Martin Perina" 
> > > To: [e-mail]
> > > Sent: Thursday, March 13, 2014 7:47:08 AM
> > > Subject: Adding support for manually configured fence_kdump
> > >
> > > Hi,
> > >
> > > we want to integrate support for fence_kdump into oVirt project. Marek
> > > Grac
> > > pointed me
> > > to kexec-tools-2.0.4-23 which contains automatic configuration of
> > > fence_kdump
> > > from Pacemaker config.
> > > Unfortunately for us, this support is tightly bound to Pacemaker, so I
> > > created attached patch,
> > > which adds support for manual configuration of fence_kdump:
> > >
> > > Current status without this patch (kexec-tools-2.0.4-23)
> > > when running 'kdumpctl restart' kdump detects if fence_kdump_send
> > >
> > > command is installed
> > >     and if so, it tries to:
> > >   a) Create /etc/sysconfig/fence_kdump file with command line
> > >   parameters for fence_kdump_send.
> > >      These parameters are extracted from cluster configuration
> > >      using
> > >      'pcs cluster' command
> > >   b) Create /etc/fence_kdump_nodes file with list of hosts to send
> > >   fence_kdump notification to.
> > >      These hosts are also extracted from cluster configuration
> > >      using
> > >      'pcs cluster' command
> > >   c) If these two files are successfully created, kdump tries to
> > >   configure network to be able
> > >      to reach all hosts defined in /etc/fence_kdump_nodes
> > >   d) Install these two config files into kdump's kernel initramfs
> > >
> > >
> > > Behavior with this patch
> > > when running 'kdumpctl restart' kdump detect if fence_kdump_send
> > >
> > > binary is installed
> > >     and if so, it tries to:
> > >   a) It tries to detect is cluster is configured ('pcs' command is
> > >   available). If so, it creates
> > >      needed config files as described above in steps 1a) and 1b)
> > >   b) If cluster is not configured, it tries to detect if
> > >   /etc/sysconfig/fence_kdump and
> > >      /etc/sysconfig/fence_kdump_nodes exist. If so, fence_kdump is
> > >      marked as successfully
> > >      configured and kdump continues with steps 1c) and 1d)
> > >
> > >
> > > Is it possible to integrate such patch into kexec-tools RPM?
> > >
> > > We would like to integrate fence_kdump support into oVirt 3.5 which
> > > should be
> > > released in May,
> > > so before this to happen, we need kexec-tools RPM with this patch to be
> > > available for RHEL 6 and
> > > Fedora 19/20.
> > >
> > > Thanks
> > >
> > > Martin Perina
> > > _______________________________________________
> > > kexec mailing list
> > > [e-mail]
> > > https://lists.fedoraproject.org/mailman/listinfo/kexec
> >
> > diff -rupN a/dracut-module-setup.sh b/dracut-module-setup.sh
> > --- a/dracut-module-setup.sh	2014-02-17 06:50:46.000000000 +0100
> > +++ b/dracut-module-setup.sh	2014-03-12 08:38:03.000000000 +0100
> > @@ -423,26 +423,38 @@ kdump_check_fence_kdump () {
> >      local nodes
> >      is_fence_kdump || return 1
> > # get cluster nodes from cluster cib, get interface and ip address
> > nodelist=`pcs cluster cib | xmllint --xpath
> >
> > "/cib/status/node_state/@uname" -`
> > if is_cluster; then
> >    # get cluster nodes from cluster cib, get interface and ip address
> >    nodelist=`pcs cluster cib | xmllint --xpath
> >
> > "/cib/status/node_state/@uname" -`
> > # nodelist is formed as 'uname="node1" uname="node2" ...
> >
> > uname="nodeX"'
> > # we need to convert each to node1, node2 ... nodeX in each iteration
> > for node in ${nodelist}; do
> >    # convert $node from 'uname="nodeX"' to 'nodeX'
> >    eval $node
> >    nodename=$uname
> >    # Skip its own node name
> >    if [ "$nodename" = `hostname` ]; then
> >        continue
> >    fi
> >    nodes="$nodes $nodename"
> >    # nodelist is formed as 'uname="node1" uname="node2" ...
> >
> > uname="nodeX"'
> >    # we need to convert each to node1, node2 ... nodeX in each
> >
> > iteration
> >    for node in ${nodelist}; do
> >        # convert $node from 'uname="nodeX"' to 'nodeX'
> >        eval $node
> >        nodename=$uname
> >        # Skip its own node name
> >        if [ "$nodename" = `hostname` ]; then
> >            continue
> >        fi
> >        nodes="$nodes $nodename"
> >    done
> > else
> >    # fence_kdump configured manually
> >    read nodes < $FENCE_KDUMP_NODES
>
> You may want to skip the case $FENCE_KDUMP_NODES contains the own hostname.
> Something like this:
>         # Skip its own node name
>         if [ "$nodename" = `hostname` ]; then
>             continue
>         fi
>
> W/o skipping such case, kdump could be failing setup network.

OK, I will fix this.
> >    kdump_install_net $nodename
> > # install network for each node
> > for node in ${nodes}; do
> >    kdump_install_net $node
> > done
> > echo
> > echo "$nodes" > ${initdir}/$FENCE_KDUMP_NODES
> > mkdir -p ${initdir}$FENCE_KDUMP_NODES
> > rmdir ${initdir}$FENCE_KDUMP_NODES
> > echo "$nodes" > ${initdir}$FENCE_KDUMP_NODES
>
> So $initdir has "/" suffix at the end?

No, but config files starts with "/", so in current code slashes
are doubled.
> > dracut_install $FENCE_KDUMP_SEND
> >    dracut_install -o $FENCE_KDUMP_CONFIG
> >
> > }
> > diff -rupN a/kdumpctl b/kdumpctl
> > --- a/kdumpctl	2014-02-17 06:50:46.000000000 +0100
> > +++ b/kdumpctl	2014-03-12 10:36:26.829734090 +0100
> > @@ -133,7 +133,8 @@ function check_config()
> >  }
> >
> > # check_fence_kdump <image timestamp>
> > -# return 0 if fence_kdump is configured and kdump initrd needs to be
> > rebuilt
> > +# return 0 if fence_kdump is configured in cluster and kdump initrd needs
> > +# to be rebuilt
> >  function check_fence_kdump()
> >  {
> >    local image_time=$1
> > @@ -141,6 +142,8 @@ function check_fence_kdump()
> >
> > is_fence_kdump || return 1
> > is_cluster || return 1
> > cib_time=`pcs cluster cib | xmllint --xpath
> >  'string(/cib/@cib-last-written)' - | \
> >     xargs -0 date +%s --date`
> >
> > @@ -199,6 +202,10 @@ function check_rebuild()
> >    if [ -f $FENCE_KDUMP_CONFIG ]; then
> >    	files="$files $FENCE_KDUMP_CONFIG"
> >    fi
> >
> > if [ -f $FENCE_KDUMP_NODES ]; then
> > files="$files $FENCE_KDUMP_NODES"
> >
> > fi
> >
> > check_exist "$files" && check_executable "$EXTRA_BINS"
> > [ $? -ne 0 ] && return 1
> >
> > diff -rupN a/kdump-lib.sh b/kdump-lib.sh
> > --- a/kdump-lib.sh	2014-02-17 06:50:46.000000000 +0100
> > +++ b/kdump-lib.sh	2014-03-12 10:38:29.108892093 +0100
> > @@ -5,7 +5,7 @@
> >
> > FENCE_KDUMP_CONFIG="/etc/sysconfig/fence_kdump"
> >  FENCE_KDUMP_SEND="/usr/libexec/fence_kdump_send"
> > -FENCE_KDUMP_NODES="/etc/fence_kdump_nodes"
> > +FENCE_KDUMP_NODES="/etc/sysconfig/fence_kdump_nodes"
> >
> > is_ssh_dump_target()
> >  {
> > @@ -27,13 +27,25 @@ strip_comments()
> >      echo $@ | sed -e 's/(.*)#.*/\1/'
> >  }
> >
> > +# Check if cluster environment is present
> > +is_cluster()
> > +{
> > # no pcs executable installed
> > type -P pcs > /dev/null || return 1
> >
> > +}
> >
> > # Check if fence kdump is configured in cluster
> >  is_fence_kdump()
> >  {
> > # no pcs or fence_kdump_send executables installed?
> > type -P pcs > /dev/null || return 1
> > # no fence_kdump_send executables installed?
> > [ -x $FENCE_KDUMP_SEND ] || return 1
> > # fence kdump not configured?
> > (pcs cluster cib | grep -q 'type="fence_kdump"') &> /dev/null ||
> >
> > return 1
> > if is_cluster; then
> >    # fence kdump configured in cluster?
> >    (pcs cluster cib | grep -q 'type="fence_kdump"') &> /dev/null ||
> >
> > return 1
> > else
> >    # fence kdump configured manually?
> >    [ -f $FENCE_KDUMP_CONFIG ] || return 1
>
> Does $FENCE_KDUMP_CONFIG has to exist in oVirt case?

I will remove this as we agreed on previous emails, that this config file should be optional.
But in oVirt case we will change defaults, so for oVirt this file will always exist.
> >    [ -f $FENCE_KDUMP_NODES ] || return 1
> > fi
> >
> > }
>
> I have a question in general, when both pcs and
> /etc/sysconfig/fence_kdump_nodes are present in the system, which one is
> taking precedence? pcs configured nodes or nodes listed in
> fence_kdump_nodes?

When cluster is present, nodes received from cluster configuration via pcs
will always be used.
> Thanks
> WANG Chao
> > kexec mailing list
> > [e-mail]
> > https://lists.fedoraproject.org/mailman/listinfo/kexec

[Vivek Goyal]

On Thu, Mar 13, 2014 at 02:47:08AM -0400, Martin Perina wrote:
> Hi,
>
> we want to integrate support for fence_kdump into oVirt project. Marek Grac pointed me
> to kexec-tools-2.0.4-23 which contains automatic configuration of fence_kdump from Pacemaker config.

Can you please explain how oVirt cluster is different from regular cluster. 
And why pacemaker is not usable.
> Unfortunately for us, this support is tightly bound to Pacemaker, so I created attached patch,

Why pacemaker is not usable with Ovirt.

[..]
> Behavior with this patch
>
> when running 'kdumpctl restart' kdump detect if fence_kdump_send binary is installed
> and if so, it tries to:
>
> a) It tries to detect is cluster is configured ('pcs' command is available). If so, it creates
>      needed config files as described above in steps 1a) and 1b)
>   b) If cluster is not configured, it tries to detect if /etc/sysconfig/fence_kdump and
>      /etc/sysconfig/fence_kdump_nodes exist. If so, fence_kdump is marked as successfully
>      configured and kdump continues with steps 1c) and 1d)

Who will generate /etc/sysconfig/fence_kudmp and how?

Thanks
Vivek

----- Original Message -----
> From: "Vivek Goyal" 
> To: "Martin Perina" 
> Cc: [e-mail]
> Sent: Thursday, March 13, 2014 9:37:33 PM
> Subject: Re: Adding support for manually configured fence_kdump
>
> On Thu, Mar 13, 2014 at 02:47:08AM -0400, Martin Perina wrote:
> > Hi,
> >
> > we want to integrate support for fence_kdump into oVirt project. Marek Grac
> > pointed me
> > to kexec-tools-2.0.4-23 which contains automatic configuration of
> > fence_kdump from Pacemaker config.
>
> Can you please explain how oVirt cluster is different from regular cluster.
> And why pacemaker is not usable.

oVirt cluster usually contain a lot of nodes (hosts in oVirt terminology),
which can dynamically added/removed to/from cluster. Each host is not
aware of other hosts in cluster, they are all managed by machine on which
oVirt engine is running. Each host is running VDSM which is common gateway
between engine and libvirtd, which manages virtual machines executed on host.
Each host is independent when running its own virtual machines, but engine
manages how and when are virtual machines started/stopped/migrated among
hosts. Each engine can manage multiple clusters.

More info can be found at http://www.ovirt.org/Architecture

So cluster in oVirt term is very different from cluster in Pacemaker term,
and we don't intend to install Pacemaker on oVirt hosts.
> > Unfortunately for us, this support is tightly bound to Pacemaker, so I
> > created attached patch,
>
> Why pacemaker is not usable with Ovirt.
>
> [..]
> > Behavior with this patch
> > when running 'kdumpctl restart' kdump detect if fence_kdump_send
> >
> > binary is installed
> >     and if so, it tries to:
> >   a) It tries to detect is cluster is configured ('pcs' command is
> >   available). If so, it creates
> >      needed config files as described above in steps 1a) and 1b)
> >   b) If cluster is not configured, it tries to detect if
> >   /etc/sysconfig/fence_kdump and
> >      /etc/sysconfig/fence_kdump_nodes exist. If so, fence_kdump is
> >      marked as successfully
> >      configured and kdump continues with steps 1c) and 1d)
>
> Who will generate /etc/sysconfig/fence_kudmp and how?

We plan to extend VDSM to be able to:

- detect if hosts has enabled/configured kdump support, 
  - to save fence_kdump configuration and hosts to which notification will
    be sent
  - to call kdumpctl restart after fence_kdump configuration change

And support for fence_kdump will be included into existing oVirt fencing
mechanism.
> Thanks
> Vivek

[Vivek Goyal]

On Thu, Mar 13, 2014 at 05:01:27PM -0400, Martin Perina wrote:

[..]
> Each host is independent when running its own virtual machines, but engine
> manages how and when are virtual machines started/stopped/migrated among
> hosts.

I am assuming there is one engine across whole cluster. So who all needs
to get notification that a host in cluster is saving dump. Just the engine
or other hosts too?

Thanks
Vivek

[Vivek Goyal]

On Thu, Mar 13, 2014 at 02:47:08AM -0400, Martin Perina wrote:
> Hi,
>
> we want to integrate support for fence_kdump into oVirt project. Marek Grac pointed me
> to kexec-tools-2.0.4-23 which contains automatic configuration of fence_kdump from Pacemaker config.
> Unfortunately for us, this support is tightly bound to Pacemaker, so I created attached patch,
> which adds support for manual configuration of fence_kdump:
>
> Current status without this patch (kexec-tools-2.0.4-23)
>
> when running 'kdumpctl restart' kdump detects if fence_kdump_send command is installed
> and if so, it tries to:
>
> a) Create /etc/sysconfig/fence_kdump file with command line parameters for fence_kdump_send.
>      These parameters are extracted from cluster configuration using 'pcs cluster' command

I was looking at the code. kdump does not create /etc/sysconfig/fence_kdump.
It expects it to be present there so that fence_kdump_send can read its
configuration.

If that's the case, then vdsm can just modify this configuration as needed
and kdump scripts should include this file in initramfs.
>      b) Create /etc/fence_kdump_nodes file with list of hosts to send fence_kdump notification to.
>         These hosts are also extracted from cluster configuration using 'pcs cluster' command

Yep, list of notes is generated dynamically using pcs command . Why don't
you too provide a command to generate the list and we can check if command
is present on the system, then invoke it and generate list of nodes. Just
like pcs.

Thanks
Vivek

----- Original Message -----
> From: "Vivek Goyal" 
> To: "Martin Perina" 
> Cc: [e-mail]
> Sent: Thursday, March 13, 2014 10:25:30 PM
> Subject: Re: Adding support for manually configured fence_kdump
>
> On Thu, Mar 13, 2014 at 05:01:27PM -0400, Martin Perina wrote:
>
> [..]
> > Each host is independent when running its own virtual machines, but engine
> > manages how and when are virtual machines started/stopped/migrated among
> > hosts.
>
> I am assuming there is one engine across whole cluster. So who all needs
> to get notification that a host in cluster is saving dump. Just the engine
> or other hosts too?
>
> Thanks
> Vivek

We currently have generic mechanism for hard fencing which work this way:

1) If host1 become Non Responsive, engine will choose host2 (fencing proxy)
   from the same cluster on which configure fencing agent will be executed
2) Then engine communicates with VDSM on host2
     a) Get host1 status using fencing device
     b) If status if DOWN, goto e)
     c) If status is UP, shutdown host1
     d) Check host1 status until is DOWN
     e) Startup host1
     f) Check host1 status until it's UP

The process is more complicated, but that's the general idea.

With fence_kdump we have several options:
  1) Use same mechanism as for hard fencing
       - PROBLEM: we would need to update each host with list of all other
         hosts in cluster, because any of them could become fencing proxy
       - PROBLEM: sending messages to huge number of hosts in large clusters
                  may not be efficient
       - PROBLEM: updating all hosts and regenerating its kdump initramfs
                  for each add/remove host to/from cluster may not be
                  efficient

2) Modify fence_kdump to be able to send notifications to level2
     broadcast address
       - PROBLEM: we would need a patch for fence_kdump which may or may
                  not be accepted
       - PROBLEM: not sure if all of our customers could use level 2
                  broadcast among cluster's hosts

3) Execute fence kdump from engine host
       - PROBLEM: we would need the engine host to accessible for UDP
                  messaging on port 7410
       - PROBLEM: we would need to write our own fence_kdump message
                  listener in case when multiple fence_kdump notifications
                  would be needed to resolve at the same time

We are currently discussing which option would be best, but for all options
we need possibility to configure fence_kdump without Pacemake cluster config.

Martin