how engine get files from node？

Hi all,
   I have submitted a patch about watchdog device 
http://gerrit.ovirt.org/#/c/7535/
   If we set 'dump' action for watchdog, qemu will generate a dump file.
   And I have get some feedback, how engine get these files?

   scp? or Will vdsm support new API to list and get these files?
   IMO, The new API should  list and get not only dump files,  but also 
other kinds of files.


-- 
Sheldon Feng<shaohef(a)linux.vnet.ibm.com>
IBM Linux Technology Center

On 09/21/2012 05:42 PM, Sheldon wrote:
>  Hi all,
>    I have submitted a patch about watchdog device 
>  http://gerrit.ovirt.org/#/c/7535/
>    If we set 'dump' action for watchdog, qemu will generate a dump file 
>  when watchdog event.
>    And I have get some feedback, how engine get these files?
>
>    scp? or Will vdsm support new API to list and get these files?
>    IMO, The new API should  list and get not only dump files,  but also 
>  other kinds of files.
>
>
>  -- 
>  Sheldon Feng<shaohef(a)linux.vnet.ibm.com>
>  IBM Linux Technology Center 

-- 
Sheldon Feng(冯少合)<shaohef(a)linux.vnet.ibm.com>
IBM Linux Technology Center

[Ryan Harper]

* Sheldon <shaohef(a)linux.vnet.ibm.com&gt; [2012-09-21 04:42]:
>  Hi all,
>    I have submitted a patch about watchdog device
>  http://gerrit.ovirt.org/#/c/7535/
>    If we set 'dump' action for watchdog, qemu will generate a dump file.
>    And I have get some feedback, how engine get these files?
>  
>    scp? or Will vdsm support new API to list and get these files?
>    IMO, The new API should  list and get not only dump files,  but
>  also other kinds of files. 
That's a good question.  We know that vdsm captured core files today.  I
don't think there is an API mechanism to retrieve the core files
remotely yet.

If we did propose such a call, what sets of files would we want to
include?  Would this be a query for fetching debug data for a single VM,
for the host (which would/could include all cores and dumps)?

>  
>  
>  -- 
>  Sheldon Feng&lt;shaohef(a)linux.vnet.ibm.com&gt;
>  IBM Linux Technology Center
>   
-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
ryanh(a)us.ibm.com

于 2012-9-21 20:01, Ryan Harper:
>  * Sheldon <shaohef(a)linux.vnet.ibm.com&gt; [2012-09-21 04:42]:
> > Hi all,
> >    I have submitted a patch about watchdog device
> > http://gerrit.ovirt.org/#/c/7535/
> >    If we set 'dump' action for watchdog, qemu will generate a dump file.
> >    And I have get some feedback, how engine get these files?
> >
> >    scp? or Will vdsm support new API to list and get these files?
> >    IMO, The new API should  list and get not only dump files,  but
> > also other kinds of files.
>  That's a good question.  We know that vdsm captured core files today.  I
>  don't think there is an API mechanism to retrieve the core files
>  remotely yet.
>
>  If we did propose such a call, what sets of files would we want to
>  include?  Would this be a query for fetching debug data for a single VM,
>  for the host (which would/could include all cores and dumps)? 
At least, the log files of VDSM(/var/log/vdsm/), the log files of 
libvirt VMs( /var/log/libvirt/qemu/), VM panic cores and Qemu crash core 
dumps can be included.  Among them, the log files of VDSM are global 
ones and the others should be queried based on a specific VM.

> >
> > -- 
> > Sheldon Feng&lt;shaohef(a)linux.vnet.ibm.com&gt;
> > IBM Linux Technology Center
> > 

-- 
---
舒明 Shu Ming
Open Virtualization Engineerning; CSTL, IBM Corp.
Tel: [phone]  Tieline: 9051626 E-mail: shuming(a)cn.ibm.com or
shuming(a)linux.vnet.ibm.com
Address: 3/F Ring Building, ZhongGuanCun Software Park, Haidian District, Beijing 100193,
PRC

----- Original Message -----
>  From: "Sheldon" <shaohef(a)linux.vnet.ibm.com&gt;
>  To: vdsm-devel(a)lists.fedorahosted.org
>  Sent: Friday, September 21, 2012 11:42:05 AM
>  Subject: [vdsm]  how engine get files from node？
>  
>  
>  Hi all,
>  I have submitted a patch about watchdog device
>  http://gerrit.ovirt.org/#/c/7535/
>  If we set 'dump' action for watchdog, qemu will generate a dump file.
>  And I have get some feedback, how engine get these files? 
When adding the host (if installation is enabled) the engine does some ssh/scp to the new
host, but this can be turned off. Also, the guideline is to have a single interface with
the host, and after installation the engine does not try to communicate with vdsm in any
other way than the xmlrpc interface.

>  
>  scp? or Will vdsm support new API to list and get these files?
>  IMO, The new API should list and get not only dump files, but also
>  other kinds of files.
>  
>  
>  --
>  Sheldon Feng <shaohef(a)linux.vnet.ibm.com&gt; IBM Linux Technology Center
>  _______________________________________________
>  vdsm-devel mailing list
>  vdsm-devel(a)lists.fedorahosted.org
>  https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
>

On 09/21/2012 08:14 PM, Laszlo Hornyak wrote:
>  ----- Original Message -----
> > From: "Sheldon" <shaohef(a)linux.vnet.ibm.com&gt;
> > To: vdsm-devel(a)lists.fedorahosted.org
> > Sent: Friday, September 21, 2012 11:42:05 AM
> > Subject: [vdsm]  how engine get files from node？
> >
> >
> > Hi all,
> > I have submitted a patch about watchdog device
> > http://gerrit.ovirt.org/#/c/7535/
> > If we set 'dump' action for watchdog, qemu will generate a dump file.
> > And I have get some feedback, how engine get these files?
>  When adding the host (if installation is enabled) the engine does some ssh/scp to the new
> host, but this can be turned off. Also, the guideline is to have a single interface with
> the host, and after installation the engine does not try to communicate with vdsm in any
> other way than the xmlrpc interface. 
do you means "ssh/scp" may not be available on host, So a xmlrpc interface
should be added to get the files on host?

> > scp? or Will vdsm support new API to list and get these files?
> > IMO, The new API should list and get not only dump files, but also
> > other kinds of files.
> >
> >
> > --
> > Sheldon Feng <shaohef(a)linux.vnet.ibm.com&gt; IBM Linux Technology Center
> > _______________________________________________
> > vdsm-devel mailing list
> > vdsm-devel(a)lists.fedorahosted.org
> > https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
> > 

-- 
Sheldon Feng(冯少合)&lt;shaohef(a)linux.vnet.ibm.com&gt;
IBM Linux Technology Center

[Ryan Harper]

* Sheldon <shaohef(a)linux.vnet.ibm.com&gt; [2012-09-24 06:50]:
>  On 09/21/2012 08:14 PM, Laszlo Hornyak wrote:
>  >
>  >----- Original Message -----
>  >>From: "Sheldon" <shaohef(a)linux.vnet.ibm.com&gt;
>  >>To: vdsm-devel(a)lists.fedorahosted.org
>  >>Sent: Friday, September 21, 2012 11:42:05 AM
>  >>Subject: [vdsm]  how engine get files from node???
>  >>
>  >>
>  >>Hi all,
>  >>I have submitted a patch about watchdog device
>  >>http://gerrit.ovirt.org/#/c/7535/
>  >>If we set 'dump' action for watchdog, qemu will generate a dump file.
>  >>And I have get some feedback, how engine get these files?
>  >When adding the host (if installation is enabled) the engine does some ssh/scp to the
> new host, but this can be turned off. Also, the guideline is to have a single interface
> with the host, and after installation the engine does not try to communicate with vdsm in
> any other way than the xmlrpc interface.
>  
>  do you means "ssh/scp" may not be available on host, So a xmlrpc
>  interface should be added to get the files on host? 
ssh/scp is available, but requires user authentication.  We already have
one entry point into the node, the VDSM API, which is currently running
over XMLRPC, so, instead of attempting to coordinate additional users
and authenication, provide a way to programatically get at the data via
XMLRPC.

>  
>  >>scp? or Will vdsm support new API to list and get these files?
>  >>IMO, The new API should list and get not only dump files, but also
>  >>other kinds of files.
>  >>
>  >>
>  >>--
>  >>Sheldon Feng <shaohef(a)linux.vnet.ibm.com&gt; IBM Linux Technology Center
>  >>_______________________________________________
>  >>vdsm-devel mailing list
>  &gt;&gt;vdsm-devel(a)lists.fedorahosted.org
>  >>https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
>  >>
>  
>  
>  -- 
>  Sheldon Feng(?????????)&lt;shaohef(a)linux.vnet.ibm.com&gt;
>  IBM Linux Technology Center 
-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
ryanh(a)us.ibm.com

On 09/24/2012 02:46 PM, Ryan Harper wrote:
>  * Sheldon <shaohef(a)linux.vnet.ibm.com&gt; [2012-09-24 06:50]:
> > On 09/21/2012 08:14 PM, Laszlo Hornyak wrote:
> >>
> >> ----- Original Message -----
> >>> From: "Sheldon" <shaohef(a)linux.vnet.ibm.com&gt;
> >>> To: vdsm-devel(a)lists.fedorahosted.org
> >>> Sent: Friday, September 21, 2012 11:42:05 AM
> >>> Subject: [vdsm]  how engine get files from node???
> >>>
> >>>
> >>> Hi all,
> >>> I have submitted a patch about watchdog device
> >>> http://gerrit.ovirt.org/#/c/7535/
> >>> If we set 'dump' action for watchdog, qemu will generate a dump
> file.
> >>> And I have get some feedback, how engine get these files?
> >> When adding the host (if installation is enabled) the engine does some ssh/scp to
> the new host, but this can be turned off. Also, the guideline is to have a single
> interface with the host, and after installation the engine does not try to communicate
> with vdsm in any other way than the xmlrpc interface.
> >
> > do you means "ssh/scp" may not be available on host, So a xmlrpc
> > interface should be added to get the files on host?
>
>  ssh/scp is available, but requires user authentication.  We already have
>  one entry point into the node, the VDSM API, which is currently running
>  over XMLRPC, so, instead of attempting to coordinate additional users
>  and authenication, provide a way to programatically get at the data via
>  XMLRPC. 
is this needed by the engine interactively, or just for log colleciton?
logs/cores/dumps are collected today by the log collector utility, based 
on ssh/scp leveraging sos.
but this isn't "engine getting logs", rather a stand alone utility for 
collecting them.
is the file needed by engine for some flow, or just log collection for 
user benefit?

> >
> >>> scp? or Will vdsm support new API to list and get these files?
> >>> IMO, The new API should list and get not only dump files, but also
> >>> other kinds of files.
> >>>
> >>>
> >>> --
> >>> Sheldon Feng <shaohef(a)linux.vnet.ibm.com&gt; IBM Linux Technology Center
> >>> _______________________________________________
> >>> vdsm-devel mailing list
> >>> vdsm-devel(a)lists.fedorahosted.org
> >>> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
> >>>
> >
> >
> > --
> > Sheldon Feng(?????????)&lt;shaohef(a)linux.vnet.ibm.com&gt;
> > IBM Linux Technology Center
>

[Ryan Harper]

* Itamar Heim <iheim(a)redhat.com&gt; [2012-09-27 02:21]:
>  On 09/24/2012 02:46 PM, Ryan Harper wrote:
>  >* Sheldon <shaohef(a)linux.vnet.ibm.com&gt; [2012-09-24 06:50]:
>  >>On 09/21/2012 08:14 PM, Laszlo Hornyak wrote:
>  >>>
>  >>>----- Original Message -----
>  >>>>From: "Sheldon" <shaohef(a)linux.vnet.ibm.com&gt;
>  >>>>To: vdsm-devel(a)lists.fedorahosted.org
>  >>>>Sent: Friday, September 21, 2012 11:42:05 AM
>  >>>>Subject: [vdsm]  how engine get files from node???
>  >>>>
>  >>>>
>  >>>>Hi all,
>  >>>>I have submitted a patch about watchdog device
>  >>>>http://gerrit.ovirt.org/#/c/7535/
>  >>>>If we set 'dump' action for watchdog, qemu will generate a dump
> file.
>  >>>>And I have get some feedback, how engine get these files?
>  >>>When adding the host (if installation is enabled) the engine does some
> ssh/scp to the new host, but this can be turned off. Also, the guideline is to have a
> single interface with the host, and after installation the engine does not try to
> communicate with vdsm in any other way than the xmlrpc interface.
>  >>
>  >>do you means "ssh/scp" may not be available on host, So a xmlrpc
>  >>interface should be added to get the files on host?
>  >
>  >ssh/scp is available, but requires user authentication.  We already have
>  >one entry point into the node, the VDSM API, which is currently running
>  >over XMLRPC, so, instead of attempting to coordinate additional users
>  >and authenication, provide a way to programatically get at the data via
>  >XMLRPC.
>  
>  is this needed by the engine interactively, or just for log colleciton?
>  logs/cores/dumps are collected today by the log collector utility,
>  based on ssh/scp leveraging sos.
>  but this isn't "engine getting logs", rather a stand alone utility
>  for collecting them.
>  is the file needed by engine for some flow, or just log collection
>  for user benefit? 
I can imagine both scenarios.  In general, if the node has
cores/logs/dumps for a particular VM, it would be nice to have those
available for debugging, reporting issues, etc.  We always have
scenarios where we wish we had a better picture of what was happening on
the system at the time of failure.  For vdsm, we do since we have host
logs/stats as well as anything we captured from the guest (cores, etc).  

I'd like a way for the mgmt server to specify either, all of the host
logs, or a specific guest log, and fetch all of that data from the
end-point.  Certainly something with ssh/scp can be worked out, but it
would be nice to have this available programatically.

Think of the engine flow around a VM that's tagged as HA.  If engine
detects a failure, it will create the VM again on a different host; then
engine could pull debugging data and log files from the host to store
that for the admin to push to her support connection.

And instead of having the admin know to run this stand alone utility, it
could be tied into an existing HA recover use-case.

Another case some of our mgmt products use is to do collect error
reports and statistics immediately after a failure has occured and
stores them on the management server, raising a notice to the admin for
submission to support.

These use-cases are easier to drive with an API call rather than the
standalone utility.

Now, on the contrary; if vdsm itself is having issues, then it seems
that such a separate utility is still needed. 

>  
>  >
>  >>
>  >>>>scp? or Will vdsm support new API to list and get these files?
>  >>>>IMO, The new API should list and get not only dump files, but also
>  >>>>other kinds of files.
>  >>>>
>  >>>>
>  >>>>--
>  >>>>Sheldon Feng <shaohef(a)linux.vnet.ibm.com&gt; IBM Linux Technology
> Center
>  >>>>_______________________________________________
>  >>>>vdsm-devel mailing list
>  &gt;&gt;&gt;&gt;vdsm-devel(a)lists.fedorahosted.org
>  >>>>https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
>  >>>>
>  >>
>  >>
>  >>--
>  >>Sheldon Feng(?????????)&lt;shaohef(a)linux.vnet.ibm.com&gt;
>  >>IBM Linux Technology Center
>  >
>   
-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
ryanh(a)us.ibm.com

----- Original Message -----
>  From: "Ryan Harper" <ryanh(a)us.ibm.com&gt;
>  To: "Itamar Heim" <iheim(a)redhat.com&gt;
>  Cc: vdsm-devel(a)lists.fedorahosted.org, "Ming M Shu" <shuming(a)cn.ibm.com&gt;,
> "Zheng Sheng ZS Zhou" <zhshzhou(a)cn.ibm.com&gt;
>  Sent: Thursday, September 27, 2012 4:07:56 PM
>  Subject: Re: [vdsm] how engine get files from node???
>  
>  * Itamar Heim <iheim(a)redhat.com&gt; [2012-09-27 02:21]:
>  > On 09/24/2012 02:46 PM, Ryan Harper wrote:
>  > >* Sheldon <shaohef(a)linux.vnet.ibm.com&gt; [2012-09-24 06:50]:
>  > >>On 09/21/2012 08:14 PM, Laszlo Hornyak wrote:
>  > >>>
>  > >>>----- Original Message -----
>  > >>>>From: "Sheldon" <shaohef(a)linux.vnet.ibm.com&gt;
>  > >>>>To: vdsm-devel(a)lists.fedorahosted.org
>  > >>>>Sent: Friday, September 21, 2012 11:42:05 AM
>  > >>>>Subject: [vdsm]  how engine get files from node???
>  > >>>>
>  > >>>>
>  > >>>>Hi all,
>  > >>>>I have submitted a patch about watchdog device
>  > >>>>http://gerrit.ovirt.org/#/c/7535/
>  > >>>>If we set 'dump' action for watchdog, qemu will generate a
> dump
>  > >>>>file.
>  > >>>>And I have get some feedback, how engine get these files?
>  > >>>When adding the host (if installation is enabled) the engine
>  > >>>does some ssh/scp to the new host, but this can be turned off.
>  > >>>Also, the guideline is to have a single interface with the
>  > >>>host, and after installation the engine does not try to
>  > >>>communicate with vdsm in any other way than the xmlrpc
>  > >>>interface.
>  > >>
>  > >>do you means "ssh/scp" may not be available on host, So a xmlrpc
>  > >>interface should be added to get the files on host?
>  > >
>  > >ssh/scp is available, but requires user authentication.  We
>  > >already have
>  > >one entry point into the node, the VDSM API, which is currently
>  > >running
>  > >over XMLRPC, so, instead of attempting to coordinate additional
>  > >users
>  > >and authenication, provide a way to programatically get at the
>  > >data via
>  > >XMLRPC.
>  > 
>  > is this needed by the engine interactively, or just for log
>  > colleciton?
>  > logs/cores/dumps are collected today by the log collector utility,
>  > based on ssh/scp leveraging sos.
>  > but this isn't "engine getting logs", rather a stand alone utility
>  > for collecting them.
>  > is the file needed by engine for some flow, or just log collection
>  > for user benefit?
>  
>  I can imagine both scenarios.  In general, if the node has
>  cores/logs/dumps for a particular VM, it would be nice to have those
>  available for debugging, reporting issues, etc.  We always have
>  scenarios where we wish we had a better picture of what was happening
>  on
>  the system at the time of failure.  For vdsm, we do since we have
>  host
>  logs/stats as well as anything we captured from the guest (cores,
>  etc).
>  
>  I'd like a way for the mgmt server to specify either, all of the host
>  logs, or a specific guest log, and fetch all of that data from the
>  end-point.  Certainly something with ssh/scp can be worked out, but
>  it
>  would be nice to have this available programatically.
>  
>  Think of the engine flow around a VM that's tagged as HA.  If engine
>  detects a failure, it will create the VM again on a different host;
>  then
>  engine could pull debugging data and log files from the host to store
>  that for the admin to push to her support connection.
>  
>  And instead of having the admin know to run this stand alone utility,
>  it
>  could be tied into an existing HA recover use-case.
>  
>  Another case some of our mgmt products use is to do collect error
>  reports and statistics immediately after a failure has occured and
>  stores them on the management server, raising a notice to the admin
>  for
>  submission to support.
>  
>  These use-cases are easier to drive with an API call rather than the
>  standalone utility.
>  
>  Now, on the contrary; if vdsm itself is having issues, then it seems
>  that such a separate utility is still needed.
>   
As I mentioned in another related thread, ovirt-log-collector is currently
handling this functionality.
HA mechanisms tend to handle the relevant entity's survival (VM in this case),
and leave other activities to offline or out-of-band processes.
So for the above flows I'd think on integrating ovirt-log-collector with
the engine UI in order to improve the admin UX. In terms of functionality
I'd be cautious with piggy-backing such file transfer this over vdsm.

>  > 
>  > >
>  > >>
>  > >>>>scp? or Will vdsm support new API to list and get these files?
>  > >>>>IMO, The new API should list and get not only dump files, but
>  > >>>>also
>  > >>>>other kinds of files.
>  > >>>>
>  > >>>>
>  > >>>>--
>  > >>>>Sheldon Feng <shaohef(a)linux.vnet.ibm.com&gt; IBM Linux
> Technology
>  > >>>>Center
>  > >>>>_______________________________________________
>  > >>>>vdsm-devel mailing list
>  > &gt;&gt;&gt;&gt;vdsm-devel(a)lists.fedorahosted.org
>  > >>>>https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
>  > >>>>
>  > >>
>  > >>
>  > >>--
>  > >>Sheldon Feng(?????????)&lt;shaohef(a)linux.vnet.ibm.com&gt;
>  > >>IBM Linux Technology Center
>  > >
>  > 
>  
>  --
>  Ryan Harper
>  Software Engineer; Linux Technology Center
>  IBM Corp., Austin, Tx
>  ryanh(a)us.ibm.com
>  
>  _______________________________________________
>  vdsm-devel mailing list
>  vdsm-devel(a)lists.fedorahosted.org
>  https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
>

----- Original Message -----
>  From: "Sheldon" <shaohef(a)linux.vnet.ibm.com&gt;
>  To: "Laszlo Hornyak" <lhornyak(a)redhat.com&gt;
>  Cc: vdsm-devel(a)lists.fedorahosted.org, "Zheng Sheng ZS Zhou"
> <zhshzhou(a)cn.ibm.com&gt;, "Ming M Shu" <shuming(a)cn.ibm.com&gt;
>  Sent: Monday, September 24, 2012 1:48:58 PM
>  Subject: Re: [vdsm] how engine get files from node？
>  
>  On 09/21/2012 08:14 PM, Laszlo Hornyak wrote:
>  >
>  > ----- Original Message -----
>  >> From: "Sheldon" <shaohef(a)linux.vnet.ibm.com&gt;
>  >> To: vdsm-devel(a)lists.fedorahosted.org
>  >> Sent: Friday, September 21, 2012 11:42:05 AM
>  >> Subject: [vdsm]  how engine get files from node？
>  >>
>  >>
>  >> Hi all,
>  >> I have submitted a patch about watchdog device
>  >> http://gerrit.ovirt.org/#/c/7535/
>  >> If we set 'dump' action for watchdog, qemu will generate a dump
>  >> file.
>  >> And I have get some feedback, how engine get these files?
>  > When adding the host (if installation is enabled) the engine does
>  > some ssh/scp to the new host, but this can be turned off. Also,
>  > the guideline is to have a single interface with the host, and
>  > after installation the engine does not try to communicate with
>  > vdsm in any other way than the xmlrpc interface.
>  
>  do you means "ssh/scp" may not be available on host, So a xmlrpc
>  interface should be added to get the files on host?
>  
>  >> scp? or Will vdsm support new API to list and get these files?
>  >> IMO, The new API should list and get not only dump files, but also
>  >> other kinds of files.
>  >>
>  >>
>  >> --
>  >> Sheldon Feng <shaohef(a)linux.vnet.ibm.com&gt; IBM Linux Technology
>  >> Center
>  >> _______________________________________________
>  >> vdsm-devel mailing list
>  >> vdsm-devel(a)lists.fedorahosted.org
>  >> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
>  >>
>  
>  
>  --
>  Sheldon Feng(冯少合)&lt;shaohef(a)linux.vnet.ibm.com&gt;
>  IBM Linux Technology Center
>   
(Added Keith)

Hi Sheldon,
I'd expect ovirt-log-collector to collect these as well, as it does with all
relevant dump files. Obviously you need to make sure these dump files
are handled, and if needed suggest the relevant patch.