Re: [RFC][PATCH] Add --split support for dump on filesystem

[Vivek Goyal]

On Fri, Apr 11, 2014 at 10:47:14AM +0900, HATAYAMA Daisuke wrote:
> From: Vivek Goyal 
> Subject: Re: [RFC][PATCH] Add --split support for dump on filesystem
> Date: Thu, 10 Apr 2014 10:40:03 -0400
> > On Wed, Apr 09, 2014 at 03:46:22PM +0900, HATAYAMA Daisuke wrote:
> >
> > Hi Hatayama,
> > > So, I'll post a patch to support --split patch as a means of
> > > supporting parallelism. Do you agree with this direction?
> >
> > --split is already an existing parameter which very clearly means
> > that split a file into multiple files. I don't know how --split is
> > implemented, but I am assuming it creates as many threads as there
> > are split files and these threads will do filtering, compression and
> > IO.
> >
> > Usage of --split requires that we *have* to specify multiple files
> > as output files.
> >
> > How about creating a new option say --parallel-dump <nr-threads>. This
> > option can take number of threads to create for filetering and compression
> > and possibly for doing IO also  (to single file).
> >
> > --parallel-dump will take only single output file as argument. So dump
> > is always saved in single file.
> >
> > We can also make argument <nr-threads> optional. If user specifies number
> > of threads, those many threads will be launched. Otherwise makedumpfile
> > can detect how many cpus are online and launch those many threads.
> >
> > For example.
> >
> > # makedumpfile --parallel-dump 2 /proc/vmcore /var/crash/saved-vmcore
> >
> > In this case 2 threads will be launched which will coordinate internally
> > on filtering and compression and possibly on doing IO too.
> >
> > # maedumpfile --parallel-dump /proc/vmcore /var/cras/saved-vmcore
> >
> > In this case makedumpfile will determine how many cpus are online and
> > launch as many threads (one thread for each cpu). This is close to "-j"
> > option of "make" utility. Difference is that "-j" without option launches
> > as many threads as possible.
>
> This interface is almost the same as what I think.

So I have a question. How many threads does --split create? Does it
create 1 thread right now to dump to multiple devices?

If yes, we could possibly extend the semantics of --split instead of
introducing new option --parallel-dump.

#makedumpfile --split [<nr-threads>] /proc/vmcore <dump-files>

Now one can specify number of threads to launch to do parallel processing
to save to dump files.

--split should allow even single dump file to be specified.

if nr-threads is not specified, then --split will launch as many threads
as thre are cpus available to take max advantage of parallelism.

#makedupfile --split /proc/vmcore  dumpfile1 dumpfile2 ....
    - Launch as many threads as there are cpus.

#makedupfile --split N /proc/vmcore  dumpfile1 dumpfile2 ....
    - Launch N threads.

#makedupfile --split N /proc/vmcore  dumpfile
    - Launch N threads and save dump to 1 file.

#makedupfile --split  /proc/vmcore  dumpfile
    - Launch as many threads as therea re cpus and save dump to 1 file.

[..]
> > Right now we don't seem to have a way to specify destination vmcore
> > file name. If need be, we can possibly create a new option to specify
> > file names and then --split option should work.
>
> How about the following directive?
>
> # core_collector_vmcore_arguments
> #
> #                       - This directive allows you to specify vmcore file
> #                         names to makedumpfile. Default is vmcore.
> #
> #                         A specific usecase of this directive is to specify
> #                         multiple vmcore file names for --split option.
> #                         See /sbin/makedumpfile --help for --split option.
> #
> #                         This directive is ignored if core_collector is not
> #                         used.
>
> Usage of this directive is like this:
>
> core_collector_vmcore_arguments vmcore-0 vmcore-1 vmcore-2
>
> Users know how many arguments they should specify here from
> documentation.

May be something like

vmcore_name vmcore-0 vmcore-1 vmcore-2

Other option could be that we start asking for absolute paths of vmcore
files.

absolute_path /var/crash/vmcore-0  /var/crash/vmcore-1 .....

This way one would automatically decide the underlying disk based on
path and then last string in path will suggest file path.

But I am not sure, I will have to think more about it.

We can first make the case of 1 dump file work and then look into extending
kdump.conf to support multiple files. Above will turn into a more
complicated project. With lots of error checks to make sure all paths lead
to same device. We really don't have a way to support multiple devices
properly.

Thanks
Vivek