crash on F20 beta says log_buf data structure has changed

[Vivek Goyal]

Hi Dave,

I captured a crash of F20 beta system and opened vmcore. crash gives me
warning that log_buf has changed and "log" command does not output
anything.

I thought you fixed it. Is it just a problem of pushing crash update
to F20?

Thanks
Vivek

# crash /usr/lib/debug/lib/modules/3.11.8-300.fc20.x86_64/vmlinux vmcore

crash 7.0.1-2.fc20
Copyright (C) 2002-2013  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for
details.

GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

KERNEL: /usr/lib/debug/lib/modules/3.11.8-300.fc20.x86_64/vmlinux
    DUMPFILE: vmcore  [PARTIAL DUMP]
        CPUS: 4
        DATE: Tue Nov 19 16:33:43 2013
      UPTIME: 00:03:24
LOAD AVERAGE: 0.18, 0.37, 0.18
       TASKS: 172
    NODENAME: <Edited>
     RELEASE: 3.11.8-300.fc20.x86_64
     VERSION: #1 SMP Wed Nov 13 16:34:27 UTC 2013
     MACHINE: x86_64  (2333 Mhz)
      MEMORY: 4 GB
       PANIC: 
WARNING: log buf data structure(s) have changed
""
         PID: 10584
     COMMAND: "bash"
        TASK: ffff8801371d96e0  [THREAD_INFO: ffff880138dd8000]
         CPU: 3
       STATE: TASK_RUNNING (PANIC)

crash> log

log: WARNING: log buf data structure(s) have changed

[Dave Anderson]

----- Original Message -----
> Hi Dave,
>
> I captured a crash of F20 beta system and opened vmcore. crash gives me
> warning that log_buf has changed and "log" command does not output
> anything.
>
> I thought you fixed it. Is it just a problem of pushing crash update
> to F20?

I did, but as part of it, there's also this check for further changes:

/*
                 * If things change, don't kill a dumpfile session
                 * searching for a panic message.
                 */
                if (INVALID_SIZE(log) ||
                    INVALID_MEMBER(log_ts_nsec) ||
                    INVALID_MEMBER(log_len) ||
                    INVALID_MEMBER(log_text_len) ||
                    INVALID_MEMBER(log_dict_len) ||
                    (INVALID_MEMBER(log_level) && INVALID_MEMBER(log_flags_level)) ||
                    !kernel_symbol_exists("log_buf_len") ||
                    !kernel_symbol_exists("log_buf")) {
                        error(WARNING, "\nlog buf data structure(s) have changed\n");
                        return;
                }

Is the system still available, or do you have a vmcore from that kernel?

Dave
> Thanks
> Vivek
>
> # crash /usr/lib/debug/lib/modules/3.11.8-300.fc20.x86_64/vmlinux vmcore
>
> crash 7.0.1-2.fc20
> Copyright (C) 2002-2013  Red Hat, Inc.
> Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
> Copyright (C) 1999-2006  Hewlett-Packard Co
> Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
> Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
> Copyright (C) 2005, 2011  NEC Corporation
> Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
> Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
> This program is free software, covered by the GNU General Public License,
> and you are welcome to change it and/or distribute copies of it under
> certain conditions.  Enter "help copying" to see the conditions.
> This program has absolutely no warranty.  Enter "help warranty" for
> details.
>
> GNU gdb (GDB) 7.6
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> http://gnu.org/licenses/gpl.html
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu"...
>   KERNEL: /usr/lib/debug/lib/modules/3.11.8-300.fc20.x86_64/vmlinux
> DUMPFILE: vmcore  [PARTIAL DUMP]
>     CPUS: 4
>     DATE: Tue Nov 19 16:33:43 2013
>   UPTIME: 00:03:24
>
> LOAD AVERAGE: 0.18, 0.37, 0.18
>        TASKS: 172
>     NODENAME: <Edited>
>      RELEASE: 3.11.8-300.fc20.x86_64
>      VERSION: #1 SMP Wed Nov 13 16:34:27 UTC 2013
>      MACHINE: x86_64  (2333 Mhz)
>       MEMORY: 4 GB
>        PANIC:
> WARNING: log buf data structure(s) have changed
> ""
>          PID: 10584
>      COMMAND: "bash"
>         TASK: ffff8801371d96e0  [THREAD_INFO: ffff880138dd8000]
>          CPU: 3
>        STATE: TASK_RUNNING (PANIC)
>
> crash> log
>
> log: WARNING: log buf data structure(s) have changed

[Dave Anderson]

----- Original Message -----
> ----- Original Message -----
> > Hi Dave,
> >
> > I captured a crash of F20 beta system and opened vmcore. crash gives me
> > warning that log_buf has changed and "log" command does not output
> > anything.
> >
> > I thought you fixed it. Is it just a problem of pushing crash update
> > to F20?

Also, what version of crash are you using?  The fix for the structure name change
from "log" to "printk_log" went into crash-7.0.2.  I'm not sure whether that made
it into f20 from rawhide, or if f20 is still sitting at crash-7.0.1.

Dave
> I did, but as part of it, there's also this check for further changes:
>             /*
>              * If things change, don't kill a dumpfile session
>              * searching for a panic message.
>              */
>             if (INVALID_SIZE(log) ||
>                 INVALID_MEMBER(log_ts_nsec) ||
>                 INVALID_MEMBER(log_len) ||
>                 INVALID_MEMBER(log_text_len) ||
>                 INVALID_MEMBER(log_dict_len) ||
>                 (INVALID_MEMBER(log_level) &&
>                 INVALID_MEMBER(log_flags_level)) ||
>                 !kernel_symbol_exists("log_buf_len") ||
>                 !kernel_symbol_exists("log_buf")) {
>                     error(WARNING, "\nlog buf data structure(s) have
>                     changed\n");
>                     return;
>             }
>
> Is the system still available, or do you have a vmcore from that kernel?
>
> Dave
> > Thanks
> > Vivek
> >
> > # crash /usr/lib/debug/lib/modules/3.11.8-300.fc20.x86_64/vmlinux vmcore
> >
> > crash 7.0.1-2.fc20
> > Copyright (C) 2002-2013  Red Hat, Inc.
> > Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
> > Copyright (C) 1999-2006  Hewlett-Packard Co
> > Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
> > Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
> > Copyright (C) 2005, 2011  NEC Corporation
> > Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
> > Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
> > This program is free software, covered by the GNU General Public License,
> > and you are welcome to change it and/or distribute copies of it under
> > certain conditions.  Enter "help copying" to see the conditions.
> > This program has absolutely no warranty.  Enter "help warranty" for
> > details.
> >
> > GNU gdb (GDB) 7.6
> > Copyright (C) 2013 Free Software Foundation, Inc.
> > License GPLv3+: GNU GPL version 3 or later
> > http://gnu.org/licenses/gpl.html
> > This is free software: you are free to change and redistribute it.
> > There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> > and "show warranty" for details.
> > This GDB was configured as "x86_64-unknown-linux-gnu"...
> >   KERNEL: /usr/lib/debug/lib/modules/3.11.8-300.fc20.x86_64/vmlinux
> > DUMPFILE: vmcore  [PARTIAL DUMP]
> >     CPUS: 4
> >     DATE: Tue Nov 19 16:33:43 2013
> >   UPTIME: 00:03:24
> >
> > LOAD AVERAGE: 0.18, 0.37, 0.18
> >        TASKS: 172
> >     NODENAME: <Edited>
> >      RELEASE: 3.11.8-300.fc20.x86_64
> >      VERSION: #1 SMP Wed Nov 13 16:34:27 UTC 2013
> >      MACHINE: x86_64  (2333 Mhz)
> >       MEMORY: 4 GB
> >        PANIC:
> > WARNING: log buf data structure(s) have changed
> > ""
> >          PID: 10584
> >      COMMAND: "bash"
> >         TASK: ffff8801371d96e0  [THREAD_INFO: ffff880138dd8000]
> >          CPU: 3
> >        STATE: TASK_RUNNING (PANIC)
> >
> > crash> log
> >
> > log: WARNING: log buf data structure(s) have changed

[Dave Anderson]

----- Original Message -----
> ----- Original Message -----
> > ----- Original Message -----
> > > Hi Dave,
> > >
> > > I captured a crash of F20 beta system and opened vmcore. crash gives me
> > > warning that log_buf has changed and "log" command does not output
> > > anything.
> > >
> > > I thought you fixed it. Is it just a problem of pushing crash update
> > > to F20?
>
> Also, what version of crash are you using?  The fix for the structure name change
> from "log" to "printk_log" went into crash-7.0.2.  I'm not sure whether that made
> it into f20 from rawhide, or if f20 is still sitting at crash-7.0.1.
>
> Dave

Sorry, I didn't check the output below -- it's still at 7.0.1-2.fc20.
Can you file a Fedora BZ, and when it opens for bug fixes, I'll push
it out.

Until then, you can get the Rawhide version here:

Information for build crash-7.0.3-1.fc21
 http://koji.fedoraproject.org/koji/buildinfo?buildID=474614

Dave
> > I did, but as part of it, there's also this check for further changes:
> >             /*
> >              * If things change, don't kill a dumpfile session
> >              * searching for a panic message.
> >              */
> >             if (INVALID_SIZE(log) ||
> >                 INVALID_MEMBER(log_ts_nsec) ||
> >                 INVALID_MEMBER(log_len) ||
> >                 INVALID_MEMBER(log_text_len) ||
> >                 INVALID_MEMBER(log_dict_len) ||
> >                 (INVALID_MEMBER(log_level) &&
> >                 INVALID_MEMBER(log_flags_level)) ||
> >                 !kernel_symbol_exists("log_buf_len") ||
> >                 !kernel_symbol_exists("log_buf")) {
> >                     error(WARNING, "\nlog buf data structure(s) have
> >                     changed\n");
> >                     return;
> >             }
> >
> > Is the system still available, or do you have a vmcore from that kernel?
> >
> > Dave
> > > Thanks
> > > Vivek
> > >
> > > # crash /usr/lib/debug/lib/modules/3.11.8-300.fc20.x86_64/vmlinux vmcore
> > >
> > > crash 7.0.1-2.fc20
> > > Copyright (C) 2002-2013  Red Hat, Inc.
> > > Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
> > > Copyright (C) 1999-2006  Hewlett-Packard Co
> > > Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
> > > Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
> > > Copyright (C) 2005, 2011  NEC Corporation
> > > Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
> > > Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
> > > This program is free software, covered by the GNU General Public License,
> > > and you are welcome to change it and/or distribute copies of it under
> > > certain conditions.  Enter "help copying" to see the conditions.
> > > This program has absolutely no warranty.  Enter "help warranty" for
> > > details.
> > >
> > > GNU gdb (GDB) 7.6
> > > Copyright (C) 2013 Free Software Foundation, Inc.
> > > License GPLv3+: GNU GPL version 3 or later
> > > http://gnu.org/licenses/gpl.html
> > > This is free software: you are free to change and redistribute it.
> > > There is NO WARRANTY, to the extent permitted by law.  Type "show
> > > copying"
> > > and "show warranty" for details.
> > > This GDB was configured as "x86_64-unknown-linux-gnu"...
> > >   KERNEL: /usr/lib/debug/lib/modules/3.11.8-300.fc20.x86_64/vmlinux
> > > DUMPFILE: vmcore  [PARTIAL DUMP]
> > >     CPUS: 4
> > >     DATE: Tue Nov 19 16:33:43 2013
> > >   UPTIME: 00:03:24
> > >
> > > LOAD AVERAGE: 0.18, 0.37, 0.18
> > >        TASKS: 172
> > >     NODENAME: <Edited>
> > >      RELEASE: 3.11.8-300.fc20.x86_64
> > >      VERSION: #1 SMP Wed Nov 13 16:34:27 UTC 2013
> > >      MACHINE: x86_64  (2333 Mhz)
> > >       MEMORY: 4 GB
> > >        PANIC:
> > > WARNING: log buf data structure(s) have changed
> > > ""
> > >          PID: 10584
> > >      COMMAND: "bash"
> > >         TASK: ffff8801371d96e0  [THREAD_INFO: ffff880138dd8000]
> > >          CPU: 3
> > >        STATE: TASK_RUNNING (PANIC)
> > >
> > > crash> log
> > >
> > > log: WARNING: log buf data structure(s) have changed

[Vivek Goyal]

On Wed, Nov 20, 2013 at 08:46:47AM -0500, Dave Anderson wrote:

[..]
> Can you file a Fedora BZ, and when it opens for bug fixes, I'll push
> it out.
>
> Until then, you can get the Rawhide version here:
>
> Information for build crash-7.0.3-1.fc21
>  http://koji.fedoraproject.org/koji/buildinfo?buildID=474614

Thanks Dave.

I opened following bug to keep track of this issue.

https://bugzilla.redhat.com/show_bug.cgi?id=1032623

Vivek