3:36 a.m.
Hi,
yesterday I was testing the latest release and I noticed that yum fails
to install debuginfo when previous yum transaction was cancelled.
packagekit just fails to install packages complaining that there are
some unfinished transactions and we get bt full of '?'. We really should
pass these errors to user and make him resolve it. Or if we don't have
debuginfo then we shouldn't even try to get bt? What do you think?
Jirka
4:05 a.m.
On Fri, 16 Oct 2009 10:36:04 +0200, Jiri Moskovcak wrote:
Or if we don't have debuginfo then we shouldn't even try to
get bt?
I think ABRT should avoid using debuginfo even if it is installed as it may be
performance-costly for the client machine and the retrace server can take care
of it better incl. by its use of entries caching across reporting machines.
But I still miss:
* Facts why ABRT is still stepping around the use of external retrace server.
* Decision if parameters and local variables should / should not be dumped.
(such as "bt vs. bt full", there are some changes of it in ABRT changelog)
- Possible security disclosure.
+ Better debuggability
(still not perfect, sometimes even a full core is not enough and one needs
a reproducer to be able to fix the bug).
(My current retrace client/server draft would need to be extended for it.)
Regards,
Jan
5:39 a.m.
On 10/16/2009 11:05 AM, Jan Kratochvil wrote:
On Fri, 16 Oct 2009 10:36:04 +0200, Jiri Moskovcak wrote:
> Or if we don't have debuginfo then we shouldn't even try to get bt?
I think ABRT should avoid using debuginfo even if it is installed as it may be
performance-costly for the client machine and the retrace server can take care
of it better incl. by its use of entries caching across reporting machines.
But I still miss:
* Facts why ABRT is still stepping around the use of external retrace server.
1. How would you imagine it to work for ordinary desktop users, should
they install their own private retrace server or send the cores
somewhere? (neither is possible)
* Decision if parameters and local variables should / should not be
dumped.
(such as "bt vs. bt full", there are some changes of it in ABRT changelog)
- Possible security disclosure.
+ Better debuggability
(still not perfect, sometimes even a full core is not enough and one needs
a reproducer to be able to fix the bug).
(My current retrace client/server draft would need to be extended for it.)
I vote for dumping the variables, but we need to change the gui and make
it easier for user to see and remove sensitive data from the BT.
Jirka
3:32 p.m.
On Fri, 16 Oct 2009 12:39:33 +0200, Jiri Moskovcak wrote:
On 10/16/2009 11:05 AM, Jan Kratochvil wrote:
>On Fri, 16 Oct 2009 10:36:04 +0200, Jiri Moskovcak wrote:
>>Or if we don't have debuginfo then we shouldn't even try to get bt?
>
>I think ABRT should avoid using debuginfo even if it is installed as it may be
>performance-costly for the client machine and the retrace server can take care
>of it better incl. by its use of entries caching across reporting machines.
>
>But I still miss:
>
>* Facts why ABRT is still stepping around the use of external retrace server.
1. How would you imagine it to work for ordinary desktop users,
should they install their own private retrace server or send the
cores somewhere? (neither is possible)
No cores are being sent anywhere. My followup mail
https://fedorahosted.org/pipermail/crash-catcher/2009-October/000056.html
contains the line
(eu-unstrip -n --core=`echo ./core.*`; echo; gdb -nx -ex "set debug-file-directory
/_N" -ex "core-file `echo ./core.*`" -ex bt -ex q) 2>/dev/null | nc
host1.dyn.jankratochvil.net 7221
so you can check yourself
(eu-unstrip -n --core=`echo ./core.*`; echo; gdb -nx -ex "set debug-file-directory
/_N" -ex "core-file `echo ./core.*`" -ex bt -ex q) 2>/dev/null
* sends no secure info
* sends no more info than will be in the final public bugreport
* contant has size 1-2KB (compared to the excessive size of core files)
Sample output being sent out of client is provided at the bottom of this mail.
>* Decision if parameters and local variables should / should not
be dumped.
> (such as "bt vs. bt full", there are some changes of it in ABRT
changelog)
> - Possible security disclosure.
> + Better debuggability
> (still not perfect, sometimes even a full core is not enough and one needs
> a reproducer to be able to fix the bug).
> (My current retrace client/server draft would need to be extended for it.)
I vote for dumping the variables, but we need to change the gui and
make it easier for user to see and remove sensitive data from the BT.
Currently the retrace server is not suitable for dumping the
parameters/variables (and while possible it is not so easy to extend it for
such functionality).
Still even if the retrace server would be extended for such remote
parameters/local variables dumping it will probably have the same security
level as what suggested Denys Vlasenko:
https://fedoraproject.org/wiki/Features/DebuginfoFS
* There is no risk of sending information client -> retrace server.
* There is a risk retrace server -> client will send back malicious debuginfo.
Currently the debuginfo installed by yum at the client is signed by the
Fedora project keys. The retrace server -> client protocol content does not
have such easy security signature.
* The information sent from client is put as a public bug entry into
bugzilla.redhat.com where the malicious retrace server can read it back from.
* debuginfo can contain arbitrary Turing-complete DWARF expressions executed
by the "virtual machine" in GDB, language described at:
http://dwarf.freestandards.org/Dwarf3.pdf
2.5 DWARF Expressions; page 14 (26/267)
Using DebuginfoFS without downloading always the whole .debug file is probably
best to be dependent on .debug_gnu_index under development:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41130
make it easier for user to see and remove sensitive data from the BT.
If the provided debuginfo is not secure (DebuginfoFS) its malicious DWARF
expression can easily encode the retrieved security information ("password")
from the core file hiding from user it can be sensitive:
#1 harmless (s=0x7fffffffd0e0 "\332\313\331\331\335\305\330\316") at x.c:13
Have you recognized this is "password" ^ 0xaa?
Thanks for the discussion, currently I do not have such retrace server if the
parameters / local variables should be present in the dumps.
Regards,
Jan
------------------------------------------------------------------------------
0x8048000+0x11c0000 c3d9884a048a99bb32d84be41a52f29fcc66f8f8@0x8048178 - - [exe]
0x64f000+0x1000 cdd447bf4574ea56b7fbf47a05d5284e8b6f003b@0x64f210 . - linux-gate.so.1
0x2da000+0x23000 a9a2db0e228986191a9c14997e6c44642159fb2e@0x2da104 /lib/libncurses.so.5 -
libncurses.so.5
0xde6000+0x13000 3c2a5009257063bc82390ac100c3539a0b429529@0xde6104 /lib/libz.so.1 -
libz.so.1
0xbb8000+0x2a000 3f8de4868b644fdfff1bed467ffda5977e742345@0xbb8164 /lib/libm.so.6 -
libm.so.6
0x110000+0x190000 f4856ff5efe128e17b3e0a109ed72726f1f7bf7b@0x110104
/usr/lib/libpython2.6.so.1.0 - libpython2.6.so.1.0
0xdb7000+0x28000 e81e2ba05ae0e2af55a870e7821373465720ee85@0xdb7104 /lib/libexpat.so.1 -
libexpat.so.1
0xbb1000+0x5000 51b669a027475003759224f29c8907c8b3516eae@0xbb1164 /lib/libdl.so.2 -
libdl.so.2
0xa32000+0x17d000 a5e40a7b67ddfaca334949ab2808a51b7fbed50d@0xa32184 /lib/libc.so.6 -
libc.so.6
0xe5f000+0x19000 b329a97458f5692d53edfe12ef412c63ebdd2b6e@0xe5f104 /lib/libtinfo.so.5 -
libtinfo.so.5
0xa0e000+0x22000 17de230718c59675603787cd20609a46f5bb94d7@0xa0e124 /lib/ld-linux.so.2 -
ld-linux.so.2
0xcf7000+0x1b000 293dc48a7175b0dabb62ed4ed8cbbb1750f83677@0xcf7164 /lib/libpthread.so.0 -
libpthread.so.0
0x5c4000+0x4000 28bac116ff41e154a69cafe4c196d33e0b96629d@0x5c4164 /lib/libutil.so.1 -
libutil.so.1
GNU gdb (GDB) Fedora (6.8.91.20090930-2.fc12)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>;.
Core was generated by
`/home/jkratoch/redhat/gdb-clean-m32/gdb/testsuite.unix.-m32/../../gdb/gdb -nw -'.
Program terminated with signal 11, Segmentation fault.
#0 0x08475ee8 in ?? ()
#0 0x08475ee8 in ?? ()
#1 0xfff10668 in ?? ()
#2 0x0818b719 in ?? ()
#3 0x95959595 in ?? ()
#4 0x0bf9e0a8 in ?? ()
#5 0x00000000 in ?? ()
4:43 a.m.
On 10/18/2009 10:32 PM, Jan Kratochvil wrote:
On Fri, 16 Oct 2009 12:39:33 +0200, Jiri Moskovcak wrote:
> On 10/16/2009 11:05 AM, Jan Kratochvil wrote:
>> On Fri, 16 Oct 2009 10:36:04 +0200, Jiri Moskovcak wrote:
>>> Or if we don't have debuginfo then we shouldn't even try to get bt?
>>
>> I think ABRT should avoid using debuginfo even if it is installed as it may be
>> performance-costly for the client machine and the retrace server can take care
>> of it better incl. by its use of entries caching across reporting machines.
>>
>> But I still miss:
>>
>> * Facts why ABRT is still stepping around the use of external retrace server.
>
> 1. How would you imagine it to work for ordinary desktop users,
> should they install their own private retrace server or send the
> cores somewhere? (neither is possible)
No cores are being sent anywhere. My followup mail
https://fedorahosted.org/pipermail/crash-catcher/2009-October/000056.html
contains the line
(eu-unstrip -n --core=`echo ./core.*`; echo; gdb -nx -ex "set debug-file-directory
/_N" -ex "core-file `echo ./core.*`" -ex bt -ex q) 2>/dev/null | nc
host1.dyn.jankratochvil.net 7221
so you can check yourself
(eu-unstrip -n --core=`echo ./core.*`; echo; gdb -nx -ex "set debug-file-directory
/_N" -ex "core-file `echo ./core.*`" -ex bt -ex q) 2>/dev/null
* sends no secure info
* sends no more info than will be in the final public bugreport
* contant has size 1-2KB (compared to the excessive size of core files)
Sample output being sent out of client is provided at the bottom of this mail.
Sorry, I didn't pay enough attention to this, as this is more Denys's
part of work in abrt. This way it sounds good.
>> * Decision if parameters and local variables should / should not be dumped.
>> (such as "bt vs. bt full", there are some changes of it in ABRT
changelog)
>> - Possible security disclosure.
>> + Better debuggability
>> (still not perfect, sometimes even a full core is not enough and one needs
>> a reproducer to be able to fix the bug).
>> (My current retrace client/server draft would need to be extended for it.)
>
> I vote for dumping the variables, but we need to change the gui and
> make it easier for user to see and remove sensitive data from the BT.
Currently the retrace server is not suitable for dumping the
parameters/variables (and while possible it is not so easy to extend it for
such functionality).
Nevermind retrace without parameters/variables is better then no retrace :)
Still even if the retrace server would be extended for such remote
parameters/local variables dumping it will probably have the same security
level as what suggested Denys Vlasenko:
https://fedoraproject.org/wiki/Features/DebuginfoFS
* There is no risk of sending information client -> retrace server.
* There is a risk retrace server -> client will send back malicious debuginfo.
Currently the debuginfo installed by yum at the client is signed by the
Fedora project keys. The retrace server -> client protocol content does not
have such easy security signature.
* The information sent from client is put as a public bug entry into
bugzilla.redhat.com where the malicious retrace server can read it back from.
* debuginfo can contain arbitrary Turing-complete DWARF expressions executed
by the "virtual machine" in GDB, language described at:
http://dwarf.freestandards.org/Dwarf3.pdf
2.5 DWARF Expressions; page 14 (26/267)
I don't get this (probably becouse of my lack of knowledge of dwarfs,
elfs, etc ...), why would retrace server send any debuginfo back to
client? I thought the debuginfo (from signed packages) will live on
retrace server and the client just get the backtrace (and even this
seems unnecessary). Or are you talking about debuginfofs server? You
could probably enlighten us in person.
Using DebuginfoFS without downloading always the whole .debug file is
probably
best to be dependent on .debug_gnu_index under development:
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=41130
> make it easier for user to see and remove sensitive data from the BT.
If the provided debuginfo is not secure (DebuginfoFS) its malicious DWARF
expression can easily encode the retrieved security information ("password")
from the core file hiding from user it can be sensitive:
#1 harmless (s=0x7fffffffd0e0 "\332\313\331\331\335\305\330\316") at x.c:13
Have you recognized this is "password" ^ 0xaa?
Yes, I' aware of this, but in this case the only solution I can think of
is to not dump the variables.
Thanks for the discussion, currently I do not have such retrace server if the
parameters / local variables should be present in the dumps.
Regards,
Jan
Thanks,
Jirka
2:59 a.m.
On Mon, 19 Oct 2009 11:43:08 +0200, Jiri Moskovcak wrote:
On 10/18/2009 10:32 PM, Jan Kratochvil wrote:
[...]
> Still even if the retrace server would be extended for such
remote
> parameters/local variables dumping it will probably have the same security
> level as what suggested Denys Vlasenko:
> https://fedoraproject.org/wiki/Features/DebuginfoFS
>
> * There is no risk of sending information client -> retrace server.
>
> * There is a risk retrace server -> client will send back malicious debuginfo.
> Currently the debuginfo installed by yum at the client is signed by the
> Fedora project keys. The retrace server -> client protocol content does not
> have such easy security signature.
>
> * The information sent from client is put as a public bug entry into
> bugzilla.redhat.com where the malicious retrace server can read it back from.
>
> * debuginfo can contain arbitrary Turing-complete DWARF expressions executed
> by the "virtual machine" in GDB, language described at:
> http://dwarf.freestandards.org/Dwarf3.pdf
> 2.5 DWARF Expressions; page 14 (26/267)
>
I don't get this (probably becouse of my lack of knowledge of
dwarfs, elfs, etc ...), why would retrace server send any debuginfo
back to client? I thought the debuginfo (from signed packages) will
live on retrace server and the client just get the backtrace (and
even this seems unnecessary). Or are you talking about debuginfofs
server?
You said you do not trust the retrace server as you cannot send the whole core
file client->retrace server. The retrace server needs various chunks of the
core file to display the backtrace. As the client does not have the debuginfo
it does not know which core chunks are needed for the backtrace. If the
client will send arbitrary chunks of the core as requested by the retrace
server it can disclose any secure information the retrace server would request.
The client must know which info to send to the server. For the
parameters/variables-less backtrace it knows it must send the PC (program
counter) addresses. But determining which memory content will be required to
print the parameters/variables requires general debuginfo access.
debuginfo (.debug file) contains a bytecode, it is like a Java .class file.
It cannot do any harm to your system (it has no filesystem access etc.) but
malicious debuginfo can intelligently access anything from the core file and
present arbitrarily encoded=hidden information in the backtrace.
> > make it easier for user to see and remove sensitive data
from the BT.
>
> If the provided debuginfo is not secure (DebuginfoFS) its malicious DWARF
> expression can easily encode the retrieved security information
("password")
> from the core file hiding from user it can be sensitive:
> #1 harmless (s=0x7fffffffd0e0 "\332\313\331\331\335\305\330\316") at
x.c:13
> Have you recognized this is "password" ^ 0xaa?
Yes, I' aware of this, but in this case the only solution I can
think of is to not dump the variables.
This is a political decision of trust. On Fedora system you already must
trust the binaries you used for the system installation and you trust any
binary rpm updates signed by fedoraproject.org.
For the backtrace you must have trust in the debuginfo content.
* The full rpm-signed *-debuginfo.arch.rpm is too big to transfer to client.
* Client must not send any core file content before it was approved by the
user. User must see the backtrace based on the core memory formatting done
by trusted parts of debuginfo.
=> The only way out is a trusted retrace server based on fedoraproject.org
signing key.
Which technical way to choose (gdbserver protocol from core file, DebuginfoFS
with fixed/improved GDB debuginfo reading etc.) is a secondary question.
The previous paragraph applies exactly the same way to DebuginfoFS server.
Another question if such complicated discussions should be made. Majority of
the Fedora users installs proprietary software on their machine and so they
already gave up their system security in a much significant way than some
trust in retrace server. The few users keeping their system in trust only
with fedoraproject.org may not have any problem downloading full rpm-signed
*-debuginfo.arch.rpm to the client (as myself).
And also I find it invalid to think a user is able to decide if some backtrace
contains secure information, this is technically very complicated decision.
Various private keys looking as binary data passed as parameters etc.
Regards,
Jan
7:03 a.m.
On Fri, 2009-10-16 at 12:39 +0200, Jiri Moskovcak wrote:
On 10/16/2009 11:05 AM, Jan Kratochvil wrote:
> On Fri, 16 Oct 2009 10:36:04 +0200, Jiri Moskovcak wrote:
>> Or if we don't have debuginfo then we shouldn't even try to get bt?
>
> I think ABRT should avoid using debuginfo even if it is installed as it may be
> performance-costly for the client machine and the retrace server can take care
> of it better incl. by its use of entries caching across reporting machines.
>
> But I still miss:
>
> * Facts why ABRT is still stepping around the use of external retrace server.
1. How would you imagine it to work for ordinary desktop users, should
they install their own private retrace server or send the cores
somewhere? (neither is possible)
It's possible. I think we just do not want to make it mandatory for
everyone.
As it stands now, abrt is not working well enough yet even in the
one mode we implemented for it, without retrace servers and such.
Jan, we do understand that in some cases people will want to collect
coredumps and perform backtraces on a dedicated servers, and we want
to eventually have this functionality too. We have it in our design
plans.
But today we are concentrating at making at least the initial feature
set to work correctly. It's not a good time to work on additional
features, when basic features are half-broken. We do not want
to alienate users by giving them a half-broken tool.
That's why I want to make local backtrace generation to work
more reliably.
--
vda
1:34 p.m.
On Tue, 20 Oct 2009 14:03:28 +0200, Denys Vlasenko wrote:
As it stands now, abrt is not working well enough yet even in the
one mode we implemented for it, without retrace servers and such.
[...]
But today we are concentrating at making at least the initial
feature
set to work correctly. It's not a good time to work on additional
features, when basic features are half-broken.
I started pushing the retrace server when I have seen various difficulties you
face to access the debuginfos at client locally and which are only temporary
before the retrace server gets deployed. Using the external retrace server
may be even easier, primarily for user interaction (debuginfos download time,
debuginfos installation privileges, disk/memory+swap space required with user
experience of the low performance; the latter due to GDB insufficiencies).
I think we just do not want to make it mandatory for everyone.
I still do not follow why a retrace server should not be required as you
already have the dependency on public bugzilla.redhat.com for ABRT.
In extreme cases it can work even with the retrace server at localhost.
Regards,
Jan
1:40 p.m.
On Tue, 20 Oct 2009 20:34:58 +0200, Jan Kratochvil wrote:
I started pushing the retrace server when I have seen various
difficulties you
face to access the debuginfos at client locally
[...]
debuginfos installation privileges,
disk space
OK, these are the only two I have seen being solved while going to be
obsoleted by the retrace server.
Regards,
Jan
5323
days inactive
5327
days old
crash-catcher@lists.fedorahosted.org
8 comments
3 participants
participants (3)
-
Denys Vlasenko -
Jan Kratochvil -
Jiri Moskovcak