At the risk of looking like a complete idiot, I'd like to report an apparently serious problem with the recent glibc etc stuff in (I think) Fedora-Development.
In an excess of zeal yesterday I upgraded some packages from the development set and now various programs report "buffer overflow detected" and like messages, and abort. These programs include bash and my usual mail reader. I've reverted my glibc to 2.3.4 from fedora-updates and things are a bit better but not totally fixed, so I figure I've still got some more packages to locate and revert:-(
I suspect this behaviour stems from recent builds using GCC 4 and -D_FORTIFY_SOURCE=2 from the release notes and a bit of perusal of the glibc-2.3.5 sources. Could someone (Jakub?) confirm or discredit this notion please?
If confirmed, is there a URL that documents the effects of this? Is there a runtime way to turn these from "abort" into "warn but proceed"?
If discredited, what then _is_ going on?
I'd like to suggest that this kind of build not be done for any release versions; while all the crashing programs are almost certainly buggy, unless the user can switch the behaviour _off_ they will be very very unhappy.
Yes it's fedora-devel, and I accept I've shot myself in the foot. But a user hoping to test some fedora-devel stuff can too easily end up with a system that is totally uncooperatives as opposed to having a few apps a bit buggy.
Remarks and advice? -- Cameron Simpson cs@zip.com.au DoD#743 http://www.cskk.ezoshosting.com/cs/
If it can't be turned off, it's not a feature. - Karl Heuer
On Thu, Apr 14, 2005 at 04:25:48PM +1000, Cameron Simpson wrote:
At the risk of looking like a complete idiot, I'd like to report an apparently serious problem with the recent glibc etc stuff in (I think) Fedora-Development.
In an excess of zeal yesterday I upgraded some packages from the development set and now various programs report "buffer overflow detected" and like messages, and abort. These programs include bash and my usual mail reader. I've reverted my glibc to 2.3.4 from fedora-updates and things are a bit better but not totally fixed, so I figure I've still got some more packages to locate and revert:-(
No, reversion is not the right first step here. Whenever you see such messages, you should see how is it possible to reproduce it, ideally install the corresponding *debuginfo*.rpm package, get a backtrace from where the buffer overflow happened and report it into bugzilla. Rawhide glibc even prints some limited backtrace by default when the overflow happens.
There are no known bugs in the -D_FORTIFY_SOURCE=2 patches and all buffer overflows it detected (several dozens) turned up to be real bugs in the programs, sometimes very severe.
If confirmed, is there a URL that documents the effects of this? Is there a runtime way to turn these from "abort" into "warn but proceed"?
There is no way to turn it off. If you overflow an buffer, all bets are off what the program is actually doing.
I'd like to suggest that this kind of build not be done for any release versions; while all the crashing programs are almost certainly buggy, unless the user can switch the behaviour _off_ they will be very very unhappy.
In development versions, the intent is obviously that as many such problems are detected and fixed. For released versions it must stay too, because although all/most problems in the usual usage of the programs will be fixed, when you start doing something exceptional/hostile to the programs, such as trying to give it unusually big inputs or exploit in some other way, the aborts are going to turn the vulnerability into a DoS (and show where the problem was, so that it also can be fixed soon if reported).
Jakub
On 14Apr2005 02:54, Jakub Jelinek jakub@redhat.com wrote: | On Thu, Apr 14, 2005 at 04:25:48PM +1000, Cameron Simpson wrote: | > In an excess of zeal yesterday I upgraded some packages from the | > development set and now various programs report "buffer overflow detected" | > and like messages, and abort. These programs include bash [...] | | No, reversion is not the right first step here. | Whenever you see such messages, you should see how is it possible | to reproduce it, ideally install the corresponding *debuginfo*.rpm package, | get a backtrace from where the buffer overflow happened and report | it into bugzilla. Rawhide glibc even prints some limited backtrace | by default when the overflow happens.
I'll give this approach a spin, thanks.
| > If confirmed, is there a URL that documents the effects of this?
??
| > Is there a runtime way to turn these from "abort" into "warn but proceed"? | There is no way to turn it off. If you overflow an buffer, all bets are | off what the program is actually doing.
I agree, being a purist myself, but often the obscure overflows let the app continue apparently unharmed. When it's a core tool like /bin/sh that can occasionally be desirable.
| > I'd like to suggest that this kind of build not be done for any release | > versions; while all the crashing programs are almost certainly buggy, | > unless the user can switch the behaviour _off_ they will be very very | > unhappy. | | In development versions, the intent is obviously that as many such problems | are detected and fixed.
Ah, now there's the rub. Some naive people (like me) looked to the rawhide/devel repository as a source of "more current" packages i.e. closer to the packages author's live set with some desirable features not in the QAed release package. A slightly different expectation than a "testing arbitrary build features" variety of repository, which is what fedora-devel is right now.
| For released versions it must stay too, because | although all/most problems in the usual usage of the programs will be fixed, | when you start doing something exceptional/hostile to the programs, such as | trying to give it unusually big inputs or exploit in some other way, | the aborts are going to turn the vulnerability into a DoS (and show where | the problem was, so that it also can be fixed soon if reported).
Interesting. Even for core apps like /bin/sh? (Or course, one can successfully argue that it's even more important to abort core apps before really nasty stuff happens.) I'm not disagreeing with you, btw, just interested in the thinking.
Cheers,
I agree, being a purist myself, but often the obscure overflows let the app continue apparently unharmed. When it's a core tool like /bin/sh that can occasionally be desirable.
well when a buffer overflow happens basically 3 things can happen
1) the overflow is limited to the padding space between variables on the stack 2) the overflow also overwrites other variables on the stack 3) the overflow gets as far as overwriting the return address on the stack
3) is the most common exploit vector. 2) sometimes can be exploited too, but that is rare. HOWEVER 2) is also a case that leads to crashes or data corruption. 1) is harmless of course.
Now... even when you don't hit a security exploit... do you *really* want the risk of data corruption ???
On Thu, 2005-04-14 at 16:25 +1000, Cameron Simpson wrote:
At the risk of looking like a complete idiot, I'd like to report an apparently serious problem with the recent glibc etc stuff in (I think) Fedora-Development.
In an excess of zeal yesterday I upgraded some packages from the development set and now various programs report "buffer overflow detected" and like messages, and abort. These programs include bash and my usual mail reader. I've reverted my glibc to 2.3.4 from fedora-updates and things are a bit better but not totally fixed, so I figure I've still got some more packages to locate and revert:-(
please please please file bugs about these... we really have to fix all of these urgently since buffer overflows are one of the most dangerous security exploit vectors!
Cameron Simpson wrote:
At the risk of looking like a complete idiot, I'd like to report an apparently serious problem with the recent glibc etc stuff in (I think) Fedora-Development.
In an excess of zeal yesterday I upgraded some packages from the development set and now various programs report "buffer overflow detected" and like messages, and abort. These programs include bash and my usual mail reader. I've reverted my glibc to 2.3.4 from fedora-updates and things are a bit better but not totally fixed, so I figure I've still got some more packages to locate and revert:-(
Umm, it sounds like you tried to install FC4 packages onto FC3. DO NOT DO THAT. That is totally unsupported and very likely the reason for things exploding.
Warren Togami wtogami@redhat.com
On 13Apr2005 23:06, Warren Togami wtogami@redhat.com wrote: | Cameron Simpson wrote: | >At the risk of looking like a complete idiot, I'd like to report [...that...] | >In an excess of zeal yesterday I upgraded some packages from the | >development set and now various programs report "buffer overflow detected" | >and like messages, and abort. [...] | | Umm, it sounds like you tried to install FC4 packages onto FC3.
Correct.
| DO NOT DO THAT. That is totally unsupported and very likely the reason for | things exploding.
I can believe that, but can you elaborate briefly on why this might be so, in terms of mechanisms?
Does the fortify stuff put canaries etc in data structures and require apps to be built with the same flags to insert things like that that are later checked by the lower layers (eg glibc)?
Still, my /etc/yum.repos.d/fedora-devel.repo came from my FC3 install, so one might imagine that although they are development packages and could exhibit any behaviour, things might work. And, in fact, broadly they do. The bash explosion, while perfectly repeatable, is proving very difficult to reproduce in a test case I can show someone else.
Now, one interesting thing is that I upgraded these things with yum, which was following the usual dependency stuff. I didn't lie to RPM anywhere. Looking at an ldd of bash it loads this:
linux-gate.so.1 => (0xffffe000) libtermcap.so.2 => /lib/libtermcap.so.2 (0x00583000) libdl.so.2 => /lib/libdl.so.2 (0xb7f01000) libc.so.6 => /lib/tls/libc.so.6 (0xb7dd7000) /lib/ld-linux.so.2 (0xb7f29000)
They would all have been updated. Now, bash _should_ be getting a perfectly clean fedora-devel set of libs and of course grows from an exec(), so to my mind there should be no opportunity for a non-FC4 facility to be getting into the picture. Should _that_ be expected to work, regardless of other bustedness I may have introduced to my system?
I accept that bustedness might be expected of other, mixed FC3/FC4, apps but I'd have thought for bash and its few libs I'd be in a "pure FC4" zone.
BTW, while exhibiting my cluelessness, what does this mean?
[~]zoob*> rpm -qf /lib/libdl.so.2 glibc-2.3.5-0.fc3.1 glibc-2.3.4-2.fc3
I've downgraded glibc to the FC3 one for now (which actually hasn't changed the behaviours), which doubtless caused the above thing but I confess to being surprised that the RPM db could claim the above.
I'll see if I can find a on which box to run up FC4test2 today, too.
Cheers,
-
Arjan van de Ven -
Cameron Simpson -
Jakub Jelinek -
Warren Togami