8:50 a.m.
Hello,
Every now and then I look at the distribution to see that from an auditing
perspective the OS is nicely behaving in the absence of intrusion. Meaning we
are not getting audit events unnecessarily. One of the typical rules required
by the DISA STIG is to watch for file access being denied due to permissions.
This could be indicative of someone trying to access something they shouldn't.
The rule is:
-a always,exit -F arch=b64 -S creat -S open -S openat -S truncate -S ftruncate
-F exit=-EPERM -F auid>=500 -F auid!=4294967295 -k access
Lately F18 has been showing lots of audit events where access is being denied
opening a file:
# aureport --start today --key --summary
Key Summary Report
===========================
total key
===========================
537 access
25 module-load
6 container-config
6 system-locale
6 bypass
This is after using the system for 2 hours. Let's see what files were trying to
be accessed:
]# ausearch --start today -k access --raw | aureport --file --summary | head
File Summary Report
===========================
total file
===========================
88 /dev/shm/pulse-shm-838240362
88 /dev/shm/pulse-shm-191856280
88 /dev/shm/pulse-shm-3756395503
7 /dev/shm/pulse-shm-1545675388
6 /usr/share/icons/hicolor/48x48/apps/firefox.png
Let's look at one of these pule-shm events:
# ausearch --start today -k access -f pulse-shm -i --just-one
----
type=PATH msg=audit(06/07/2013 07:13:46.377:215) : item=0 name=/dev/shm/pulse-
shm-3756395503 inode=25089 dev=00:10 mode=file,400 ouid=gdm ogid=gdm rdev=00:00
obj=system_u:object_r:user_tmpfs_t:s0
type=CWD msg=audit(06/07/2013 07:13:46.377:215) : cwd=/
type=SYSCALL msg=audit(06/07/2013 07:13:46.377:215) : arch=x86_64 syscall=open
success=no exit=-13(Permission denied) a0=0x7fffbeda83c0 a1=O_RDONLY|
O_NOFOLLOW|O_CLOEXEC a2=0x0 a3=0x0 items=1 ppid=2369 pid=2370 auid=sgrubb
uid=sgrubb gid=sgrubb euid=sgrubb suid=sgrubb fsuid=sgrubb egid=sgrubb
sgid=sgrubb fsgid=sgrubb ses=2 tty=(none) comm=pulseaudio
exe=/usr/bin/pulseaudio subj=unconfined_u:system_r:unconfined_t:s0 key=access
So, gdm is creating a file 400 and pulse-audio can't open it. Is it suppose to
be like that?
What about the firefox one?
type=PATH msg=audit(06/07/2013 08:13:52.465:633) : item=0
name=/usr/share/icons/hicolor/22x22/apps/firefox.png inode=16943495 dev=09:01
mode=file,644 ouid=root ogid=root rdev=00:00 obj=system_u:object_r:usr_t:s0
type=CWD msg=audit(06/07/2013 08:13:52.465:633) : cwd=/home/sgrubb
type=SYSCALL msg=audit(06/07/2013 08:13:52.465:633) : arch=x86_64 syscall=open
success=no exit=-1(Operation not permitted) a0=0x7fc3be80e880 a1=O_RDONLY|
O_NOATIME a2=0x0 a3=0x0 items=1 ppid=2395 pid=2670 auid=sgrubb uid=sgrubb
gid=sgrubb euid=sgrubb suid=sgrubb fsuid=sgrubb egid=sgrubb sgid=sgrubb
fsgid=sgrubb ses=2 tty=(none) comm=firefox exe=/usr/lib64/firefox/firefox
subj=unconfined_u:unconfined_r:unconfined_t:s0 key=access
It is using the O_NOATIME flag. To use O_NOATIME requires CAP_FOWNER and firefox
should no have capabilities. So, how bad is the NOATIME problem?
# ausearch --start today -k access -i | grep NOATIME | awk '{ print $30 }' |
~sgrubb/working/BUILD/numeric-tools/summary
exe=/usr/bin/cinnamon 193
exe=/usr/lib64/firefox/firefox 31
exe=/usr/bin/gnome-terminal 12
exe=/usr/bin/nautilus 10
exe=/usr/bin/xchat 4
exe=/usr/bin/nm-applet 2
exe=/usr/bin/kontact 2
This seems to account for a good number of the problematic accesses. Anything
except backup programs using O_NOATIME is probably thinking they are making
the program faster, but instead is failing to open files.
-Steve
10:14 a.m.
On Fri, 07.06.13 09:50, Steve Grubb (sgrubb(a)redhat.com) wrote:
Let's look at one of these pule-shm events:
# ausearch --start today -k access -f pulse-shm -i --just-one
----
type=PATH msg=audit(06/07/2013 07:13:46.377:215) : item=0 name=/dev/shm/pulse-
shm-3756395503 inode=25089 dev=00:10 mode=file,400 ouid=gdm ogid=gdm rdev=00:00
obj=system_u:object_r:user_tmpfs_t:s0
type=CWD msg=audit(06/07/2013 07:13:46.377:215) : cwd=/
type=SYSCALL msg=audit(06/07/2013 07:13:46.377:215) : arch=x86_64 syscall=open
success=no exit=-13(Permission denied) a0=0x7fffbeda83c0 a1=O_RDONLY|
O_NOFOLLOW|O_CLOEXEC a2=0x0 a3=0x0 items=1 ppid=2369 pid=2370 auid=sgrubb
uid=sgrubb gid=sgrubb euid=sgrubb suid=sgrubb fsuid=sgrubb egid=sgrubb
sgid=sgrubb fsgid=sgrubb ses=2 tty=(none) comm=pulseaudio
exe=/usr/bin/pulseaudio subj=unconfined_u:system_r:unconfined_t:s0 key=access
So, gdm is creating a file 400 and pulse-audio can't open it. Is it suppose to
be like that?
Yes, it is.
POSIX shared memory doesn't define any useful scheme for automatic
removing of shared memory segments from /dev/shm after use. Hence, in
order to make sure that left-over segments don't fill up /dev/shm
forever PA will try to GC dead segments from /dev/shm on each
start-up. For that it iterates through /dev/shm/pulse-shm*, tries to
read the PID that is stored in there. When the PID still exists it goes
to the next file. If the PID doesn't exist it unlinks the file and then
goes to the next one. It's a simple scheme that works around the
limitations of POSIX shm. Of course /dev/shm is a single namespace for
all users, hence not all files can be opened, and that's totally cool
and expected, and they will be skipped.
Shared memory on Linux is a mess. Not automatic clean up, no quota
limits, it's a sad story. If you care about security and reliability, it
would be great doing something about this, so that arbitrary users
cannot DoS the system this easily anymore...
Lennart
--
Lennart Poettering - Red Hat, Inc.
10:44 a.m.
On Friday, June 07, 2013 05:14:30 PM Lennart Poettering wrote:
On Fri, 07.06.13 09:50, Steve Grubb (sgrubb(a)redhat.com) wrote:
> Let's look at one of these pule-shm events:
> # ausearch --start today -k access -f pulse-shm -i --just-one
> ----
> type=PATH msg=audit(06/07/2013 07:13:46.377:215) : item=0
> name=/dev/shm/pulse- shm-3756395503 inode=25089 dev=00:10 mode=file,400
> ouid=gdm ogid=gdm rdev=00:00 obj=system_u:object_r:user_tmpfs_t:s0
> type=CWD msg=audit(06/07/2013 07:13:46.377:215) : cwd=/
> type=SYSCALL msg=audit(06/07/2013 07:13:46.377:215) : arch=x86_64
> syscall=open success=no exit=-13(Permission denied) a0=0x7fffbeda83c0
> a1=O_RDONLY| O_NOFOLLOW|O_CLOEXEC a2=0x0 a3=0x0 items=1 ppid=2369
> pid=2370 auid=sgrubb uid=sgrubb gid=sgrubb euid=sgrubb suid=sgrubb
> fsuid=sgrubb egid=sgrubb sgid=sgrubb fsgid=sgrubb ses=2 tty=(none)
> comm=pulseaudio
> exe=/usr/bin/pulseaudio subj=unconfined_u:system_r:unconfined_t:s0
> key=access
>
> So, gdm is creating a file 400 and pulse-audio can't open it. Is it
> suppose to be like that?
Yes, it is.
88 times? Something changed. It didn't used to be this bad. Its doing this
over and over on the same file it was denied access on previously.
POSIX shared memory doesn't define any useful scheme for
automatic
removing of shared memory segments from /dev/shm after use. Hence, in
order to make sure that left-over segments don't fill up /dev/shm
forever PA will try to GC dead segments from /dev/shm on each
start-up. For that it iterates through /dev/shm/pulse-shm*, tries to
read the PID that is stored in there.
Maybe the uid can be encoded in the name so that wrong uid's are skipped?
When the PID still exists it goes
to the next file. If the PID doesn't exist it unlinks the file and then
goes to the next one. It's a simple scheme that works around the
limitations of POSIX shm. Of course /dev/shm is a single namespace for
all users, hence not all files can be opened, and that's totally cool
and expected, and they will be skipped.
Shared memory on Linux is a mess. Not automatic clean up,
shmctl(<id>, IPC_RMID doesn't help?
-Steve
no quota limits, it's a sad story. If you care about security
and
reliability, it would be great doing something about this, so that arbitrary
users cannot DoS the system this easily anymore...
10:48 a.m.
On Fri, 07.06.13 11:44, Steve Grubb (sgrubb(a)redhat.com) wrote:
88 times? Something changed. It didn't used to be this bad. Its
doing this
over and over on the same file it was denied access on previously.
Actually all libpulse clients do this.
> POSIX shared memory doesn't define any useful scheme for
automatic
> removing of shared memory segments from /dev/shm after use. Hence, in
> order to make sure that left-over segments don't fill up /dev/shm
> forever PA will try to GC dead segments from /dev/shm on each
> start-up. For that it iterates through /dev/shm/pulse-shm*, tries to
> read the PID that is stored in there.
Maybe the uid can be encoded in the name so that wrong uid's are
skipped?
But why? This stuff should be simple, and it's always a better idea to
simply let the kernel do EACCES rather then trying to be smarter than
the kernel and reimplement access control in userspace.
> When the PID still exists it goes
> to the next file. If the PID doesn't exist it unlinks the file and then
> goes to the next one. It's a simple scheme that works around the
> limitations of POSIX shm. Of course /dev/shm is a single namespace for
> all users, hence not all files can be opened, and that's totally cool
> and expected, and they will be skipped.
>
> Shared memory on Linux is a mess. Not automatic clean up,
shmctl(<id>, IPC_RMID doesn't help?
shmctl() is SysV shared memory. /dev/shm is POSIX shared memory. So, no,
it wouldn't help. And IPC_RMID is in no way nicer than unlink() of a
file. Both are invitations for leaving dead files around. They are both
badly designed interfaces.
Lennart
--
Lennart Poettering - Red Hat, Inc.
11:09 a.m.
On Friday, June 07, 2013 05:48:39 PM Lennart Poettering wrote:
On Fri, 07.06.13 11:44, Steve Grubb (sgrubb(a)redhat.com) wrote:
> 88 times? Something changed. It didn't used to be this bad. Its doing this
> over and over on the same file it was denied access on previously.
Actually all libpulse clients do this.
> > POSIX shared memory doesn't define any useful scheme for automatic
> > removing of shared memory segments from /dev/shm after use. Hence, in
> > order to make sure that left-over segments don't fill up /dev/shm
> > forever PA will try to GC dead segments from /dev/shm on each
> > start-up. For that it iterates through /dev/shm/pulse-shm*, tries to
> > read the PID that is stored in there.
>
> Maybe the uid can be encoded in the name so that wrong uid's are
> skipped?
But why?
So that you are not filling up the audit logs. There are people that have to
use these audit rules and we need a normally operating system to produce as
few false positives as possible.
This stuff should be simple, and it's always a better idea to
simply let the kernel do EACCES rather then trying to be smarter than
the kernel and reimplement access control in userspace.
Well, you are already trusting the name. Who's to say some rougue process
didn't create the file name to try and trick pulseaudio? How about doing like
some processes do in the /tmp dir...put the segments in a directory
/dev/shm/<user name>/ and then scan all the files that belong to that user?
There are ways to make this better if you are willing. :-)
Thanks,
-Steve
11:21 a.m.
On Fri, 07.06.13 12:09, Steve Grubb (sgrubb(a)redhat.com) wrote:
> > > POSIX shared memory doesn't define any useful
scheme for automatic
> > > removing of shared memory segments from /dev/shm after use. Hence, in
> > > order to make sure that left-over segments don't fill up /dev/shm
> > > forever PA will try to GC dead segments from /dev/shm on each
> > > start-up. For that it iterates through /dev/shm/pulse-shm*, tries to
> > > read the PID that is stored in there.
> >
> > Maybe the uid can be encoded in the name so that wrong uid's are
> > skipped?
>
> But why?
So that you are not filling up the audit logs. There are people that have to
use these audit rules and we need a normally operating system to produce as
few false positives as possible.
Maybe the audit system should be fixed to not trip up by this?
> This stuff should be simple, and it's always a better idea
to
> simply let the kernel do EACCES rather then trying to be smarter than
> the kernel and reimplement access control in userspace.
Well, you are already trusting the name. Who's to say some rougue process
didn't create the file name to try and trick pulseaudio?
Well, if the process did that, then we'd just delete his shared memory
block. I don't feel particularly tricked there... ;-) And besides that,
note that PA also checks a file signature before deleting the file, just
to be sure to not delete anything of importance...
How about doing like some processes do in the /tmp dir...put the
segments in a directory /dev/shm/<user name>/ and then scan all the
files that belong to that user?
Guessable directory names in a world-writable directory? I am sorry,
that's not an option. The stuff is already DoS-able enough, I am pretty
sure I don't want to open the door even wider. (Also what is this,
anyway? of all people, you as a security guy should know what bad an
idea that is...)
There are ways to make this better if you are willing. :-)
Well, or you could make audit better, if you are willing.
Lennart
--
Lennart Poettering - Red Hat, Inc.
11:35 a.m.
On 7 June 2013 10:21, Lennart Poettering <mzerqung(a)0pointer.de> wrote:
On Fri, 07.06.13 12:09, Steve Grubb (sgrubb(a)redhat.com) wrote:
> > > > POSIX shared memory doesn't define any useful scheme for
automatic
> > > > removing of shared memory segments from /dev/shm after use. Hence,
in
> > > > order to make sure that left-over segments don't fill up
/dev/shm
> > > > forever PA will try to GC dead segments from /dev/shm on each
> > > > start-up. For that it iterates through /dev/shm/pulse-shm*, tries
to
> > > > read the PID that is stored in there.
> > >
> > > Maybe the uid can be encoded in the name so that wrong uid's are
> > > skipped?
> >
> > But why?
>
> So that you are not filling up the audit logs. There are people that
have to
> use these audit rules and we need a normally operating system to produce
as
> few false positives as possible.
Maybe the audit system should be fixed to not trip up by this?
The audit system is doing what it is supposed to do. Report that a file is
being tried to be opened with no permissions to do so. That is pretty much
the business rule required to be audited. Exceptions to that rule go up a
long chain of command and usually take various signoffs to make sure that
you aren't trying to get around the system somewhere. This kind of thing
affects everyone who has to deal with auditing in one form or another (PCI,
various .gov, various health agencies etc). They get a rule which is
written as simple as possible and then the audit system is to find all
attempts against that simple rule. And then someone or some group at the
site has to go through each attempt and figure out if the problem is a
legitimate one, a hacker, etc. When they find out it is a legitimate one,
the process is the following:
1) Open a ticket with the vendor that they have a problem.
2) Open a ticket up the chain of command to have an exception allowed.
[Ticket 1 usually takes months to years but is usually faster than ticket
2.]
Then most large sites go through their chain of command tickets and find
out which ones cost the most and then put those vendors on notice that
their products are not meeting expectations and that they want either a
larger discount to cover the hidden costs or they will be finding another
vendor.
--
Stephen J Smoogen.
11:42 a.m.
On Fri, 2013-06-07 at 18:21 +0200, Lennart Poettering wrote:
On Fri, 07.06.13 12:09, Steve Grubb (sgrubb(a)redhat.com) wrote:
> > > > POSIX shared memory doesn't define any useful scheme for
automatic
> > > > removing of shared memory segments from /dev/shm after use. Hence,
in
> > > > order to make sure that left-over segments don't fill up
/dev/shm
> > > > forever PA will try to GC dead segments from /dev/shm on each
> > > > start-up. For that it iterates through /dev/shm/pulse-shm*, tries to
> > > > read the PID that is stored in there.
> > >
> > > Maybe the uid can be encoded in the name so that wrong uid's are
> > > skipped?
> >
> > But why?
>
> So that you are not filling up the audit logs. There are people that have to
> use these audit rules and we need a normally operating system to produce as
> few false positives as possible.
Maybe the audit system should be fixed to not trip up by this?
Lennart,
it's fun to troll, but seriously, the audit system *audits*, it's not
hard to understand.
> > This stuff should be simple, and it's always a better
idea to
> > simply let the kernel do EACCES rather then trying to be smarter than
> > the kernel and reimplement access control in userspace.
>
> Well, you are already trusting the name. Who's to say some rougue process
> didn't create the file name to try and trick pulseaudio?
Well, if the process did that, then we'd just delete his shared memory
block. I don't feel particularly tricked there... ;-) And besides that,
note that PA also checks a file signature before deleting the file, just
to be sure to not delete anything of importance...
> How about doing like some processes do in the /tmp dir...put the
> segments in a directory /dev/shm/<user name>/ and then scan all the
> files that belong to that user?
Guessable directory names in a world-writable directory? I am sorry,
that's not an option. The stuff is already DoS-able enough, I am pretty
sure I don't want to open the door even wider. (Also what is this,
anyway? of all people, you as a security guy should know what bad an
idea that is...)
Sorry but what makes /dev/shm/pulse-shm-3756395503 more/less guessable
than /dev/shm/<uidnumber>/pulse-shm-3756395503 ?
> There are ways to make this better if you are willing. :-)
Well, or you could make audit better, if you are willing.
Or you could actually listen.
Simo.
--
Simo Sorce * Red Hat, Inc * New York
11:55 a.m.
On Fri, 07.06.13 12:42, Simo Sorce (simo(a)redhat.com) wrote:
On Fri, 2013-06-07 at 18:21 +0200, Lennart Poettering wrote:
> On Fri, 07.06.13 12:09, Steve Grubb (sgrubb(a)redhat.com) wrote:
>
> > > > > POSIX shared memory doesn't define any useful scheme for
automatic
> > > > > removing of shared memory segments from /dev/shm after use.
Hence, in
> > > > > order to make sure that left-over segments don't fill up
/dev/shm
> > > > > forever PA will try to GC dead segments from /dev/shm on each
> > > > > start-up. For that it iterates through /dev/shm/pulse-shm*,
tries to
> > > > > read the PID that is stored in there.
> > > >
> > > > Maybe the uid can be encoded in the name so that wrong uid's are
> > > > skipped?
> > >
> > > But why?
> >
> > So that you are not filling up the audit logs. There are people that have to
> > use these audit rules and we need a normally operating system to produce as
> > few false positives as possible.
>
> Maybe the audit system should be fixed to not trip up by this?
Lennart,
it's fun to troll, but seriously, the audit system *audits*, it's not
hard to understand.
Well, you know, this problem isn't new. Some SELinux AVCs can be set to
ignored for precisely reasons like this one, because it is common that
things like these happen: accesses which fail where that is
expected. You can call me a troll as much as you want, but then please
call the Selinux folks trolls first.
> Guessable directory names in a world-writable directory? I am
sorry,
> that's not an option. The stuff is already DoS-able enough, I am pretty
> sure I don't want to open the door even wider. (Also what is this,
> anyway? of all people, you as a security guy should know what bad an
> idea that is...)
Sorry but what makes /dev/shm/pulse-shm-3756395503 more/less guessable
than /dev/shm/<uidnumber>/pulse-shm-3756395503 ?
User "simo" creates /dev/shm/1000/ even though 1000 is the UID of user
"lennart". Lennart can never start PA again, ever. And can't do anything
about it, because "simo" is in control, and /dev/shm is sticky.
It's really disappointing if people as seasoned as you still don't
understand these shared-namespace issues of world-writable directories.
> > There are ways to make this better if you are willing.
:-)
>
> Well, or you could make audit better, if you are willing.
Or you could actually listen.
Well, so far you didn't make a very convincing point, did you?
Lennart
--
Lennart Poettering - Red Hat, Inc.
1:02 p.m.
On Fri, 2013-06-07 at 18:55 +0200, Lennart Poettering wrote:
On Fri, 07.06.13 12:42, Simo Sorce (simo(a)redhat.com) wrote:
> On Fri, 2013-06-07 at 18:21 +0200, Lennart Poettering wrote:
> > On Fri, 07.06.13 12:09, Steve Grubb (sgrubb(a)redhat.com) wrote:
> >
> > > > > > POSIX shared memory doesn't define any useful scheme
for automatic
> > > > > > removing of shared memory segments from /dev/shm after use.
Hence, in
> > > > > > order to make sure that left-over segments don't fill
up /dev/shm
> > > > > > forever PA will try to GC dead segments from /dev/shm on
each
> > > > > > start-up. For that it iterates through /dev/shm/pulse-shm*,
tries to
> > > > > > read the PID that is stored in there.
> > > > >
> > > > > Maybe the uid can be encoded in the name so that wrong uid's
are
> > > > > skipped?
> > > >
> > > > But why?
> > >
> > > So that you are not filling up the audit logs. There are people that have
to
> > > use these audit rules and we need a normally operating system to produce
as
> > > few false positives as possible.
> >
> > Maybe the audit system should be fixed to not trip up by this?
>
> Lennart,
> it's fun to troll, but seriously, the audit system *audits*, it's not
> hard to understand.
Well, you know, this problem isn't new. Some SELinux AVCs can be set to
ignored for precisely reasons like this one, because it is common that
things like these happen: accesses which fail where that is
expected. You can call me a troll as much as you want, but then please
call the Selinux folks trolls first.
The SELinux folks, being the practical bunch they are simply work around
bad programs. That doesn't mean they are trolls.
Also the SeLinux folks have 'context', which is something an audit
system may not have.
Finally the fact SELinux folks are considerate does not mean that
programs need not be fixed to behave better.
Of course the author can always decide to show the finger and do what he
wants anyway, we live in a free and unpolite world after all.
> > Guessable directory names in a world-writable directory? I
am sorry,
> > that's not an option. The stuff is already DoS-able enough, I am pretty
> > sure I don't want to open the door even wider. (Also what is this,
> > anyway? of all people, you as a security guy should know what bad an
> > idea that is...)
>
> Sorry but what makes /dev/shm/pulse-shm-3756395503 more/less guessable
> than /dev/shm/<uidnumber>/pulse-shm-3756395503 ?
User "simo" creates /dev/shm/1000/ even though 1000 is the UID of user
"lennart". Lennart can never start PA again, ever. And can't do anything
about it, because "simo" is in control, and /dev/shm is sticky.
That could be solved by precreating the directory at login time, like we
do for many other things. There isn't just one way.
Or you could even stop using /dev/shm and we'd have one less problem
entirely :)
The point is that we are simply throwing ideas off the wall as an aid in
finding a way to solve the issue for all.
It's really disappointing if people as seasoned as you still
don't
understand these shared-namespace issues of world-writable directories.
Yeah, just like it is disappointing to see that your first reaction is
always to show the finger hard and solid instead of trying to have a
reasonable discussion about the problem.
> > > There are ways to make this better if you are willing.
:-)
> >
> > Well, or you could make audit better, if you are willing.
>
> Or you could actually listen.
Well, so far you didn't make a very convincing point, did you?
That is entirely possible, but it doesn't look like you are making one
either.
Steve asked a pretty simple question, based on clear premises.
You could have simply explained why you can't change it or solicited
ideas, instead you attacked him and replied as if Steve had accused you
of something awful. And of course you extend the same courtesy to anyone
else who dares offering some ideas. This is not the way to discuss
technical problems.
Simo.
--
Simo Sorce * Red Hat, Inc * New York
1:29 p.m.
On Fri, Jun 07, 2013 at 02:02:14PM -0400, Simo Sorce wrote:
The point is that we are simply throwing ideas off the wall as an aid
in
finding a way to solve the issue for all.
So why not add a mechanism to permit applications to indicate that
certain accesses they make should be ignored by audit?
--
Matthew Garrett | mjg59(a)srcf.ucam.org
1:38 p.m.
On Fri, Jun 7, 2013 at 8:29 PM, Matthew Garrett <mjg59(a)srcf.ucam.org> wrote:
So why not add a mechanism to permit applications to indicate that
certain accesses they make should be ignored by audit?
Because it would be primarily useful to the attackers' applications.
Or am I missing something? (BTW, audit already has something like
"dontaudit" rules. But it has limited information to work with.)
Mirek
1:52 p.m.
On Fri, Jun 07, 2013 at 08:38:56PM +0200, Miloslav Trmač wrote:
On Fri, Jun 7, 2013 at 8:29 PM, Matthew Garrett
<mjg59(a)srcf.ucam.org> wrote:
> So why not add a mechanism to permit applications to indicate that
> certain accesses they make should be ignored by audit?
Because it would be primarily useful to the attackers' applications.
Or am I missing something? (BTW, audit already has something like
"dontaudit" rules. But it has limited information to work with.)
If the attacker has root then the attacker can just change the file
permissions anyway?
--
Matthew Garrett | mjg59(a)srcf.ucam.org
2:35 p.m.
On Friday, June 07, 2013 07:29:56 PM Matthew Garrett wrote:
On Fri, Jun 07, 2013 at 02:02:14PM -0400, Simo Sorce wrote:
> The point is that we are simply throwing ideas off the wall as an aid in
> finding a way to solve the issue for all.
So why not add a mechanism to permit applications to indicate that
certain accesses they make should be ignored by audit?
We've never needed an exception in the past. What I'm reporting is there is
now a trend on the rise where apps are trying to open files that do not belong
to them or open them not wanting the access time updated which attempts to
bypass forensic time stamps.
So far, the discussion has focused on pulseaudio. But what about the O_NOATIME
issue? I wrote an article [1] for the hack in the box magazine a while back
about using the audit system to look for application problems across the whole
distribution at once. Its good at doing that. And like SE Linux, sometimes the
fix is not to avoid auditing bad behaving apps, but to fix them.
As for the O_NOATIME...cinnamon is the prime offender and neither it nor muffin
have O_NOATIME anywhere in the code. So, its coming from a library. Anyone
have any ideas? If we can fix that one at least we can make some progress.
Thanks,
-Steve
[1] - http://magazine.hitb.org/issues/HITB-Ezine-Issue-005.pdf
2:42 p.m.
On Fri, Jun 07, 2013 at 03:35:28PM -0400, Steve Grubb wrote:
So far, the discussion has focused on pulseaudio. But what about the
O_NOATIME
issue?
Without further analysis, it doesn't tell us much. Does the code attempt
to open a file O_NOATIME and then fall back to trying it without?
--
Matthew Garrett | mjg59(a)srcf.ucam.org
3:05 p.m.
On Fri, 2013-06-07 at 20:42 +0100, Matthew Garrett wrote:
Without further analysis, it doesn't tell us much. Does the code
attempt
to open a file O_NOATIME and then fall back to trying it without?
It's likely:
https://bugzilla.gnome.org/show_bug.cgi?id=680326
Code:
https://git.gnome.org/browse/libgsystem/tree/gsystem-file-utils.c#n67
3:14 p.m.
On Fri, Jun 7, 2013 at 10:05 PM, Colin Walters <walters(a)verbum.org> wrote:
On Fri, 2013-06-07 at 20:42 +0100, Matthew Garrett wrote:
> Without further analysis, it doesn't tell us much. Does the code attempt
> to open a file O_NOATIME and then fall back to trying it without?
It's likely:
https://bugzilla.gnome.org/show_bug.cgi?id=680326
Is there any more rationale for the change available?
(IMHO only very special applications should use O_NOATIME; if it is
not predictable which accesses do/don't update atime, the field
completely loses its value. The definition we have is "time of last
access" (see (man stat.h) for the POSIX wording), not "time of last
user-initiated access" or "time used for $specific_purpose"; given the
lack of $specific_purpose it's difficult to say what accesses
should/should not be excluded but that's, I think, even more of a
reason to stick to the formal definition. OTOH mlocate does use
O_NOATIME so that atime isn't updated for every modified directory,
and the line in the sand between updatedb and thumbnail generation
admittedly isn't all that clear.)
Mirek
3:57 p.m.
On Fri, Jun 07, 2013 at 10:14:36PM +0200, Miloslav Trmač wrote:
(IMHO only very special applications should use O_NOATIME; if it is
not predictable which accesses do/don't update atime, the field
completely loses its value.
It's already not especially predictable - we've been using relatime
rather than strict atime for years.
--
Matthew Garrett | mjg59(a)srcf.ucam.org
4:02 p.m.
On Fri, 2013-06-07 at 22:14 +0200, Miloslav Trmač wrote:
On Fri, Jun 7, 2013 at 10:05 PM, Colin Walters
<walters(a)verbum.org> wrote:
> On Fri, 2013-06-07 at 20:42 +0100, Matthew Garrett wrote:
>
>> Without further analysis, it doesn't tell us much. Does the code attempt
>> to open a file O_NOATIME and then fall back to trying it without?
>
> It's likely:
> https://bugzilla.gnome.org/show_bug.cgi?id=680326
Is there any more rationale for the change available?
http://lwn.net/Articles/244829/
and the line in the sand between updatedb and thumbnail generation
admittedly isn't all that clear.)
Right.
I guess we really want O_NOATIME_IF_POSSIBLE. Although I wonder if
there's any reason not to just silently ignore O_NOATIME if it's not
permitted, rather than force apps to double open().
4:24 p.m.
On Friday, June 07, 2013 05:02:41 PM Colin Walters wrote:
On Fri, 2013-06-07 at 22:14 +0200, Miloslav Trmač wrote:
> On Fri, Jun 7, 2013 at 10:05 PM, Colin Walters <walters(a)verbum.org> wrote:
> > On Fri, 2013-06-07 at 20:42 +0100, Matthew Garrett wrote:
> >> Without further analysis, it doesn't tell us much. Does the code
> >> attempt
> >> to open a file O_NOATIME and then fall back to trying it without?
> >
> > It's likely:
> > https://bugzilla.gnome.org/show_bug.cgi?id=680326
>
> Is there any more rationale for the change available?
http://lwn.net/Articles/244829/
> and the line in the sand between updatedb and thumbnail generation
> admittedly isn't all that clear.)
Right.
I guess we really want O_NOATIME_IF_POSSIBLE. Although I wonder if
there's any reason not to just silently ignore O_NOATIME if it's not
permitted, rather than force apps to double open().
Hmm...sounds like kernel change. But in the meantime, most of the offenders I
see seem to have something to do with loading icons:
name=/usr/share/icons/gnome/16x16/mimetypes/x-office-document.png 30
name=/etc/pki/tls/certs/localhost.crt 25
name=/usr/share/icons/hicolor/48x48/apps/firefox.png 11
name=/usr/share/icons/hicolor/22x22/apps/firefox.png 10
name=/usr/share/icons/hicolor/24x24/apps/firefox.png 9
name=/usr/share/icons/hicolor/32x32/apps/firefox.png 9
name=/usr/share/icons/gnome/16x16/actions/window-close.png 9
name=/usr/share/icons/hicolor/256x256/apps/firefox.png 8
name=/usr/share/icons/hicolor/16x16/apps/firefox.png 8
name=/usr/share/icons/gnome/24x24/actions/go-up.png 7
name=/usr/share/icons/gnome/24x24/actions/go-down.png 7
name=/usr/share/icons/gnome/24x24/places/folder.png 7
I think we can assume root owns the icons which mean calls with NOATIME as a
normal user session will fail. Maybe the call that opens the icon can be
switched to a regular open?
Thanks,
Steve
12:36 a.m.
On Fri, Jun 07, 2013 at 05:24:30PM -0400, Steve Grubb wrote:
Hmm...sounds like kernel change. But in the meantime, most of the
offenders I
see seem to have something to do with loading icons:
Sounds like code that doesn't differentiate between files that are in
user-local directories and system-global directories. That's something
that could presumably be fixed, but it seems like a bunch of effort.
Other than a heuristic based on whether a path is in the user's home
directory or not, the only way to avoid this is to stat before opening -
and that's obviously prone to failure.
--
Matthew Garrett | mjg59(a)srcf.ucam.org
8:34 a.m.
On Saturday, June 08, 2013 06:36:38 AM Matthew Garrett wrote:
On Fri, Jun 07, 2013 at 05:24:30PM -0400, Steve Grubb wrote:
> Hmm...sounds like kernel change. But in the meantime, most of the
> offenders I see seem to have something to do with loading icons
Sounds like code that doesn't differentiate between files that are in
user-local directories and system-global directories. That's something
that could presumably be fixed, but it seems like a bunch of effort.
From the code I saw previously, it seems like just changing the
function being
called to the variant without noatime in its name. The comments in
the code
Colin pointed to say this:
* gs_file_read_noatime:
* @file: a #GFile
* @cancellable: a #GCancellable
* @error: a #GError
*
* Like g_file_read(), but try to avoid updating the file's
* access time. This should be used by background scanning
* components such as search indexers, antivirus programs, etc.
And evince, firefox, or openoffice are not any of those ^^. :-)
Other than a heuristic based on whether a path is in the user's
home
directory or not, the only way to avoid this is to stat before opening -
and that's obviously prone to failure.
Does opening with noatime really make a measurable difference (assuming it
worked)? I suspect not since what we have now is 2 syscalls. It would probably
be faster to load icons without trying to set NOATIME.
Thanks,
-Steve
8:49 a.m.
On Sat, Jun 08, 2013 at 09:34:22AM -0400, Steve Grubb wrote:
> Other than a heuristic based on whether a path is in the
user's home
> directory or not, the only way to avoid this is to stat before opening -
> and that's obviously prone to failure.
Does opening with noatime really make a measurable difference (assuming it
worked)? I suspect not since what we have now is 2 syscalls. It would probably
be faster to load icons without trying to set NOATIME.
PLUS, we are mounting filesystems with relatime anyway.
--
Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ <mattdm(a)fedoraproject.org>
8:53 a.m.
On Saturday, June 08, 2013 09:34:22 AM Steve Grubb wrote:
Does opening with noatime really make a measurable difference
(assuming it
worked)? I suspect not since what we have now is 2 syscalls. It would
probably be faster to load icons without trying to set NOATIME.
Answering my own question....
#include <stdio.h>
#define __USE_GNU
#include <fcntl.h>
#include <unistd.h>
#include <string.h>
void noatime(void)
{
int fd = open("/usr/share/icons/hicolor/48x48/apps/firefox.png",
O_RDONLY|O_NOATIME);
if (fd>=0)
close(fd);
else {
fd = open("/usr/share/icons/hicolor/48x48/apps/firefox.png",
O_RDONLY);
if (fd>=0)
close(fd);
}
}
void atime(void)
{
int fd = open("/usr/share/icons/hicolor/48x48/apps/firefox.png",
O_RDONLY);
if (fd>=0)
close(fd);
}
int main(int argc, char *argv[])
{
int i, mode=0;
if (argc == 2 && strcmp("noatime", argv[1]) == 0)
mode = 1;
for (i=0; i<5000; i++) {
if (mode)
noatime();
else
atime();
}
return 0;
}
As a normal user:
$ time ./test noatime
real 0m12.645s
user 0m0.003s
sys 0m0.159s
$ time ./test atime
real 0m0.019s
user 0m0.002s
sys 0m0.016s
Way faster doing a normal open. As root:
# time ./test noatime
real 0m0.019s
user 0m0.000s
sys 0m0.019s
# time ./test atime
real 0m0.019s
user 0m0.001s
sys 0m0.017s
No real difference between them.
-Steve
8:57 a.m.
On 06/08/2013 09:53 AM, Steve Grubb wrote:
On Saturday, June 08, 2013 09:34:22 AM Steve Grubb wrote:
> Does opening with noatime really make a measurable difference (assuming it
> worked)? I suspect not since what we have now is 2 syscalls. It would
> probably be faster to load icons without trying to set NOATIME.
Answering my own question....
#include <stdio.h>
#define __USE_GNU
#include <fcntl.h>
#include <unistd.h>
#include <string.h>
void noatime(void)
{
int fd = open("/usr/share/icons/hicolor/48x48/apps/firefox.png",
O_RDONLY|O_NOATIME);
if (fd>=0)
close(fd);
else {
fd = open("/usr/share/icons/hicolor/48x48/apps/firefox.png",
O_RDONLY);
if (fd>=0)
close(fd);
}
}
void atime(void)
{
int fd = open("/usr/share/icons/hicolor/48x48/apps/firefox.png",
O_RDONLY);
if (fd>=0)
close(fd);
}
int main(int argc, char *argv[])
{
int i, mode=0;
if (argc == 2 && strcmp("noatime", argv[1]) == 0)
mode = 1;
for (i=0; i<5000; i++) {
if (mode)
noatime();
else
atime();
}
return 0;
}
As a normal user:
$ time ./test noatime
real 0m12.645s
user 0m0.003s
sys 0m0.159s
$ time ./test atime
real 0m0.019s
user 0m0.002s
sys 0m0.016s
Way faster doing a normal open. As root:
Bad test. The first run took the hit for getting the file info into
page cache, after that, everything was run from cache and you got the
second result above and the results below. You have to make sure that
from run to run the cache state of the file in question is identical.
# time ./test noatime
real 0m0.019s
user 0m0.000s
sys 0m0.019s
# time ./test atime
real 0m0.019s
user 0m0.001s
sys 0m0.017s
No real difference between them.
-Steve
9:10 a.m.
On Saturday, June 08, 2013 09:57:03 AM Doug Ledford wrote:
Bad test. The first run took the hit for getting the file info into
page cache, after that, everything was run from cache and you got the
second result above and the results below. You have to make sure that
from run to run the cache state of the file in question is identical.
Try it yourself. :-) I know what you are saying and run the test probably 8
times before posting results. I also have the audit rule loaded...so removing
it:
[sgrubb@x2 noatime]$ time ./test noatime
real 0m0.031s
user 0m0.006s
sys 0m0.024s
[sgrubb@x2 noatime]$ time ./test noatime
real 0m0.033s
user 0m0.002s
sys 0m0.032s
[sgrubb@x2 noatime]$ time ./test noatime
real 0m0.036s
user 0m0.002s
sys 0m0.031s
[sgrubb@x2 noatime]$ time ./test atime
real 0m0.023s
user 0m0.001s
sys 0m0.021s
[sgrubb@x2 noatime]$ time ./test atime
real 0m0.022s
user 0m0.003s
sys 0m0.019s
[sgrubb@x2 noatime]$ time ./test atime
real 0m0.023s
user 0m0.002s
sys 0m0.019s
Without the audit rules, it is faster. But again opening with noatime
attempted is measurably slower.
-Steve
9:13 a.m.
On 06/08/2013 10:10 AM, Steve Grubb wrote:
On Saturday, June 08, 2013 09:57:03 AM Doug Ledford wrote:
> Bad test. The first run took the hit for getting the file info into
> page cache, after that, everything was run from cache and you got the
> second result above and the results below. You have to make sure that
> from run to run the cache state of the file in question is identical.
Try it yourself. :-) I know what you are saying and run the test probably 8
times before posting results. I also have the audit rule loaded...so removing
it:
[sgrubb@x2 noatime]$ time ./test noatime
real 0m0.031s
user 0m0.006s
sys 0m0.024s
[sgrubb@x2 noatime]$ time ./test noatime
real 0m0.033s
user 0m0.002s
sys 0m0.032s
[sgrubb@x2 noatime]$ time ./test noatime
real 0m0.036s
user 0m0.002s
sys 0m0.031s
[sgrubb@x2 noatime]$ time ./test atime
real 0m0.023s
user 0m0.001s
sys 0m0.021s
[sgrubb@x2 noatime]$ time ./test atime
real 0m0.022s
user 0m0.003s
sys 0m0.019s
[sgrubb@x2 noatime]$ time ./test atime
real 0m0.023s
user 0m0.002s
sys 0m0.019s
Without the audit rules, it is faster. But again opening with noatime
attempted is measurably slower.
Yes, but none of these results show the .12s time that your first
noatime test run showed in your original post. If you are now saying
that atime is faster than noatime by about .005 to .010s, then these
results seem to show that. But your original post was from .019 to .12,
or a difference of .10+s. That was cache load time, not just the
syscall difference.
9:29 a.m.
On Saturday, June 08, 2013 10:13:45 AM Doug Ledford wrote:
Yes, but none of these results show the .12s time that your first
noatime test run showed in your original post. If you are now saying
that atime is faster than noatime by about .005 to .010s, then these
results seem to show that. But your original post was from .019 to .12,
or a difference of .10+s. That was cache load time, not just the
syscall difference.
I chalk that up to the audit system. The audit system tries real hard to stay
out of the way since the vast majority of syscalls are not interesting. But if
you trigger an event, it has to get recorded in gory detail and that takes
time. (The first run did trigger 5000 audit events, the others didn't.) This is
another reason (but not the main reason) we need to try to avoid triggering
events in a normally operating machine.
-Steve
7:24 p.m.
On 06/08/2013 10:29 AM, Steve Grubb wrote:
On Saturday, June 08, 2013 10:13:45 AM Doug Ledford wrote:
> Yes, but none of these results show the .12s time that your first
> noatime test run showed in your original post. If you are now saying
> that atime is faster than noatime by about .005 to .010s, then these
> results seem to show that. But your original post was from .019 to .12,
> or a difference of .10+s. That was cache load time, not just the
> syscall difference.
I chalk that up to the audit system. The audit system tries real hard to stay
out of the way since the vast majority of syscalls are not interesting. But if
you trigger an event, it has to get recorded in gory detail and that takes
time. (The first run did trigger 5000 audit events, the others didn't.) This is
another reason (but not the main reason) we need to try to avoid triggering
events in a normally operating machine.
OK, that makes more sense. So the audit events on a 5000 loop take up
the .10 seconds in time difference. Fine, I'll grant that this
legitimately belongs in the time difference then.
8:53 a.m.
On 06/08/2013 04:13 PM, Doug Ledford wrote:
Yes, but none of these results show the .12s time that your first
noatime test run showed in your original post. If you are now saying
that atime is faster than noatime by about .005 to .010s, then these
results seem to show that. But your original post was from .019 to .12,
or a difference of .10+s. That was cache load time, not just the
syscall difference.
Hmm, someone is misreading the results.
I've reread multiple times, and I see a difference of 12s, not .12s.
---> real 0m12.645s
---> user 0m0.003s
---> sys 0m0.159s
And 12 seconds (elapsed, with 0.159s system) means 12s/5000=2.4ms
which could only be explained with the auditing system doing fsync
calls on its log files.
--
Roberto Ragusa mail at robertoragusa.it
9:40 a.m.
On 06/09/2013 09:53 AM, Roberto Ragusa wrote:
On 06/08/2013 04:13 PM, Doug Ledford wrote:
>
> Yes, but none of these results show the .12s time that your first
> noatime test run showed in your original post. If you are now saying
> that atime is faster than noatime by about .005 to .010s, then these
> results seem to show that. But your original post was from .019 to .12,
> or a difference of .10+s. That was cache load time, not just the
> syscall difference.
Hmm, someone is misreading the results.
I've reread multiple times, and I see a difference of 12s, not .12s.
---> real 0m12.645s
---> user 0m0.003s
---> sys 0m0.159s
And 12 seconds (elapsed, with 0.159s system) means 12s/5000=2.4ms
which could only be explained with the auditing system doing fsync
calls on its log files.
You're right, I was going from memory and I did not remember it being 12
seconds, I thought it was .12 seconds. This most certainly is not just
the effects of loading cache. A 12 second cache load would only happen
if you are loading from stone tablet...
3:06 p.m.
On Friday, June 07, 2013 08:42:09 PM Matthew Garrett wrote:
On Fri, Jun 07, 2013 at 03:35:28PM -0400, Steve Grubb wrote:
> So far, the discussion has focused on pulseaudio. But what about the
> O_NOATIME issue?
Without further analysis, it doesn't tell us much. Does the code attempt
to open a file O_NOATIME and then fall back to trying it without?
It would appear so:
open("/usr/share/icons/gnome/48x48/status/dialog-password.png", O_RDONLY|
O_NOATIME) = -1 EPERM (Operation not permitted)
open("/usr/share/icons/gnome/48x48/status/dialog-password.png", O_RDONLY) = 12
read(12,
"\211PNG\r\n\32\n\0\0\0\rIHDR\0\0\0000\0\0\0000\10\6\0\0\0W\2\371"..., 4096) =
2083
close(12) = 0
Which is a bad patterm. O_NOATIME requires CAP_FOWNER and I don't think
graphic programs are supposed to run as root/privileged. So, there seems to be
a misunderstanding of what O_NOATIME is for. It seems to be related to loading
icons. Is there a common library for that?
-Steve
3:19 p.m.
On 6/7/13 3:06 PM, Steve Grubb wrote:
On Friday, June 07, 2013 08:42:09 PM Matthew Garrett wrote:
> On Fri, Jun 07, 2013 at 03:35:28PM -0400, Steve Grubb wrote:
>> So far, the discussion has focused on pulseaudio. But what about the
>> O_NOATIME issue?
>
> Without further analysis, it doesn't tell us much. Does the code attempt
> to open a file O_NOATIME and then fall back to trying it without?
It would appear so:
open("/usr/share/icons/gnome/48x48/status/dialog-password.png", O_RDONLY|
O_NOATIME) = -1 EPERM (Operation not permitted)
open("/usr/share/icons/gnome/48x48/status/dialog-password.png", O_RDONLY) = 12
read(12,
"\211PNG\r\n\32\n\0\0\0\rIHDR\0\0\0000\0\0\0000\10\6\0\0\0W\2\371"..., 4096) =
2083
close(12) = 0
Which is a bad patterm. O_NOATIME requires CAP_FOWNER and I don't think
graphic programs are supposed to run as root/privileged. So, there seems to be
a misunderstanding of what O_NOATIME is for. It seems to be related to loading
icons. Is there a common library for that?
-Steve
This maybe?
http://permalink.gmane.org/gmane.comp.gnome.svn/621716
commit 4d2f77074802ac8b2d05f7cdfecef9bebfba2647
Author: Colin Walters <walters <at> verbum.org>
Date: Mon Aug 27 16:07:39 2012 -0400
core: Use O_NOATIME to open metadata
We really don't need atime for metadata, it's just a speed hit.
<it does try, and fall back>
-Eric
4:03 p.m.
On Fri, Jun 07, 2013 at 04:06:30PM -0400, Steve Grubb wrote:
Which is a bad patterm. O_NOATIME requires CAP_FOWNER
Documentation disagrees:
EPERM The O_NOATIME flag was specified, but the effective user ID of
the caller did not match the owner of the file and the caller
was not privileged (CAP_FOWNER).
--
Matthew Garrett | mjg59(a)srcf.ucam.org
10:07 a.m.
On Fri, 07.06.13 15:35, Steve Grubb (sgrubb(a)redhat.com) wrote:
On Friday, June 07, 2013 07:29:56 PM Matthew Garrett wrote:
> On Fri, Jun 07, 2013 at 02:02:14PM -0400, Simo Sorce wrote:
> > The point is that we are simply throwing ideas off the wall as an aid in
> > finding a way to solve the issue for all.
>
> So why not add a mechanism to permit applications to indicate that
> certain accesses they make should be ignored by audit?
We've never needed an exception in the past. What I'm reporting is there is
now a trend on the rise where apps are trying to open files that do not belong
to them or open them not wanting the access time updated which attempts to
bypass forensic time stamps.
This is hardly a "new trend" btw. PA has been doing this since about
forever and has been default since Fedora 8. Which is more than 5 years
ago.
Lennart
--
Lennart Poettering - Red Hat, Inc.
8:03 p.m.
On 7 June 2013 12:29, Matthew Garrett <mjg59(a)srcf.ucam.org> wrote:
On Fri, Jun 07, 2013 at 02:02:14PM -0400, Simo Sorce wrote:
> The point is that we are simply throwing ideas off the wall as an aid in
> finding a way to solve the issue for all.
So why not add a mechanism to permit applications to indicate that
certain accesses they make should be ignored by audit?
Just so people know, this is like one of the the oldest auditing argument
in the world. I have had programmers say that since the 1990's. [The
standard counter story is that user program X says "don't audit anything I
do in /etc." The programmer counters with adding in a black list of
directories that can't be audited, this gets countered by something else
and eventually you have a process where programs that have a GPG signature
that has been accepted as valid by the audit program can say which of the
white listed files it wants opened without audit are dealt with... and then
some other programmer comes in and shows the 20,000 lines of need to be
audited code replaces 40 lines of C code in the programs that were causing
the problem.]
The problem is that the issue is a social one and not a technical one which
is why i think there is so much hostility towards auditing. You can't fix
it with a technical fix, you have to fix it via social methods and a lot of
time. In this case, the general rule is "Audit all failed accesses."
Programs and methods which allow for automatic getting around that get
rejected by higher ups ( I have seen several teams fight that mountain over
the years).
Instead what the higher ups want is that the site knows what is causing
problems, why it is causing problems and only then and it has been proven
by code audit that it can't get around it can you add a line in an
/etc/audit that accesses to this directory are not to be audited. If you
are lucky and working in a .gov/.mil setting that might take 6 months. If
you are unlucky and working at a doctor's office (HIPPAA) or a store with
credit cards (PCI) it can take a lot of extra money or time to get around
this.
The worst audit I dealt with was with a hospitals cafeteria which had to be
closed for one day and go cash-only for a couple other until the audit
rules were in place that met the audit teams specifications. The next year
another audit team required other changes and a removal of various items of
the first. All of it social items that the sys admins and coders involved
had no technical fix for.
--
Matthew Garrett | mjg59(a)srcf.ucam.org
--
devel mailing list
devel(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
--
Stephen J Smoogen.
12:33 a.m.
On Fri, Jun 07, 2013 at 07:03:24PM -0600, Stephen John Smoogen wrote:
On 7 June 2013 12:29, Matthew Garrett <mjg59(a)srcf.ucam.org>
wrote:
> So why not add a mechanism to permit applications to indicate that
> certain accesses they make should be ignored by audit?
>
Just so people know, this is like one of the the oldest auditing argument
in the world. I have had programmers say that since the 1990's. [The
standard counter story is that user program X says "don't audit anything I
do in /etc." The programmer counters with adding in a black list of
directories that can't be audited, this gets countered by something else
and eventually you have a process where programs that have a GPG signature
that has been accepted as valid by the audit program can say which of the
white listed files it wants opened without audit are dealt with... and then
some other programmer comes in and shows the 20,000 lines of need to be
audited code replaces 40 lines of C code in the programs that were causing
the problem.]
Well, http://www.stigviewer.com/check/V-29067 implies that filtering of
audit records is a reasonable thing to do. I have no expectation that
arbitrary user applications should be able to do whatever they want
without leaving an audit trail, but I don't see what that has to do with
system applications. Root has the ability to modify the selinux policy,
so root (and packages installed by root) should have the ability to
modify the set of behaviours that trigger audit records.
The problem is that the issue is a social one and not a technical one
which
is why i think there is so much hostility towards auditing. You can't fix
it with a technical fix, you have to fix it via social methods and a lot of
time. In this case, the general rule is "Audit all failed accesses."
Programs and methods which allow for automatic getting around that get
rejected by higher ups ( I have seen several teams fight that mountain over
the years).
POSIX doesn't provide a way for you to guarantee that you can open a
file other than actually attempting to open that file. That's not a
social problem. That's a technical limitation of the OS that we develop.
If our existing audit system can't cope with that then that's a problem.
--
Matthew Garrett | mjg59(a)srcf.ucam.org
8:25 a.m.
On Saturday, June 08, 2013 06:33:11 AM Matthew Garrett wrote:
On Fri, Jun 07, 2013 at 07:03:24PM -0600, Stephen John Smoogen
wrote:
> On 7 June 2013 12:29, Matthew Garrett <mjg59(a)srcf.ucam.org> wrote:
> > So why not add a mechanism to permit applications to indicate that
> > certain accesses they make should be ignored by audit?
>
> Just so people know, this is like one of the the oldest auditing argument
> in the world. I have had programmers say that since the 1990's. [The
> standard counter story is that user program X says "don't audit anything I
> do in /etc." The programmer counters with adding in a black list of
> directories that can't be audited, this gets countered by something else
> and eventually you have a process where programs that have a GPG signature
> that has been accepted as valid by the audit program can say which of the
> white listed files it wants opened without audit are dealt with... and
> then
> some other programmer comes in and shows the 20,000 lines of need to be
> audited code replaces 40 lines of C code in the programs that were causing
> the problem.]
Well, http://www.stigviewer.com/check/V-29067 implies that filtering of
audit records is a reasonable thing to do.
What this requirement is talking about is that we must provide something like
ausearch. OK, we do that. What I am telling you is that the OS has changed in
a bad way in the last year or two. It has _never_ been this noisy for
auditing. Look at this:
# aureport --start this-week --key --summary
Key Summary Report
===========================
total key
===========================
73520 access
562 module-load
149 module-unload
135 bypass
132 system-locale
132 container-config
113 time-change
110 identity
100 data-injection
88 container-create
88 export
58 register-injection
44 code-injection
29 power-abuse
22 modules-del
22 modules-add
22 MAC-policy
The bad access events dominate the event log.
I have no expectation that arbitrary user applications should be able
to do
whatever they want without leaving an audit trail, but I don't see what that
has to do with system applications. Root has the ability to modify the
selinux policy, so root (and packages installed by root) should have the
ability to modify the set of behaviours that trigger audit records.
Its not quite like this. What I need is the OS to be well behaved under normal
conditions so that when problems come along they are easily spotted. Fedora
has been a fairly well behaved OS over the years. I have had to get a few apps
fixed in the past and the maintainers have always been accommodating. But this
time I am finding we have a serious problem worse than in the past.
Thanks,
-Steve
1:35 p.m.
On Sat, 2013-06-08 at 09:25 -0400, Steve Grubb wrote:
Its not quite like this. What I need is the OS to be well behaved
under normal
conditions so that when problems come along they are easily spotted. Fedora
has been a fairly well behaved OS over the years. I have had to get a few apps
fixed in the past and the maintainers have always been accommodating. But this
time I am finding we have a serious problem worse than in the past.
Well, you're defining something as 'bad behaviour' fairly arbitrarily -
or at least controversially: not everyone agrees with your definition.
Continuing to simply assert that the behaviour is bad is not driving the
conversation forward, you're just repeating a position that others have
already raised objections to. Those who are disputing your position are
not saying 'this behaviour is not happening', they are saying 'we
disagree with your definition of "bad behaviour"'.
If it's not 'bad behaviour', the fact that it didn't happen before is
fairly irrelevant. I could come up with any arbitrary 'test' for some
action that Fedora 19 does that Fedora 18 does not; that doesn't mean I
can then show up on the list waving my test results about and declaring
that there's a problem. First there has to be solid agreement that I'm
actually testing for something we shouldn't be doing.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
7:28 p.m.
On 06/08/2013 02:35 PM, Adam Williamson wrote:
On Sat, 2013-06-08 at 09:25 -0400, Steve Grubb wrote:
> Its not quite like this. What I need is the OS to be well behaved under normal
> conditions so that when problems come along they are easily spotted. Fedora
> has been a fairly well behaved OS over the years. I have had to get a few apps
> fixed in the past and the maintainers have always been accommodating. But this
> time I am finding we have a serious problem worse than in the past.
Well, you're defining something as 'bad behaviour' fairly arbitrarily -
or at least controversially: not everyone agrees with your definition.
Speaking as a former sysadmin responsible for intrusion detection, this
is not a controversial definition at all (namely that anything that
creates audit events without a reasonably just cause is 'bad behavior').
It is the only sane definition of 'bad behavior'. Anything that makes
an admin go chasing ghosts for no good reason is most definitely 'bad
behavior', and every single audit event on a system must be identifiable
by the admins before you know your system is secure.
Continuing to simply assert that the behaviour is bad is not driving
the
conversation forward, you're just repeating a position that others have
already raised objections to. Those who are disputing your position are
not saying 'this behaviour is not happening', they are saying 'we
disagree with your definition of "bad behaviour"'.
If it's not 'bad behaviour', the fact that it didn't happen before is
fairly irrelevant. I could come up with any arbitrary 'test' for some
action that Fedora 19 does that Fedora 18 does not; that doesn't mean I
can then show up on the list waving my test results about and declaring
that there's a problem. First there has to be solid agreement that I'm
actually testing for something we shouldn't be doing.
11:56 p.m.
On Sat, Jun 08, 2013 at 08:28:48PM -0400, Doug Ledford wrote:
On 06/08/2013 02:35 PM, Adam Williamson wrote:
> Well, you're defining something as 'bad behaviour' fairly arbitrarily -
> or at least controversially: not everyone agrees with your definition.
Speaking as a former sysadmin responsible for intrusion detection, this
is not a controversial definition at all (namely that anything that
creates audit events without a reasonably just cause is 'bad behavior').
It is the only sane definition of 'bad behavior'. Anything that makes
an admin go chasing ghosts for no good reason is most definitely 'bad
behavior', and every single audit event on a system must be identifiable
by the admins before you know your system is secure.
I don't think anyone wants these accesses to generate audit records. The
question is whether the right way to fix that is to avoid those accesses
in the first place or to provide a mechanism so that legitimate accesses
don't generate audit records.
--
Matthew Garrett | mjg59(a)srcf.ucam.org
9:03 a.m.
On Sunday, June 09, 2013 05:56:42 AM Matthew Garrett wrote:
On Sat, Jun 08, 2013 at 08:28:48PM -0400, Doug Ledford wrote:
> On 06/08/2013 02:35 PM, Adam Williamson wrote:
> > Well, you're defining something as 'bad behaviour' fairly
arbitrarily -
> > or at least controversially: not everyone agrees with your definition.
>
> Speaking as a former sysadmin responsible for intrusion detection, this
> is not a controversial definition at all (namely that anything that
> creates audit events without a reasonably just cause is 'bad behavior').
> It is the only sane definition of 'bad behavior'. Anything that makes
> an admin go chasing ghosts for no good reason is most definitely 'bad
> behavior', and every single audit event on a system must be identifiable
> by the admins before you know your system is secure.
Thanks Doug. I don't think I could have said it better myself.
These events also consume space. Each one of these occupy about 600 bytes. But
when you multiply that by the number I gave you yesterday for a week of these
events, it gets large. Because I work on the audit system, I have a 100Mb
dedicated partition for audit logs. Until recently, that would hold about a
year's worth of events. The audit rules have not changed in maybe 4 years.
When I run aureport --log today, my logs only go back to May 30 of this year.
That's how big of an impact this is. As to the reason why I didn't bring this
up earlier, upstream broke the audit system on the 3.7 kernel and it wasn't
fixed until the 3.9.4 kernel.
I don't think anyone wants these accesses to generate audit
records. The
question is whether the right way to fix that is to avoid those accesses
in the first place or to provide a mechanism so that legitimate accesses
don't generate audit records.
There isn't a mechanism to allow these to slip through. Over the years I have
come to realize that the audit system can be a great resource for debugging
user space. It was sitting through one of Dave Jones' why userspace sucks
lectures and afterwards pouring through audit logs that I saw that we can find
some of these problems. If part of the goals when writing software is
correctness and efficiency, then wouldn't failing syscalls be of interest? Not
just in the case of EPERM, but also for example EINVAL?
Why would anyone write software that is incorrect enough the OS spits it back
as EINVAL? I think its because they didn't know and in some cases it was
written on another OS that accepts that particular input. To that point, you
can load these two rules:
auditctl -a exit,never -F arch=b64 -S rt_sigreturn
auditctl -a exit,always -S all -F exit=-EINVAL -k einval-test
and then use the computer for a while. To see what you get:
# ausearch --start today -k einval -m SYSCALL --raw | aureport -x --summary
Executable Summary Report
=================================
total file
=================================
165 /usr/libexec/mysqld
149 /usr/bin/man
105 /usr/lib/udisks2/udisksd
74 /usr/sbin/libvirtd
16 /usr/lib/systemd/systemd-udevd
8 /usr/lib64/firefox/firefox
5 /usr/bin/python2.7
2 /usr/sbin/modem-manager
2 /usr/sbin/NetworkManager
2 /usr/bin/gnome-terminal
1 /usr/bin/qemu-kvm
1 /usr/bin/knotify4
1 /usr/bin/xchat
Who knew all these programs were making invalid syscalls? Which syscalls are
we having a problem with?
# ausearch --start today -k einval -m SYSCALL --raw | aureport -i --syscall --
summary
Syscall Summary Report
==========================
total syscall
==========================
320 readlink
149 munmap
74 rt_sigaction
9 ioctl
I'll leave it here for anyone curious enough to dig out the details of how
each syscall is wrong. But its my belief that these are not intentionally
written to fail and people didn't know they were issuing syscalls that will
never work.
The audit system can be used to find software problems so the system operates
more efficiently.
Thanks,
-Steve
9:34 a.m.
On Sun, 2013-06-09 at 10:03 -0400, Steve Grubb wrote:
> I don't think anyone wants these accesses to generate audit
records. The
> question is whether the right way to fix that is to avoid those accesses
> in the first place or to provide a mechanism so that legitimate accesses
> don't generate audit records.
There isn't a mechanism to allow these to slip through. Over the years I have
come to realize that the audit system can be a great resource for debugging
user space. It was sitting through one of Dave Jones' why userspace sucks
lectures and afterwards pouring through audit logs that I saw that we can find
some of these problems. If part of the goals when writing software is
correctness and efficiency, then wouldn't failing syscalls be of interest? Not
just in the case of EPERM, but also for example EINVAL?
Well what I'm trying to say is that you're acting as if the entire
'audit system' was carved on stone tablets and handed down from God. It
wasn't. It's just a set of checks, the logic behind _each of which_ is
as open to question as anything else. Just because a test for all EPERM
syscall fails is a part of 'the audit system' does not make it an
unquestionable totem. Instead of answering the question "do we actually
believe that all cases of EPERM should be 'fixed', or in some cases
would the cure be worse than the disease?" you seem to just keep saying
"The Holy Audit System told me there's a problem!"
I don't know who's right, in this case. But looking at the debate, I see
one side raising what looks like a legitimate line of inquiry, and you
just batting it back with 'The Holy Audit System has no flaws'.
"There isn't a mechanism", okay, point taken. But that can be a flaw of
the audit system as much as anything else.
Why would anyone write software that is incorrect enough the OS spits
it back
as EINVAL?
This is entirely irrelevant. From a QA monkey perspective, I'm comparing
this with the case where we have a suite of tests, and someone raises
the question if one of them is a sensible test. Talking about how good
one of the others is is entirely out of scope. The fact we put them all
together and called them a 'test suite' is really neither here nor
there. The question here is not 'is auditing useful?', it's 'is this
particular audit check one which always indicates a genuine bug that
must be fixed?'
I'll leave it here for anyone curious enough to dig out the
details of how
each syscall is wrong. But its my belief that these are not intentionally
written to fail and people didn't know they were issuing syscalls that will
never work.
Well, that's clearly not the case in the situation we're actually
discussing: the author of one of the pieces of software you audited says
he knows about the failed syscalls and does not think they're a problem.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
10:05 a.m.
On 06/09/2013 10:34 AM, Adam Williamson wrote:
On Sun, 2013-06-09 at 10:03 -0400, Steve Grubb wrote:
>> I don't think anyone wants these accesses to generate audit records. The
>> question is whether the right way to fix that is to avoid those accesses
>> in the first place or to provide a mechanism so that legitimate accesses
>> don't generate audit records.
>
> There isn't a mechanism to allow these to slip through. Over the years I have
> come to realize that the audit system can be a great resource for debugging
> user space. It was sitting through one of Dave Jones' why userspace sucks
> lectures and afterwards pouring through audit logs that I saw that we can find
> some of these problems. If part of the goals when writing software is
> correctness and efficiency, then wouldn't failing syscalls be of interest? Not
> just in the case of EPERM, but also for example EINVAL?
Well what I'm trying to say is that you're acting as if the entire
'audit system' was carved on stone tablets and handed down from God. It
wasn't. It's just a set of checks, the logic behind _each of which_ is
as open to question as anything else. Just because a test for all EPERM
syscall fails is a part of 'the audit system' does not make it an
unquestionable totem. Instead of answering the question "do we actually
believe that all cases of EPERM should be 'fixed', or in some cases
would the cure be worse than the disease?" you seem to just keep saying
"The Holy Audit System told me there's a problem!"
I don't know who's right, in this case. But looking at the debate, I see
one side raising what looks like a legitimate line of inquiry, and you
just batting it back with 'The Holy Audit System has no flaws'.
"There isn't a mechanism", okay, point taken. But that can be a flaw of
the audit system as much as anything else.
Not necessarily. The audit system is part of a security verification
system. Those have always had different rules than, as you bring up
later, a test suite. I remember the days when it was considered safe
practice to plug a dot-matrix line printer into your server and pipe the
output of syslog directly to the line printer so that if you got hacked,
the hackers could not erase their log trail.
The audit system is just a more modern version of that same thing. And
the second you put any sort of exception into the audit rules, then you
have to verify that the exception can never be used to circumvent the
legitimate purpose of the rule. So, if we put an exception into the
system so that PulseAudio can open these shm files and not be audited,
we would have to prove conclusively that no other application can ever
use that exception to hide their tracks (a rather difficult task) or
risk loosing some of our security certifications. At least, that's my
understanding of how these security certifications work, Steve can
verify this.
So is the audit system the "Holy Audit System"? Not in so many words,
but in some ways, yes, it is. It is not the same as the rest of the
system. Fedora has its concept of critical path packages for the
release, and for security certifications there is an entirely different
critical path set and the audit subsystem is right up top on that list,
and the kernel is right up there with it. So if your first thought
about any conflict between the audit subsystem and normal packages is
"Should we fix this in the audit subsystem or in the package?", then a
necessary part of evaluating whether or not it should be fixed in the
audit subsystem *must* be adding the burden of proving that your fix in
the audit subsystem can not be abused by malicious hackers to help
subvert your system (or to hide the evidence of their
subversion/attempts). If you aren't immediately considering that burden
when thinking about where to fix things, then you aren't considering the
entirety of the work that must be done to solve the problem in the audit
space.
> Why would anyone write software that is incorrect enough the OS
spits it back
> as EINVAL?
This is entirely irrelevant. From a QA monkey perspective, I'm comparing
this with the case where we have a suite of tests, and someone raises
the question if one of them is a sensible test. Talking about how good
one of the others is is entirely out of scope. The fact we put them all
together and called them a 'test suite' is really neither here nor
there. The question here is not 'is auditing useful?', it's 'is this
particular audit check one which always indicates a genuine bug that
must be fixed?'
> I'll leave it here for anyone curious enough to dig out the details of how
> each syscall is wrong. But its my belief that these are not intentionally
> written to fail and people didn't know they were issuing syscalls that will
> never work.
Well, that's clearly not the case in the situation we're actually
discussing: the author of one of the pieces of software you audited says
he knows about the failed syscalls and does not think they're a problem.
To answer Steve's question: because the kernel is the ultimate arbiter
of what's allowed and what isn't, so it's easier and quicker to not
bother with checking the legitimacy of your options and simply allow the
kernel to do it for you. It's lazy, but not wrong. It is, however,
inefficient in some cases. It can cause extra overhead that the user
may not know about, but might not appreciate if they did. If it just
means wasted syscalls, it doesn't waste too much CPU resources, but if
it generates audit events, the waste is much higher as Steve's tests showed.
But, putting that sysadmin hat back on, once this level of auditing is
enabled, and I as a sysadmin can see in black and white which
programmers are being lazy and which ones are properly implementing
defensive programming in their code instead of just implementing a
"throw it at the kernel and see if it breaks" approach to programming, I
start removing the lazy programmer's code from my system so my logs are
clean and I have more faith in the code on my system.
And really, we've spent more time on this thread than it would take
Lennart to fix PA. Just a quick stat and check of uid before trying to
remove the stale files and this would all go away. Sure, your stat and
remove could race, but this is nothing more than a garbage collection
process anyway, so who cares? We'll just get it next time.
10:42 a.m.
On Sun, Jun 09, 2013 at 11:05:44AM -0400, Doug Ledford wrote:
And really, we've spent more time on this thread than it would
take
Lennart to fix PA. Just a quick stat and check of uid before trying to
remove the stale files and this would all go away. Sure, your stat and
remove could race, but this is nothing more than a garbage collection
process anyway, so who cares? We'll just get it next time.
"Stat and then open" is inherently open to race conditions. In this
specific case it probably doesn't realistically matter, but it's not
behaviour that should be encouraged. The only way to know for certain
whether you're going to be able to open or unlink a file is to attempt
to open or unlink that file, and we shouldn't encourage people to think
otherwise.
--
Matthew Garrett | mjg59(a)srcf.ucam.org
3:55 p.m.
On 06/09/2013 11:42 AM, Matthew Garrett wrote:
On Sun, Jun 09, 2013 at 11:05:44AM -0400, Doug Ledford wrote:
> And really, we've spent more time on this thread than it would take
> Lennart to fix PA. Just a quick stat and check of uid before trying to
> remove the stale files and this would all go away. Sure, your stat and
> remove could race, but this is nothing more than a garbage collection
> process anyway, so who cares? We'll just get it next time.
"Stat and then open" is inherently open to race conditions.
Of course it is.
In this
specific case it probably doesn't realistically matter,
Exactly my point.
but it's not
behaviour that should be encouraged.
I would never encourage it in terms of suggesting people try to second
guess the kernel's rules and limitations. However, you can use such a
technique to weed out otherwise known to fail cases, at least in
instances like this where it is a garbage collection operation and
failure to guess properly is obviously far from critical.
The only way to know for certain
whether you're going to be able to open or unlink a file is to attempt
to open or unlink that file, and we shouldn't encourage people to think
otherwise.
Agreed.
5:21 p.m.
On Sun, Jun 09, 2013 at 04:55:40PM -0400, Doug Ledford wrote:
I would never encourage it in terms of suggesting people try to
second
guess the kernel's rules and limitations. However, you can use such a
technique to weed out otherwise known to fail cases, at least in
instances like this where it is a garbage collection operation and
failure to guess properly is obviously far from critical.
You can in some circumstances, and not in others. And in the
circumstances where you can't, you're going to generate audit records.
That's undesirable, and so we should try to find a solution that works
under all circumstances, not just the ones where we can add a stat() in
front and call it done.
--
Matthew Garrett | mjg59(a)srcf.ucam.org
3:43 a.m.
On Sun, 09.06.13 11:05, Doug Ledford (dledford(a)redhat.com) wrote:
The audit system is just a more modern version of that same thing.
And
the second you put any sort of exception into the audit rules, then you
have to verify that the exception can never be used to circumvent the
legitimate purpose of the rule. So, if we put an exception into the
system so that PulseAudio can open these shm files and not be audited,
we would have to prove conclusively that no other application can ever
use that exception to hide their tracks (a rather difficult task) or
risk loosing some of our security certifications. At least, that's my
understanding of how these security certifications work, Steve can
verify this.
You know, audit has so many holes anyway, it's super easy to circumvent
it entirely for any attacker. For example, if I never want to show up in
audit logs about forbidden file accesses I could write a tiny
preloadable library that replaces open()/fopen() by stat() or access()
right before open()/fopen(). Since stat()/access() are generally not
audited this allows me to completely evade any auditing (of course it
would still be ugly and racy as hell, but why would a hacker
care?). That this is possible you are admitting yourself by suggesting
me to do this for the special case of PA. Now, audit of course logs more
than just failed open()s, but I'd be almost willing to bet you that I
can easily find a way to circumvent generation of almost any audit
message in the system.
audit hooks into various subsystems of the OS. It makes assumptions
about why people call certain interfaces, but these assumption are
frequently wrong. And it assumes that people won't hide their intentions
when using these APIs. It assumes bad file accesses were something
unexpected in all cases, and it expects that people which want to steal
all data they can will always try to open the files directly. And in
both cases it is wrong. As the case of PA shows.
Now, this fuzziness of audit doesn't really make it a useless tool, far
from that, but it does make clear that our APIs are not designed with
audit and only audit in mind. Our APIs are usually designed to be
race-free, fast, simple, atomic, secure and a lot of other things in
mind, but auditability is really something that never was on the
table. Because for that they'd have to declare the intention why people
call these functions, and our APIs coud not have been this redundant.
What to make of this? Well, audit has to deal with the fact that its
data is incomplete in some areas, and incldues too much information in
others. Hence its emphasis should be on making the best of its dataset
but not assume too much about it. However, Steve is kinda assuming he
could rearrange his dataset instead. But that's simply not feasible. Not
feasible because of the size of our codebase, not feasible with the
current APIs, and simply because for developers correctness,
race-freeness, simplicity is more important.
audit subsystem *must* be adding the burden of proving that your fix
in
the audit subsystem can not be abused by malicious hackers to help
subvert your system (or to hide the evidence of their
subversion/attempts).
Ncie idea, but that's precisely the problem. Audit can be circumvented
to easily anyway (see above), this is definitely not something to
check. In our current OS it's the job of the audit guys to make their
reporting tools useful to deal with its incomplete/redundant dataset
rather than the one of the rest of the OS developers to generate audit
data that is perfect in the eyes of the audit guys, at the expense of
code correctness, race-freenes and simplicity.
> Well, that's clearly not the case in the situation we're
actually
> discussing: the author of one of the pieces of software you audited says
> he knows about the failed syscalls and does not think they're a problem.
To answer Steve's question: because the kernel is the ultimate arbiter
of what's allowed and what isn't, so it's easier and quicker to not
bother with checking the legitimacy of your options and simply allow the
kernel to do it for you. It's lazy, but not wrong. It is, however,
It's not lazy. It's the only correct thing to do. Reimplementing the
kernel's security checks is nearly impossible. Capabilities, file ACLs,
multiple uids, security frameworks make it incredibly hard to correctly
guess from userspace whether the kernel will grant or deny file
access. And even if you write complex code for this that covers all
current security mechanisms in place, you can bet that this will be
out-of-date in a year or two when the next security technology comes along.
And really, we've spent more time on this thread than it would
take
Lennart to fix PA. Just a quick stat and check of uid before trying to
remove the stale files and this would all go away. Sure, your stat and
remove could race, but this is nothing more than a garbage collection
process anyway, so who cares? We'll just get it next time.
Yeah, but I don't do hacks like that.
Lennart
--
Lennart Poettering - Red Hat, Inc.
10 a.m.
On 06/10/2013 04:43 AM, Lennart Poettering wrote:
On Sun, 09.06.13 11:05, Doug Ledford (dledford(a)redhat.com) wrote:
> The audit system is just a more modern version of that same thing. And
> the second you put any sort of exception into the audit rules, then you
> have to verify that the exception can never be used to circumvent the
> legitimate purpose of the rule. So, if we put an exception into the
> system so that PulseAudio can open these shm files and not be audited,
> we would have to prove conclusively that no other application can ever
> use that exception to hide their tracks (a rather difficult task) or
> risk loosing some of our security certifications. At least, that's my
> understanding of how these security certifications work, Steve can
> verify this.
You know, audit has so many holes anyway, it's super easy to circumvent
it entirely for any attacker.
That depends entirely on the attack in question.
For example, if I never want to show up in
audit logs about forbidden file accesses I could write a tiny
preloadable library that replaces open()/fopen() by stat() or access()
right before open()/fopen().
Not necessarily. Lets assume that the attack in question is a local
exploit, that it involves a buffer overflow, and that the item you have
to overflow is, tada, a filename. Because of stack randomization the
exact location of the overflow changes from run to run, so you brute
force the attack by hitting the program over and over again with a known
offset that will work eventually when the right stack offset is
selected. But for each failed run, you still generate an audit event
because the actual exploit itself involves actually calling open() on
that overflowed variable.
Obviously, this is a contrived example, but my point is that your claim
that audit events can be trivially avoided simply is not always true.
Whether or not it is possible depends entirely on where the exploit
actually lies.
Since stat()/access() are generally not
audited this allows me to completely evade any auditing (of course it
would still be ugly and racy as hell, but why would a hacker
care?). That this is possible you are admitting yourself by suggesting
me to do this for the special case of PA. Now, audit of course logs more
than just failed open()s, but I'd be almost willing to bet you that I
can easily find a way to circumvent generation of almost any audit
message in the system.
audit hooks into various subsystems of the OS. It makes assumptions
about why people call certain interfaces, but these assumption are
frequently wrong. And it assumes that people won't hide their intentions
when using these APIs. It assumes bad file accesses were something
unexpected in all cases, and it expects that people which want to steal
all data they can will always try to open the files directly. And in
both cases it is wrong. As the case of PA shows.
Now, this fuzziness of audit doesn't really make it a useless tool, far
from that, but it does make clear that our APIs are not designed with
audit and only audit in mind. Our APIs are usually designed to be
race-free, fast, simple, atomic, secure and a lot of other things in
mind, but auditability is really something that never was on the
table. Because for that they'd have to declare the intention why people
call these functions, and our APIs coud not have been this redundant.
What to make of this? Well, audit has to deal with the fact that its
data is incomplete in some areas, and incldues too much information in
others. Hence its emphasis should be on making the best of its dataset
but not assume too much about it. However, Steve is kinda assuming he
could rearrange his dataset instead. But that's simply not feasible. Not
feasible because of the size of our codebase, not feasible with the
current APIs, and simply because for developers correctness,
race-freeness, simplicity is more important.
> audit subsystem *must* be adding the burden of proving that your fix in
> the audit subsystem can not be abused by malicious hackers to help
> subvert your system (or to hide the evidence of their
> subversion/attempts).
Ncie idea, but that's precisely the problem. Audit can be circumvented
to easily anyway (see above),
And see my above where I point out that the circumventability is
constrained by the locality of the fault being (or attempting to be)
taken advantage of.
this is definitely not something to
check. In our current OS it's the job of the audit guys to make their
reporting tools useful to deal with its incomplete/redundant dataset
rather than the one of the rest of the OS developers to generate audit
data that is perfect in the eyes of the audit guys, at the expense of
code correctness, race-freenes and simplicity.
>> Well, that's clearly not the case in the situation we're actually
>> discussing: the author of one of the pieces of software you audited says
>> he knows about the failed syscalls and does not think they're a problem.
>
> To answer Steve's question: because the kernel is the ultimate arbiter
> of what's allowed and what isn't, so it's easier and quicker to not
> bother with checking the legitimacy of your options and simply allow the
> kernel to do it for you. It's lazy, but not wrong. It is, however,
It's not lazy. It's the only correct thing to do. Reimplementing the
kernel's security checks is nearly impossible. Capabilities, file ACLs,
multiple uids, security frameworks make it incredibly hard to correctly
guess from userspace whether the kernel will grant or deny file
access. And even if you write complex code for this that covers all
current security mechanisms in place, you can bet that this will be
out-of-date in a year or two when the next security technology comes along.
It is lazy. It is not the only correct thing to do. An attempt to open
a file may return EPERM, and if all you are doing is calling perror()
afterwards, your error reporting is terse and in some cases downright
unhelpful. A bit of pre-checking in your code can allow you to detect
some of these EPERM situations and not only report that the permissions
are off, but why and how to fix them. This isn't second guessing the
kernel, and you don't have to implement all of the checks that the
kernel does in order to still provide useful debugging information to
the user in the event of failure.
And you don't even have to let these pre-checks alter your actual
attempt one iota. For example, in this particular case, you could do a
stat on the file ahead of time, when you see that it's owned by another
uid() you could log to syslog at level info something like "Shared
memory segment is stale, attempting to remove, but uids don't match" and
then go ahead and try anyway, knowing you are likely to fail (but maybe
there is a special circumstance that will allow it to succeed, who
knows, so you try anyway), it does fail, it creates an audit event, but
now you have a corresponding info level log event to match the audit
event. For a sysadmin trying to track down the source of audit events,
this just went from being a headache to track down to an automatic pass
and something that can be safely ignored.
This all stems from the fact that the kernel may check 20 different
things about your syscall request, but only have 5 suitable error codes
it can use on return. The kernel will not print out a verbose message
about why your attempt failed (to do so would create a user space DoS
against the kernel by filling log buffers trivially). The kernel can
not pass back finer grained error messages (that API has been in stone
for years). So, as a user space programmer, if you want more detailed
causes for failures, it is up to you to check for them. This has value
to an end user, even if it isn't a requirement of all user space code.
So I stand by what I said: just passing the call to the kernel and
bailing without doing any checks on your own is lazy. You provide
something of value to end users by pre-checking your inputs to the
kernel and getting more fine grained failure analysis, even if you still
want to let the kernel be the final arbiter of that failure.
> And really, we've spent more time on this thread than it
would take
> Lennart to fix PA. Just a quick stat and check of uid before trying to
> remove the stale files and this would all go away. Sure, your stat and
> remove could race, but this is nothing more than a garbage collection
> process anyway, so who cares? We'll just get it next time.
Yeah, but I don't do hacks like that.
What I suggested above is hardly a hack.
10:38 a.m.
On Sun, Jun 09, 2013 at 10:03:19AM -0400, Steve Grubb wrote:
There isn't a mechanism to allow these to slip through. Over the
years I have
come to realize that the audit system can be a great resource for debugging
user space. It was sitting through one of Dave Jones' why userspace sucks
lectures and afterwards pouring through audit logs that I saw that we can find
some of these problems. If part of the goals when writing software is
correctness and efficiency, then wouldn't failing syscalls be of interest? Not
just in the case of EPERM, but also for example EINVAL?
Because this is the expected behaviour in certain cases? I'm not
disputing the usefulness of generating these reports, but there are
cases where it's entirely legitimate to receive an EPERM and do
something useful with that. The audit system needs to recognise that and
provide a mechanism for packages to flag that such accesses are genuine
and uninteresting.
--
Matthew Garrett | mjg59(a)srcf.ucam.org
11:38 a.m.
On Sun, 2013-06-09 at 10:03 -0400, Steve Grubb wrote:
Why would anyone write software that is incorrect enough the OS spits
it back
as EINVAL?
One example is the btrfs ioctl() for reflink:
https://bugzilla.gnome.org/show_bug.cgi?id=626497
3:51 a.m.
On Sun, 09.06.13 12:38, Colin Walters (walters(a)verbum.org) wrote:
On Sun, 2013-06-09 at 10:03 -0400, Steve Grubb wrote:
> Why would anyone write software that is incorrect enough the OS spits it back
> as EINVAL?
One example is the btrfs ioctl() for reflink:
https://bugzilla.gnome.org/show_bug.cgi?id=626497
You also check for EINVAL when testing for certain prctl() features at
runtime.
Or for testing whether /dev/kmsg supports reading.
Or...
Lennart
--
Lennart Poettering - Red Hat, Inc.
10:27 a.m.
On Sat, 08.06.13 11:35, Adam Williamson (awilliam(a)redhat.com) wrote:
On Sat, 2013-06-08 at 09:25 -0400, Steve Grubb wrote:
> Its not quite like this. What I need is the OS to be well behaved under normal
> conditions so that when problems come along they are easily spotted. Fedora
> has been a fairly well behaved OS over the years. I have had to get a few apps
> fixed in the past and the maintainers have always been accommodating. But this
> time I am finding we have a serious problem worse than in the past.
Well, you're defining something as 'bad behaviour' fairly arbitrarily -
or at least controversially: not everyone agrees with your definition.
Continuing to simply assert that the behaviour is bad is not driving the
conversation forward, you're just repeating a position that others have
already raised objections to. Those who are disputing your position are
not saying 'this behaviour is not happening', they are saying 'we
disagree with your definition of "bad behaviour"'.
If it's not 'bad behaviour', the fact that it didn't happen before is
fairly irrelevant. I could come up with any arbitrary 'test' for some
action that Fedora 19 does that Fedora 18 does not; that doesn't mean I
can then show up on the list waving my test results about and declaring
that there's a problem. First there has to be solid agreement that I'm
actually testing for something we shouldn't be doing.
Actually it's worse than that. What Steve proposes as a programming
style is something I'd consider actively bad, something that leads to
less secure, racy code.
And then, I totally don't like this alarmist sound to it, I mean,
nothing really changed in PA in the past 5 years, so I really fail to
see what this is all about... There isn't any new "trend" I could see
here...
Lennart
--
Lennart Poettering - Red Hat, Inc.
12:26 p.m.
On 7 June 2013 23:33, Matthew Garrett <mjg59(a)srcf.ucam.org> wrote:
On Fri, Jun 07, 2013 at 07:03:24PM -0600, Stephen John Smoogen
wrote:
> On 7 June 2013 12:29, Matthew Garrett <mjg59(a)srcf.ucam.org> wrote:
> > So why not add a mechanism to permit applications to indicate that
> > certain accesses they make should be ignored by audit?
> >
> Just so people know, this is like one of the the oldest auditing argument
> in the world. I have had programmers say that since the 1990's. [The
> standard counter story is that user program X says "don't audit anything
I
> do in /etc." The programmer counters with adding in a black list of
> directories that can't be audited, this gets countered by something else
> and eventually you have a process where programs that have a GPG
signature
> that has been accepted as valid by the audit program can say which of the
> white listed files it wants opened without audit are dealt with... and
then
> some other programmer comes in and shows the 20,000 lines of need to be
> audited code replaces 40 lines of C code in the programs that were
causing
> the problem.]
Well, http://www.stigviewer.com/check/V-29067 implies that filtering of
audit records is a reasonable thing to do. I have no expectation that
arbitrary user applications should be able to do whatever they want
without leaving an audit trail, but I don't see what that has to do with
system applications. Root has the ability to modify the selinux policy,
so root (and packages installed by root) should have the ability to
modify the set of behaviours that trigger audit records.
I believe we are discussing past each other. I don't mean to say that
filtering audit records is not a reasonable thing to do. It is however
usually a site (the hospital, the business, the military base etc) specific
decision versus one that the OS or program can determine before hand. In
this case audit has a mechanism to conform with that site specific decision
by being able to filter it out. But it has to be manually added because it
has to be the decision of the local organization whether or not they
consider X to be significant or not.
The process at the site for determining whether or not a program is doing
the right thing or not is usually outside of the skill set of the people at
the site. So they will have to open a ticket with some company, that
company will have to audit the code, determine whether the accesses can be
filtered safely or not and then either tell the person to put in an audit
skip message or put in a patch to make the program not do something as
"loudly" as it was before. In most of the cases it is where a program is
trying to open every file in a directory versus having any sort of
filtering (which pulseaudio does have already) or by sticking stuff down a
tree that is owned by the userid so that the number of files is shortened
(which int he case of /tmp files is usually done) or some other step.
In the end this sort of work is a lot more expensive than if someone goes
through various audit items early on and says "hey this is triggering a lot
more than it used to, can we fix it..." which is sort of what happened here
before various ancient feuds and personalities took over the conversations.
--
Stephen J Smoogen.
10:04 a.m.
On Fri, 07.06.13 14:02, Simo Sorce (simo(a)redhat.com) wrote:
> Well, you know, this problem isn't new. Some SELinux AVCs
can be set to
> ignored for precisely reasons like this one, because it is common that
> things like these happen: accesses which fail where that is
> expected. You can call me a troll as much as you want, but then please
> call the Selinux folks trolls first.
The SELinux folks, being the practical bunch they are simply work around
bad programs. That doesn't mean they are trolls.
Also the SeLinux folks have 'context', which is something an audit
system may not have.
Finally the fact SELinux folks are considerate does not mean that
programs need not be fixed to behave better.
Of course the author can always decide to show the finger and do what he
wants anyway, we live in a free and unpolite world after all.
Well, you know this is not about me showing anybody the finger. It's
just that I fundamentally believe that the programming practices the
audit folks propose are systematically wrong, and lead to worse code.
You should not second guess the kernel, and try to bypass the kernel's
access checks by doing them in userspace. You cannot win that game,
because you simply don't have the same information the kernel has. The
kernel is both more restrictive and less restrictive than simple UID
checks they way they have been proposed on this thread, so you have the
best chance to be wrong with manual userspace UID checks. More
restrictive are SELinux checks, less restrictive are additional ACLs
that might be on a file. And to mix things up entirely with capabilities
everything becomes extra confusing.
Then, doing these things in userspace makes these checks non-atomic. If
the kernel does these checks, then the access checks and the open() will
be atomic. Of course, one can deal with these atomicity problems... or one
can just do it properly and leave all this to the kernel...
So yeah, by doing these checks in userspace they way you suggest:
a) adds more code to userspace
b) second guesses the kernel, a game which you cannot win
c) creates atomicity problems
So, yeah, just trying to open the shm files *is the right thing to
do*. And if audit doesn't like that, then audit needs to be fixed, not
PA.
> User "simo" creates /dev/shm/1000/ even though 1000 is
the UID of user
> "lennart". Lennart can never start PA again, ever. And can't do
anything
> about it, because "simo" is in control, and /dev/shm is sticky.
That could be solved by precreating the directory at login time, like we
do for many other things. There isn't just one way.
Or you could even stop using /dev/shm and we'd have one less problem
entirely :)
Login time doesn't suffice. Isn't that trivial to see?
"Simo" logs in first, his directory is created. Then he creates also
/dev/shm/1000, which is still lennart's UID. Now Lennart logs in, and
bam, now it's even worse, he cannot even log in anymore since his
directory already exists.
So, the only option you have is to create these dirs during boot-up
before the first user can log in. But in a world of LDAP that's not
really too useful, is it?
The point is that we are simply throwing ideas off the wall as an aid
in
finding a way to solve the issue for all.
Quite honestly, these problems are well known. World-writable
directories with a shared namespace are inherently broken, you can only
create randomized files/dirs in them, everything else is necessarily
broken.
> It's really disappointing if people as seasoned as you still
don't
> understand these shared-namespace issues of world-writable directories.
Yeah, just like it is disappointing to see that your first reaction is
always to show the finger hard and solid instead of trying to have a
reasonable discussion about the problem.
The problem of shared namespace world-writable dirs is hardly new,
is it?
Anyway, this is not constructive. Let's stop this here.
Lennart
--
Lennart Poettering - Red Hat, Inc.
8:31 p.m.
On Sunday, June 09, 2013 05:04:50 PM Lennart Poettering wrote:
You should not second guess the kernel, ...
True ...
Then, doing these things in userspace makes these checks non-atomic.
...
True ...
So, yeah, just trying to open the shm files *is the right thing to
do*. And if audit doesn't like that, then audit needs to be fixed, not
PA.
It seems to me the problem stems from attempting garbage collection *when starting up
PulseAudio* as a user daemon.
Garbage collection might be better handled by a separate helper operating with root
privilege
installed in /etc/cron*. I do not remember if the various display managers offer the
ability to
run arbitrary scripts before and after the user session, and under which privilege, but
this
could also be an option,
nor if a PAM session helper could run it (without waiting).
Since the problem is with files created by PulseAudio, maybe adding ACLs allowing access
to
pulse or pulse-access group would allow the operation to succeed without triggering an
audit.
Davide Bolcioni
--
There is no place like /home.
1:21 p.m.
On Fri, Jun 7, 2013 at 6:55 PM, Lennart Poettering <mzerqung(a)0pointer.de> wrote:
Well, you know, this problem isn't new. Some SELinux AVCs can be set to
ignored for precisely reasons like this one, because it is common that
things like these happen: accesses which fail where that is
expected.
Well, whether it _should be_ expected that accesses fail is the
primary question here. UNIX has somewhat a tradition of relying on
syscall error codes to avoid one more user/kernel round trip (see e.g.
the "News Need Not Be Slow"), but it's not obvious that this is the
right thing to do at all, especially in non-performance-critical code.
SELinux as a primarily access control mechanism does not have perfect
audit information _by default_, but for those who want the full audit
information, it is still available:
http://danwalsh.livejournal.com/11673.html ). Similarly, audit
doesn't log anything of this _by default_, but it is available - and
required, as Stephen has explained.
Pragmatically speaking, SELinux also has far more information, that
allows it to write reasonably dontaudit rules. For example,
dontaudit traceroute_t { port_type -port_t }:tcp_socket name_bind;
can be most likely traced to a single line of code and a single executable.
What _would_ the audit change that made an exception for PulseAudio
look like? I haven't looked at the code, but from this thread it
seems it would have to be "any application running in an user session
(or perhaps even outside of it) can try to open any shared memory
segment". That's far too broad and not acceptable.
AFAICS this can be solved by a ~100-line? patch to the pulseaudio
ecosystem, or by doing much more work and coordicnation to design a
different shm mechanism or a new shm namespace. I'm not planning to
work on it so I suppose my opinion has little weight, but this doesn't
seem like a difficult decision at all.
Mirek
10:15 a.m.
On Fri, 07.06.13 20:21, Miloslav Trmač (mitr(a)volny.cz) wrote:
AFAICS this can be solved by a ~100-line? patch to the pulseaudio
ecosystem, or by doing much more work and coordicnation to design a
different shm mechanism or a new shm namespace. I'm not planning to
work on it so I suppose my opinion has little weight, but this doesn't
seem like a difficult decision at all.
This is not that easy. In the PA protocol we pass around triplets of
numeric SHM segment IDs, offsets into the SHM segment plus a
length. That's all there is. If we cannot access one of these segments,
that's something the kernel should tell us, and we shouldn't have to
pass any additional (and most likely incorrect) meta data around about
that...
Lennart
--
Lennart Poettering - Red Hat, Inc.
4:33 p.m.
On Fri, Jun 07, 2013 at 06:55:46PM +0200, Lennart Poettering wrote:
User "simo" creates /dev/shm/1000/ even though 1000 is the
UID of user
"lennart". Lennart can never start PA again, ever. And can't do anything
about it, because "simo" is in control, and /dev/shm is sticky.
For /run we create /run/user/<uid> in pam_systemd (I think?).
Can we do the same for /dev/shm/<uid>?
Rich.
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
10:17 a.m.
On Fri, 07.06.13 22:33, Richard W.M. Jones (rjones(a)redhat.com) wrote:
On Fri, Jun 07, 2013 at 06:55:46PM +0200, Lennart Poettering wrote:
> User "simo" creates /dev/shm/1000/ even though 1000 is the UID of user
> "lennart". Lennart can never start PA again, ever. And can't do
anything
> about it, because "simo" is in control, and /dev/shm is sticky.
For /run we create /run/user/<uid> in pam_systemd (I think?).
Can we do the same for /dev/shm/<uid>?
There's no benefit in doing that.
/run/user is not world-writable. Hence creating this dir at login time
is totally safe, since only trusted code can create dirs in there. This
is different for /dev/shm which is world-writable, and where creating
dirs at login doesn't solve anything, because any unprivileged user
could easily create dirs for all users and then make it impossible to
log in for them.
Lennart
--
Lennart Poettering - Red Hat, Inc.
3:01 p.m.
On Sun, 2013-06-09 at 17:17 +0200, Lennart Poettering wrote:
On Fri, 07.06.13 22:33, Richard W.M. Jones (rjones(a)redhat.com)
wrote:
> On Fri, Jun 07, 2013 at 06:55:46PM +0200, Lennart Poettering wrote:
> > User "simo" creates /dev/shm/1000/ even though 1000 is the UID of
user
> > "lennart". Lennart can never start PA again, ever. And can't do
anything
> > about it, because "simo" is in control, and /dev/shm is sticky.
>
> For /run we create /run/user/<uid> in pam_systemd (I think?).
> Can we do the same for /dev/shm/<uid>?
There's no benefit in doing that.
/run/user is not world-writable. Hence creating this dir at login time
is totally safe, since only trusted code can create dirs in there. This
is different for /dev/shm which is world-writable, and where creating
dirs at login doesn't solve anything, because any unprivileged user
could easily create dirs for all users and then make it impossible to
log in for them.
All this makes me wonder, why are you using /dev/shm at all if it is so
bad ? There are many other ways to do IPC, so what's keeping you
on /dev/shm ?
Simo.
--
Simo Sorce * Red Hat, Inc * New York
3:13 p.m.
On Mon, 10.06.13 16:01, Simo Sorce (simo(a)redhat.com) wrote:
On Sun, 2013-06-09 at 17:17 +0200, Lennart Poettering wrote:
> On Fri, 07.06.13 22:33, Richard W.M. Jones (rjones(a)redhat.com) wrote:
>
> > On Fri, Jun 07, 2013 at 06:55:46PM +0200, Lennart Poettering wrote:
> > > User "simo" creates /dev/shm/1000/ even though 1000 is the UID
of user
> > > "lennart". Lennart can never start PA again, ever. And can't
do anything
> > > about it, because "simo" is in control, and /dev/shm is sticky.
> >
> > For /run we create /run/user/<uid> in pam_systemd (I think?).
> > Can we do the same for /dev/shm/<uid>?
>
> There's no benefit in doing that.
>
> /run/user is not world-writable. Hence creating this dir at login time
> is totally safe, since only trusted code can create dirs in there. This
> is different for /dev/shm which is world-writable, and where creating
> dirs at login doesn't solve anything, because any unprivileged user
> could easily create dirs for all users and then make it impossible to
> log in for them.
All this makes me wonder, why are you using /dev/shm at all if it is so
bad ? There are many other ways to do IPC, so what's keeping you
on /dev/shm ?
XDG_RUNTIME_DIR is newer than PA. And POSIX shared memory
(i.e. /dev/shm/) s still the best option if you actually want a shared
namespace for the segments...
Lennart
--
Lennart Poettering - Red Hat, Inc.
1:11 a.m.
On Friday 07 June 2013 18:55:46 Lennart Poettering wrote:
> > On Fri, 07.06.13 12:09, Steve Grubb (sgrubb(a)redhat.com)
wrote:
> > > > > Maybe the uid can be encoded in the name so that wrong uid's
are
> > > > > skipped?
User "simo" creates /dev/shm/1000/ even though 1000 is the UID of user
"lennart". Lennart can never start PA again, ever. And can't do anything
about it, because "simo" is in control, and /dev/shm is sticky.
Why the UID has to be encoded in the name?
* The application can simply issue an lstat() before open() and skip
files with wrong uid's.
* Obviously, an attacker could try and trigger some race condition on
the name, but than it's OK for the audit to shout about it.
What am I missing?
--
Oron Peled Voice: +972-4-8228492
oron(a)actcom.co.il http://users.actcom.co.il/~oron
You know, someone once told me that New York has more lawyers than people.
-- Warren Buffett, Fortune, 1999
12:02 p.m.
On Friday, June 07, 2013 06:21:00 PM Lennart Poettering wrote:
On Fri, 07.06.13 12:09, Steve Grubb (sgrubb(a)redhat.com) wrote:
> > > > POSIX shared memory doesn't define any useful scheme for
automatic
> > > > removing of shared memory segments from /dev/shm after use. Hence,
> > > > in
> > > > order to make sure that left-over segments don't fill up
/dev/shm
> > > > forever PA will try to GC dead segments from /dev/shm on each
> > > > start-up. For that it iterates through /dev/shm/pulse-shm*, tries to
> > > > read the PID that is stored in there.
> > >
> > > Maybe the uid can be encoded in the name so that wrong uid's are
> > > skipped?
> >
> > But why?
>
> So that you are not filling up the audit logs. There are people that have
> to use these audit rules and we need a normally operating system to
> produce as few false positives as possible.
Maybe the audit system should be fixed to not trip up by this?
As I said before, this rule comes from the DISA STIG. Its not an audit bug,
its doing exactly what its supposed to do. Its seeing questionable behavior
and logging it.
> > This stuff should be simple, and it's always a better
idea to
> > simply let the kernel do EACCES rather then trying to be smarter than
> > the kernel and reimplement access control in userspace.
>
> Well, you are already trusting the name. Who's to say some rougue process
> didn't create the file name to try and trick pulseaudio?
Well, if the process did that, then we'd just delete his shared memory
block. I don't feel particularly tricked there... ;-) And besides that,
note that PA also checks a file signature before deleting the file, just
to be sure to not delete anything of importance...
OK, good. Then we can agree that it only deletes files that look like they
belong to pulse audio and because of the sticky bit on /dev/shm, only files you
own can be deleted.
> How about doing like some processes do in the /tmp dir...put
the
> segments in a directory /dev/shm/<user name>/ and then scan all the
> files that belong to that user?
Guessable directory names in a world-writable directory? I am sorry,
that's not an option. The stuff is already DoS-able enough, I am pretty
sure I don't want to open the door even wider. (Also what is this,
anyway? of all people, you as a security guy should know what bad an
idea that is...)
The idea is not bad. Its the implementation that can be bad. Its possible to
do this safely. And if you don't like this, then there are other ways to do
it. You can add the user name to the file name and because we established above
that pulseaudio won't delete the wrong kind of file, you are now able to
determine which one to open. A benefit to doing this is that your code should
be faster because strncmp() on the name and skipping the file is faster than
doing a syscall which in turn does name resolution and then determines access
permissions. Faster is good, right? :-)
It might also be possible to unlink the file soon after creating it so that
when the last reference is closed the file goes away automatically.
-Steve
10:48 a.m.
On Fri, 2013-06-07 at 17:14 +0200, Lennart Poettering wrote:
On Fri, 07.06.13 09:50, Steve Grubb (sgrubb(a)redhat.com) wrote:
> Let's look at one of these pule-shm events:
> # ausearch --start today -k access -f pulse-shm -i --just-one
> ----
> type=PATH msg=audit(06/07/2013 07:13:46.377:215) : item=0 name=/dev/shm/pulse-
> shm-3756395503 inode=25089 dev=00:10 mode=file,400 ouid=gdm ogid=gdm rdev=00:00
> obj=system_u:object_r:user_tmpfs_t:s0
> type=CWD msg=audit(06/07/2013 07:13:46.377:215) : cwd=/
> type=SYSCALL msg=audit(06/07/2013 07:13:46.377:215) : arch=x86_64 syscall=open
> success=no exit=-13(Permission denied) a0=0x7fffbeda83c0 a1=O_RDONLY|
> O_NOFOLLOW|O_CLOEXEC a2=0x0 a3=0x0 items=1 ppid=2369 pid=2370 auid=sgrubb
> uid=sgrubb gid=sgrubb euid=sgrubb suid=sgrubb fsuid=sgrubb egid=sgrubb
> sgid=sgrubb fsgid=sgrubb ses=2 tty=(none) comm=pulseaudio
> exe=/usr/bin/pulseaudio subj=unconfined_u:system_r:unconfined_t:s0 key=access
>
> So, gdm is creating a file 400 and pulse-audio can't open it. Is it suppose to
> be like that?
Yes, it is.
POSIX shared memory doesn't define any useful scheme for automatic
removing of shared memory segments from /dev/shm after use. Hence, in
order to make sure that left-over segments don't fill up /dev/shm
forever PA will try to GC dead segments from /dev/shm on each
start-up. For that it iterates through /dev/shm/pulse-shm*, tries to
read the PID that is stored in there. When the PID still exists it goes
to the next file. If the PID doesn't exist it unlinks the file and then
goes to the next one. It's a simple scheme that works around the
limitations of POSIX shm. Of course /dev/shm is a single namespace for
all users, hence not all files can be opened, and that's totally cool
and expected, and they will be skipped.
Shared memory on Linux is a mess. Not automatic clean up, no quota
limits, it's a sad story. If you care about security and reliability, it
would be great doing something about this, so that arbitrary users
cannot DoS the system this easily anymore...
Any reason why the PID is not part of the file name so you do not have
to open files just to find out who owns them ?
Simo.
--
Simo Sorce * Red Hat, Inc * New York
1:39 p.m.
Lennart Poettering (mzerqung(a)0pointer.de) said:
Yes, it is.
POSIX shared memory doesn't define any useful scheme for automatic
removing of shared memory segments from /dev/shm after use. Hence, in
order to make sure that left-over segments don't fill up /dev/shm
forever PA will try to GC dead segments from /dev/shm on each
start-up. For that it iterates through /dev/shm/pulse-shm*, tries to
read the PID that is stored in there. When the PID still exists it goes
to the next file. If the PID doesn't exist it unlinks the file and then
goes to the next one. It's a simple scheme that works around the
limitations of POSIX shm. Of course /dev/shm is a single namespace for
all users, hence not all files can be opened, and that's totally cool
and expected, and they will be skipped.
Shared memory on Linux is a mess. Not automatic clean up, no quota
limits, it's a sad story. If you care about security and reliability, it
would be great doing something about this, so that arbitrary users
cannot DoS the system this easily anymore...
Any reason we don't run with namespaced /dev/shm vis-a-vis private /tmp?
Bill
1:45 p.m.
On Fri, Jun 7, 2013 at 8:39 PM, Bill Nottingham <notting(a)redhat.com> wrote:
Any reason we don't run with namespaced /dev/shm vis-a-vis
private /tmp?
Private /tmp is optional and not enabled for users sessions by
default. For namespaced /dev/shm to impact pulseaudio, it would have
to be applied automatically to every user session (making the
system-wide /dev/shm unaccessible?).
Mirek
2:08 p.m.
Miloslav Trmač (mitr(a)volny.cz) said:
On Fri, Jun 7, 2013 at 8:39 PM, Bill Nottingham
<notting(a)redhat.com> wrote:
> Any reason we don't run with namespaced /dev/shm vis-a-vis private /tmp?
Private /tmp is optional and not enabled for users sessions by
default. For namespaced /dev/shm to impact pulseaudio, it would have
to be applied automatically to every user session (making the
system-wide /dev/shm unaccessible?).
Right, it would break something that requires sharing SHM segments across
sessions, or between system and sessions via 0666 SHM segments. I'm just
curious what things actually do that.
Bill
2:30 p.m.
notting(a)redhat.com wrote on 06/07/2013 15:08:59:
Right, it would break something that requires sharing SHM segments
across
sessions, or between system and sessions via 0666 SHM segments.
I'm just
curious what things actually do that.
I've seen this done with the Progress RDBMS where multiple data server
processes share block buffers, semaphores, etc. They also used the trick
mentioned earlier of making a file and unlinking it while still open (or
however exactly that's done) to have certain things (local user process
block buffers, IIRC) removed automatically.
--
John Florian
10:11 a.m.
On Fri, 07.06.13 14:39, Bill Nottingham (notting(a)redhat.com) wrote:
Lennart Poettering (mzerqung(a)0pointer.de) said:
> Yes, it is.
>
> POSIX shared memory doesn't define any useful scheme for automatic
> removing of shared memory segments from /dev/shm after use. Hence, in
> order to make sure that left-over segments don't fill up /dev/shm
> forever PA will try to GC dead segments from /dev/shm on each
> start-up. For that it iterates through /dev/shm/pulse-shm*, tries to
> read the PID that is stored in there. When the PID still exists it goes
> to the next file. If the PID doesn't exist it unlinks the file and then
> goes to the next one. It's a simple scheme that works around the
> limitations of POSIX shm. Of course /dev/shm is a single namespace for
> all users, hence not all files can be opened, and that's totally cool
> and expected, and they will be skipped.
>
> Shared memory on Linux is a mess. Not automatic clean up, no quota
> limits, it's a sad story. If you care about security and reliability, it
> would be great doing something about this, so that arbitrary users
> cannot DoS the system this easily anymore...
Any reason we don't run with namespaced /dev/shm vis-a-vis private /tmp?
We kinda have that in XDG_RUNTIME_DIR now. /dev/shm is supposed to be
shared among users I guess. If you want a user-specific namespace use
XDG_RUNTIME_DIR instead, you will get pretty much the same semantics
except that shm_open() needs to be replaced by open().
Lennart
--
Lennart Poettering - Red Hat, Inc.
3944
days inactive
3974
days old
70 comments
17 participants
participants (17)
-
Adam Williamson -
Bill Nottingham -
Colin Walters -
Davide Bolcioni -
Doug Ledford -
Eric Sandeen -
John.Florian＠dart.biz -
Lennart Poettering -
Matthew Garrett -
Matthew Miller -
Miloslav Trmač -
Oron Peled -
Richard W.M. Jones -
Roberto Ragusa -
Simo Sorce -
Stephen John Smoogen -
Steve Grubb