Re: [FC3] kernel panic after selinux-policy-targeted update

[Stephen Smalley]

On Mon, 2005-06-27 at 15:22 +0100, D. D. Brierton wrote:
>  I ran sudo yum update today and selinux-policy-targeted was updated 
>  (along with another selinux related package whose name I can't remember) 
>  and immediately my system became unresponsive and I had to do a hard reboot.
>  
>  Now I cannot boot into FC3 at all (I'm posting this from Windows). This 
>  is the error I get:
>  
>  audit(1119882959.657:0): avc: denied { execmod } for pid=1 comm=init 
>  path=/lib/tls/libc-2.3.5.so dev=hda3 ino=2638668 
>  scontext=user_u:system_r:unconfined_t tcontext=system_u:object_r:shlib_t 
>  tclass=file
>  /sbin/init: error while loading shared libraries : /lib/tls/libc.so.6: 
>  cannot apply additional memory protection after relocation: Permission 
>  denied
>  Kernel panic - not syncing: Attempted to kill init!
>  
>  Any suggestions on what to do?
>  
>  I know I can boot with selinux=0. After that, what? Attempt to reinstall 
>  selinux? 
What is your hardware?  ppc32 by any chance?  execmod has to be allowed
to all file types on that platform (or, as in kernel 2.6.12, the check
has to be disabled completely for ppc32).

/usr/sbin/getsebool allow_execmod shows what?

-- 
Stephen Smalley
National Security Agency

[FC3] kernel panic after selinux-policy-targeted update

[Russell Coker]

On Tuesday 28 June 2005 00:56, Stephen Smalley <sds(a)tycho.nsa.gov&gt; wrote:
>  > Now I cannot boot into FC3 at all (I'm posting this from
> Windows). This
>  > is the error I get:
>  >
>  > audit(1119882959.657:0): avc: denied { execmod } for pid=1 comm=init
>  > path=/lib/tls/libc-2.3.5.so dev=hda3 ino=2638668
>  > scontext=user_u:system_r:unconfined_t tcontext=system_u:object_r:shlib_t
>  > tclass=file
>  > /sbin/init: error while loading shared libraries : /lib/tls/libc.so.6:
>  > cannot apply additional memory protection after relocation: Permission
>  > denied
>  > Kernel panic - not syncing: Attempted to kill init!
>
>  What is your hardware?  ppc32 by any chance?  execmod has to be allowed
>  to all file types on that platform (or, as in kernel 2.6.12, the check
>  has to be disabled completely for ppc32).
>
>  /usr/sbin/getsebool allow_execmod shows what? 
I've just tried reproducing this on a P4-1.5GHz machine specifically installed 
for the purpose.

I upgraded to all the latest packages including kernel-2.6.11-1.35_FC3 and 
selinux-policy-targeted-sources-1.17.30-3.13.  Things worked fine.

Until I get more detail on this (type of CPU, kernel version, etc) I'll 
conclude that it was a broken configuration.

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=161867

The above bugzilla has a similar bug report, I've closed it with WORKSFORME.  
The person who reported it can reopen the bug if they have more information 
that may allow me to reproduce the bug.

-- 
http://www.coker.com.au/selinux/   My NSA Security Enhanced Linux packages
http://www.coker.com.au/bonnie++/  Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/    Postal SMTP/POP benchmark
http://www.coker.com.au/~russell/  My home page

[FC3] kernel panic after selinux-policy-targeted update

[Ben Stringer]

On Tue, 2005-06-28 at 17:15 +1000, Russell Coker wrote:
>  
>  I've just tried reproducing this on a P4-1.5GHz machine specifically installed 
>  for the purpose.
>  
>  I upgraded to all the latest packages including kernel-2.6.11-1.35_FC3 and 
>  selinux-policy-targeted-sources-1.17.30-3.13.  Things worked fine.
>  
>  Until I get more detail on this (type of CPU, kernel version, etc) I'll 
>  conclude that it was a broken configuration. 
Hi Russell,

I got hit by this one. Some details:

Dell Inspiron 8600 laptop, Centrino 1.6Ghz, running 2.6.11-1.27_FC3. An
"everything" installation of FC3, kept updated from fedora-updates and
livna. Using the 2100 wireless NIC at the time.

I did an update this afternoon, which included the selinux policy update
and the latest kernel (kernel-2.6.11-1.35_FC3). During the yum update,
things started breaking as the update applied the new policies (eg. I
couldn't use ssh from the laptop to other hosts).

When I tried to shutdown, I got many messages like this:

Jun 28 18:56:00 ben8600 kernel: audit(1119948960.209:0): avc:  denied
{ execmod } for  pid=13420 comm=mingetty path=/lib/tls/libc-2.3.5.so
dev=hda11 ino=20455 scontext=user_u:system_r:unconfined_t
tcontext=system_u:object_r:lib_t tclass=file

My only option was to power off the laptop. I then had to boot with
enforcing=0 (and a considerable amount of fscking) to get back up.

If there is any other information I can give you to help reproduce this,
let me know.

Cheers, Ben

[FC3] kernel panic after selinux-policy-targeted update

[Russell Coker]

On Tuesday 28 June 2005 22:07, Ben Stringer <ben(a)burbong.com&gt; wrote:
>  On Tue, 2005-06-28 at 17:15 +1000, Russell Coker wrote:
>  > I've just tried reproducing this on a P4-1.5GHz machine specifically
>  > installed for the purpose.
>  >
>  > I upgraded to all the latest packages including kernel-2.6.11-1.35_FC3
>  > and selinux-policy-targeted-sources-1.17.30-3.13.  Things worked fine. 
>  Dell Inspiron 8600 laptop, Centrino 1.6Ghz, running 2.6.11-1.27_FC3.
> An
>  "everything" installation of FC3, kept updated from fedora-updates and
>  livna. Using the 2100 wireless NIC at the time.
>
>  I did an update this afternoon, which included the selinux policy update
>  and the latest kernel (kernel-2.6.11-1.35_FC3). During the yum update,
>  things started breaking as the update applied the new policies (eg. I
>  couldn't use ssh from the laptop to other hosts). 
Did things work better after you had booted the new kernel?  Maybe the problem 
is a combination of new policy and slightly older kernel.

>  When I tried to shutdown, I got many messages like this:
>
>  Jun 28 18:56:00 ben8600 kernel: audit(1119948960.209:0): avc:  denied
>  { execmod } for  pid=13420 comm=mingetty path=/lib/tls/libc-2.3.5.so
>  dev=hda11 ino=20455 scontext=user_u:system_r:unconfined_t
>  tcontext=system_u:object_r:lib_t tclass=file 
That's an example of a .so file which is mis-labeled.

What version of glibc?  Mine is glibc-2.3.5-0.fc3.1.

>  My only option was to power off the laptop. I then had to boot with
>  enforcing=0 (and a considerable amount of fscking) to get back up.
>
>  If there is any other information I can give you to help reproduce this,
>  let me know. 
What state is the machine in now?

-- 
http://www.coker.com.au/selinux/   My NSA Security Enhanced Linux packages
http://www.coker.com.au/bonnie++/  Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/    Postal SMTP/POP benchmark
http://www.coker.com.au/~russell/  My home page

[FC3] kernel panic after selinux-policy-targeted update

[Ben Stringer]

On Tue, 2005-06-28 at 22:27 +1000, Russell Coker wrote:
>  >
>  > I did an update this afternoon, which included the selinux policy update
>  > and the latest kernel (kernel-2.6.11-1.35_FC3). During the yum update,
>  > things started breaking as the update applied the new policies (eg. I
>  > couldn't use ssh from the laptop to other hosts).
>  
>  Did things work better after you had booted the new kernel?  Maybe the problem 
>  is a combination of new policy and slightly older kernel. 
Still have not tried the new kernel yet. I will give this a go.

>  
>  > When I tried to shutdown, I got many messages like this:
>  >
>  > Jun 28 18:56:00 ben8600 kernel: audit(1119948960.209:0): avc:  denied
>  > { execmod } for  pid=13420 comm=mingetty path=/lib/tls/libc-2.3.5.so
>  > dev=hda11 ino=20455 scontext=user_u:system_r:unconfined_t
>  > tcontext=system_u:object_r:lib_t tclass=file
>  
>  That's an example of a .so file which is mis-labeled.
>  
>  What version of glibc?  Mine is glibc-2.3.5-0.fc3.1. 
Mine is the same.

>  
>  > My only option was to power off the laptop. I then had to boot with
>  > enforcing=0 (and a considerable amount of fscking) to get back up.
>  >
>  > If there is any other information I can give you to help reproduce this,
>  > let me know.
>  
>  What state is the machine in now? 
I have dropped back to the previous policy and relabelled, using these
steps, as posted here earlier today:

        rpm -ev selinux-policy-targeted selinux-policy-targeted-sources
        rm -fR /etc/selinux/targeted/
        rpm -ivh /var/cache/yum/updates-released/packages/selinux-
policy-targeted-1.17.30-3.9.noarch.rpm /var/cache/yum/updates-
released/packages/selinux-policy-targeted-sources-1.17.30-3.9.noarch.rpm
        touch /.autorelabel

Everything seems to be back to normal. My next steps (when I can afford
the time of having the laptop unavailable) will be to boot into the new
kernel. still using the previous policy file, confirm all is good with
that, then re-apply the new policy update and see if the same problems
occur.

Cheers, Ben

[FC3] kernel panic after selinux-policy-targeted update

[Stephen Smalley]

On Tue, 2005-06-28 at 23:11 +1000, Ben Stringer wrote:
>  Everything seems to be back to normal. My next steps (when I can
> afford
>  the time of having the laptop unavailable) will be to boot into the new
>  kernel. still using the previous policy file, confirm all is good with
>  that, then re-apply the new policy update and see if the same problems
>  occur. 
When/if you do that, put the machine into permissive mode (setenforce
0), clear /var/log/messages, and enable syscall auditing (auditctl -e 1)
prior to applying the policy update.  It would also help to run one of
the failing programs under strace and collect that output.

-- 
Stephen Smalley
National Security Agency

[FC3] kernel panic after selinux-policy-targeted update

[Ben Stringer]

On Tue, 2005-06-28 at 11:20 -0400, Stephen Smalley wrote:
>  On Tue, 2005-06-28 at 23:11 +1000, Ben Stringer wrote:
>  > Everything seems to be back to normal. My next steps (when I can afford
>  > the time of having the laptop unavailable) will be to boot into the new
>  > kernel. still using the previous policy file, confirm all is good with
>  > that, then re-apply the new policy update and see if the same problems
>  > occur.
>  
>  When/if you do that, put the machine into permissive mode (setenforce
>  0), clear /var/log/messages, and enable syscall auditing (auditctl -e 1)
>  prior to applying the policy update.  It would also help to run one of
>  the failing programs under strace and collect that output.
>   
The upgrade to selinux-policy-targeted-1.17.30-3.15 went without a hitch
when done whilst running kernel-2.6.11-1.35_FC3.

Cheers, Ben

[FC3] kernel panic after selinux-policy-targeted update

[Stephen Smalley]

On Tue, 2005-06-28 at 22:27 +1000, Russell Coker wrote:
>  > Jun 28 18:56:00 ben8600 kernel: audit(1119948960.209:0): avc: 
> denied
>  > { execmod } for  pid=13420 comm=mingetty path=/lib/tls/libc-2.3.5.so
>  > dev=hda11 ino=20455 scontext=user_u:system_r:unconfined_t
>  > tcontext=system_u:object_r:lib_t tclass=file
>  
>  That's an example of a .so file which is mis-labeled. 
Not necessarily.  Prior to -3.13, shlib_t was a typealias for lib_t in
the targeted policy, so it would be normal for audit messages to display
lib_t here for a .so.  Real question is why is an execmod check being
triggered on /lib/tls/libc-2.3.5.so, as it should only occur on attempts
to make executable a previously modified private file mapping, typically
text relocation.

-- 
Stephen Smalley
National Security Agency

[FC3] kernel panic after selinux-policy-targeted update

[Chuck Anderson]

On Tue, Jun 28, 2005 at 10:27:53PM +1000, Russell Coker wrote:
>  On Tuesday 28 June 2005 22:07, Ben Stringer <ben(a)burbong.com&gt;
> wrote:
>  > I did an update this afternoon, which included the selinux policy update
>  > and the latest kernel (kernel-2.6.11-1.35_FC3). During the yum update,
>  > things started breaking as the update applied the new policies (eg. I
>  > couldn't use ssh from the laptop to other hosts).
>  
>  Did things work better after you had booted the new kernel?  Maybe the problem 
>  is a combination of new policy and slightly older kernel. 
I updated the bz ticket #161867.  All the systems I had this problem
with were running 2.6.11-1.27_FC3 at the time the update was done. 
The systems running 2.6.11-1.35_FC3 didn't experience the problem.  So
it does appear that the problem is the older kernel and the newer
policy.

[FC3] kernel panic after selinux-policy-targeted update

[Stephen Smalley]

On Tue, 2005-06-28 at 13:46 -0400, Chuck Anderson wrote:
>  I updated the bz ticket #161867.  All the systems I had this problem
>  with were running 2.6.11-1.27_FC3 at the time the update was done. 
>  The systems running 2.6.11-1.35_FC3 didn't experience the problem.  So
>  it does appear that the problem is the older kernel and the newer
>  policy. 
Hmmm...interesting, since AFAIK, the SELinux code didn't change between
those two kernels, and FC3 kernel has no SELinux-related patches in it
(it just uses the upstream code).  Side effect of another patch in the
FC3 kernel?

-- 
Stephen Smalley
National Security Agency

[FC3] kernel panic after selinux-policy-targeted update

[Oleg Makarenko]

Stephen Smalley wrote:

> On Tue, 2005-06-28 at 13:46 -0400, Chuck Anderson wrote:
>   
>
> >I updated the bz ticket #161867.  All the systems I had this problem
> >with were running 2.6.11-1.27_FC3 at the time the update was done. 
> >The systems running 2.6.11-1.35_FC3 didn't experience the problem.  So
> >it does appear that the problem is the older kernel and the newer
> >policy.
> >    
> >
>
> Hmmm...interesting, since AFAIK, the SELinux code didn't change between
> those two kernels, and FC3 kernel has no SELinux-related patches in it
> (it just uses the upstream code).  Side effect of another patch in the
> FC3 kernel?
>
>   
>  
Just to add more confusion... or probably give some hints to somebody...

I have the same problem on _both_ 1.27_FC3 and 1.35_FC3 kernels.

On 1.35_FC3 machine (remote 2 Xeon x686 server) sshd and mingetty were
broken after the recent policy update.

I rebooted it with enforcing=0 (using remote console) and then

make -W users reload

(I have policy sources installed on the machine)

Everything works fine since then with
selinux-policy-targeted-1.17.30-3.13 and kernel-smp-2.6.11-1.35_FC3. My
policy sources have very minor changes in apache.te and mysqld.te files
only. Some http related booleans are also different... May be the binary
policy in the package is broken?

On my home 1.27_FC3 machine I have just updated the policy and have not
rebooted yet. Just after the update a lot of things are broken. For
example I am unable to start a new (gnome-)terminal etc etc

setenforce 0 in the root's window (that I happen to run yum from) helps.
Now I am able to start new non root's terminal and mozilla to write this
e-mail :)

If I then do setenforce 1 and try to ls I get:

[oleg@mole ~]$ ls
ls: error while loading shared libraries: /lib/tls/librt.so.1: cannot
apply additional memory protection after relocation: Permission denied

and in /var/log/messages I see

Jun 28 23:42:01 localhost kernel: audit(1119987721.476:0): avc:  denied 
{ execmod } for  pid=5873 comm=ls path=/lib/tls/librt-2.3.5.so dev=hda3
ino=16719 scontext=user_u:system_r:unconfined_t
tcontext=system_u:object_r:lib_t tclass=file

 when I try to run ssh I get:

[oleg@mole ~]$ ssh localhost
ssh: error while loading shared libraries: /lib/libdl.so.2: cannot apply
additional memory protection after relocation: Permission denied

and

Jun 28 23:44:29 localhost kernel: audit(1119987869.572:0): avc:  denied 
{ execmod } for  pid=5882 comm=ssh path=/lib/libdl-2.3.5.so dev=hda3
ino=2052530 scontext=user_u:system_r:unconfined_t
tcontext=system_u:object_r:lib_t tclass=file

in the root's terminal  everything works fine even with setenforcing 1

hope this information may be useful.

=oleg

[FC3] kernel panic after selinux-policy-targeted update

[Oleg Makarenko]

Oleg Makarenko wrote:

>  Just to add more confusion... or probably give some hints to
> somebody...
>
> I have the same problem on _both_ 1.27_FC3 and 1.35_FC3 kernels.
>
> On 1.35_FC3 machine (remote 2 Xeon x686 server) sshd and mingetty were
> broken after the recent policy update.
>
> I rebooted it with enforcing=0 (using remote console) and then
>
> make -W users reload
>
> (I have policy sources installed on the machine)
>
> Everything works fine since then with
> selinux-policy-targeted-1.17.30-3.13 and kernel-smp-2.6.11-1.35_FC3. My
> policy sources have very minor changes in apache.te and mysqld.te files
> only. Some http related booleans are also different... May be the binary
> policy in the package is broken?
>
> On my home 1.27_FC3 machine I have just updated the policy and have not
> rebooted yet. Just after the update a lot of things are broken. For
> example I am unable to start a new (gnome-)terminal etc etc
>
> setenforce 0 in the root's window (that I happen to run yum from) helps.
> Now I am able to start new non root's terminal and mozilla to write this
> e-mail :)
>
> If I then do setenforce 1 and try to ls I get:
>
> [oleg@mole ~]$ ls
> ls: error while loading shared libraries: /lib/tls/librt.so.1: cannot
> apply additional memory protection after relocation: Permission denied
>
> and in /var/log/messages I see
>
> Jun 28 23:42:01 localhost kernel: audit(1119987721.476:0): avc:  denied 
> { execmod } for  pid=5873 comm=ls path=/lib/tls/librt-2.3.5.so dev=hda3
> ino=16719 scontext=user_u:system_r:unconfined_t
> tcontext=system_u:object_r:lib_t tclass=file
>
>  when I try to run ssh I get:
>
> [oleg@mole ~]$ ssh localhost
> ssh: error while loading shared libraries: /lib/libdl.so.2: cannot apply
> additional memory protection after relocation: Permission denied
>
> and
>
> Jun 28 23:44:29 localhost kernel: audit(1119987869.572:0): avc:  denied 
> { execmod } for  pid=5882 comm=ssh path=/lib/libdl-2.3.5.so dev=hda3
> ino=2052530 scontext=user_u:system_r:unconfined_t
> tcontext=system_u:object_r:lib_t tclass=file
>
> in the root's terminal  everything works fine even with setenforcing 1
>
> hope this information may be useful.
>
> =oleg
>   
>  
I have installed 1.35_FC3 kernel on my 1.27_FC3 machine and it works
fine with the latest policy without any additional tricks. With exactly
the same settings and policy 1.27_FC3 doesn't boot as /sbin/init
triggers avc: denied { execmod }.

1.14 doesn't work either while kernel-2.6.10-1.770_FC3 works fine with
the new policy.

Policy rebuilding doesn't help here so probably my 1.35_FC3 machine
actually run kernel 1.27_FC3 at the update time.

Sorry for confusion.

So I also see the problem only on 1.14 and 1.27 kernels.

=oleg

[FC3] kernel panic after selinux-policy-targeted update

[Stephen Smalley]

On Tue, 2005-06-28 at 17:15 +1000, Russell Coker wrote:
>  I've just tried reproducing this on a P4-1.5GHz machine
> specifically installed 
>  for the purpose.
>  
>  I upgraded to all the latest packages including kernel-2.6.11-1.35_FC3 and 
>  selinux-policy-targeted-sources-1.17.30-3.13.  Things worked fine.
>  
>  Until I get more detail on this (type of CPU, kernel version, etc) I'll 
>  conclude that it was a broken configuration. 
That doesn't reproduce the sequence for real users, because they would
have gone through a series of policy and kernel updates over time, with
any potential pairing of released policies and kernels running at any
given time.

If you look at the diffs between successive policy updates for FC3,
there are some obvious issues there.  I started with 2.96 and did
incremental diffs up to 3.13.  Note that:

- shlib_t is a type alias for lib_t in 2.96, but in 3.13, is suddenly
changed to a separate type.  So if you booted your kernel any policy
prior to 3.13 and just did policy reload for the update to 3.13, you
will have incore inodes that have already been mapped to lib_t
internally that should be shlib_t, and any rules on shlib_t will no
longer be applied to them.
- texrel_shlib_t is a separate type in 3.2, is changed to a type alias
for lib_t in 3.9, and is changed back to a separate type in 3.13.
Similar issues, albeit with lesser impact.
- allow rules for execmem/execmod that were added for backward
compatibility in FC3 when the 2.6.11-based kernel update was released
are suddenly dropped in 3.13.  These were necessary because 2.6.11
lacked the checkreqprot and ppc32 compatibility code that was included
in early FC4/devel kernels and ultimately upstreamed into 2.6.12, and
for whatever reason, the FC3 update kernel for 2.6.11 did not include
that patch, so the policy had to include those allow rules instead.

However, I'm still not clear as to why FC3 users are seeing the
particular denials that they are seeing (e.g. on x86, for /sbin/init,
execmod to /lib/tls/libc.*), and that would be worth investigating
further.

-- 
Stephen Smalley
National Security Agency