Re: [atomic-devel] tools and systemtap containers are available in Fedora

[William Cohen]

On 10/05/2017 10:33 AM, Jeremy Eder wrote:
>  Forgot to add Will Cohen (discussed stap errors with him briefly). 
> Also my replies won't make it to the dev list since I am not subscribed (just fyi I
> guess).
>  
>  On Thu, Oct 5, 2017 at 9:10 AM, Jeremy Eder <jeder(a)redhat.com
> <mailto:[e-mail]>> wrote:
>  
>      First of all, that readme is awesome.
>  
>      spot checking the tools container...seems to all "just work" when I run it
> with atomic run ...
>      blktrace works
>      ethtool works (-K -i -c -S specifically)
>      netstat works
>      pstack works
>      perf top,record,report works
>      iotop works
>      slabtop works
>      lstopo works
>      htop works (wish this was in rhel)
>      nstat works
>      ss works (-tmpie)
>      ifpps works (wish this was in rhel)
>      numastat works (-mczs)
>      pmap works
>      all the sysstat tools work
>      strace works
>      tcpdump works
>      sar works but you have to prepend the /host directory (so, sar -f
> /host/var/log/sa/sa05)
>      my god tmux is in here?? yes!
>  
>  
>      ​systemtap (aww, no readme?)
>  
>      doesnt work:
>      ​[root@8b7437fed211 /]# cd /usr/share/systemtap/examples/process/                    
>                                                                                          
>               
>      [root@8b7437fed211 process]# stap cycle_thief.stp
>      ERROR: Couldn't insert module
> '/tmp/stapslabb9/stap_0811c9eea1bbb81f2fbc5f7bf9df4506_8509.ko': Operation not
> permitted
>      WARNING: /usr/bin/staprun exited with status: 1
>      Pass 5: run failed.  [man error::pass5]
>      [root@8b7437fed211 process]# 
>  
>  
>  
>      [root@dhcp23-91 ~]# atomic run --spc
> candidate-registry.fedoraproject.org/f26/systemtap
> <http://candidate-registry.fedoraproject.org/f26/systemtap>
>      docker run --cap-add SYS_MODULE -v /sys/kernel/debug:/sys/kernel/debug -v
> /usr/src/kernels:/usr/src/kernels -v /usr/lib/modules/:/usr/lib/modules/ -v
> /usr/lib/debug:/usr/lib/debug -t -i --name systemtap-spc
> candidate-registry.fedoraproject.org/f26/systemtap
> <http://candidate-registry.fedoraproject.org/f26/systemtap>
>  
>      This container uses privileged security switches:
>  
>      INFO: --cap-add 
>            Adding capabilities to your container could allow processes from the container
> to break out onto your host system.
>  
>      For more information on these switches and their security implications, consult the
> manpage for 'docker run'.
>  
>      [root@10accce504c2 /]# cd /usr/share/systemtap/examples/process/
>      [root@10accce504c2 process]# stap cycle_thief.stp 
>      ERROR: Couldn't insert module
> '/tmp/stapNEjJDX/stap_4f013e7562b546a0316af840de9f0713_8509.ko': Operation not
> permitted
>      WARNING: /usr/bin/staprun exited with status: 1
>      Pass 5: run failed.  [man error::pass5]
>  
>  
>  
>      On Thu, Oct 5, 2017 at 3:09 AM, Tomas Tomecek <ttomecek(a)redhat.com
> <mailto:[e-mail]>> wrote:
>  
>          Not sure if the question is for me -- I literally have no idea how to do that.
>  
>  
>          Let me know how I can help,
>  
>          Tomas
>  
>  
>          On Thu, Oct 5, 2017 at 5:04 AM, Dusty Mabe <dusty(a)dustymabe.com
> <mailto:[e-mail]>> wrote:
>  
>  
>  
>              On 09/18/2017 10:48 AM, Tomas Tomecek wrote:
>              > Hello,
>              >
>              > we managed to move tools container from Fedora Dockerfiles github repo
> to Fedora infra [1]. As a side effects, we put systemtap in a dedicated container.
>              >
>              > We would very much appreciate your feedback here: so if you have some
> time to take a look at these containers and try them out, it would mean a lot to us.
>              >
>              > Repos:
>              > https://src.fedoraproject.org/container/systemtap
> <https://src.fedoraproject.org/container/systemtap>
>              > https://src.fedoraproject.org/container/tools
> <https://src.fedoraproject.org/container/tools>
>              >
>              > The way to access the images:
>              > docker pull candidate-registry.fedoraproject.org/f26/tools
> <http://candidate-registry.fedoraproject.org/f26/tools>
> <http://candidate-registry.fedoraproject.org/f26/tools
> <http://candidate-registry.fedoraproject.org/f26/tools>>
>  
>              just tested out the tools container. can we get this into the official
> registry?
>  
>              > docker pull candidate-registry.fedoraproject.org/f26/systemtap
> <http://candidate-registry.fedoraproject.org/f26/systemtap>
> <http://candidate-registry.fedoraproject.org/f26/systemtap
> <http://candidate-registry.fedoraproject.org/f26/systemtap>>
>              >
>              > Both images have help files, so please read them prior using the
> containers:
>              >
> https://src.fedoraproject.org/container/tools/blob/master/f/root/README.md
> <https://src.fedoraproject.org/container/tools/blob/master/f/root/README.m...
>              > https://github.com/container-images/systemtap/blob/master/help/help.md
> <https://github.com/container-images/systemtap/blob/master/help/help.md>
>              >
>              > (or `atomic help $the_container_image`)
>              >
>              > [1] https://pagure.io/atomic-wg/issue/214
> <https://pagure.io/atomic-wg/issue/214>
>  
>  
>  
>  
>  
>      -- 
>  
>      -- Jeremy Eder
>  
>  
>  
>  
>  -- 
>  
>  -- Jeremy Eder 
Hi,

I have done some probing around on the environment that Jeremy setup.

The problem seems to be limited to the actual loading of the the generated module in the
container.
I was able to do:

# stap -p4 -m cycle_thief /usr/share/systemtap/examples/process/cycle_thief.stp

Then copy the cycle_thief.ko from inside the container to the host machine.  The following
command to run things on the host works fine:

# staprun ./cycle_thief.ko

Conversely was albe to load and unload various kernel modules on the host with modprobe
and rmprobe, but unable to same operations within the kernel.

What is the list of syscalls allowed?

Maybe run container-check.stp on the host looking at the container that we are trying to
run systemtap inside.  How do we find out the process that spawned off that container? 
Installed "pstree", started a process in the client that could find in pstree
output.  Then:

# ./container_check.stp -v -x 2816
Pass 1: parsed user script and 471 library scripts using
139876virt/46200res/7696shr/38748data kb, in 140usr/30sys/175real ms.
Pass 2: analyzed script: 582 probes, 21 functions, 104 embeds, 110 globals using
308456virt/216372res/9060shr/207328data kb, in 29990usr/390sys/30531real ms.
Pass 3: translated to C into
"/tmp/stapVQGA4T/stap_942629388b1b117eb698f8777091b161_1001584_src.c" using
308456virt/216372res/9060shr/207328data kb, in 2880usr/20sys/2926real ms.
Pass 4: compiled C into "stap_942629388b1b117eb698f8777091b161_1001584.ko" in
76330usr/1700sys/78140real ms.
Pass 5: starting run.
starting container_check.stp. monitoring 2816
^C

capabilities used by executables
      executable:      prob capability



capabilities used by syscalls
      executable,              syscall (       capability ) :            count


forbidden syscalls
      executable,              syscall:            count


failed syscalls
      executable,              syscall =            errno:            count
            bash,                 stat =           ENOENT:                1
            bash,                wait4 =           ECHILD:                1
         staprun,          init_module =            EPERM:                1
         staprun,               access =           ENOENT:                1
         staprun,                 stat =           ENOENT:                1
Pass 5: run completed in 10usr/9170sys/22614real ms.


So it looks like init_module syscall is not being allowed.

-Will

[atomic-devel] tools and systemtap containers are available in Fedora

[Frank Ch. Eigler]

wcohen forwarded:

>  [...]
> >   [root@dhcp23-91 ~]# atomic run --spc
> candidate-registry.fedoraproject.org/f26/systemtap
> <http://candidate-registry.fedoraproject.org/f26/systemtap>
> >     docker run --cap-add SYS_MODULE -v /sys/kernel/debug:/sys/kernel/debug -v
> /usr/src/kernels:/usr/src/kernels -v /usr/lib/modules/:/usr/lib/modules/ -v
> /usr/lib/debug:/usr/lib/debug -t -i --name systemtap-spc
> candidate-registry.fedoraproject.org/f26/systemtap
> <http://candidate-registry.fedoraproject.org/f26/systemtap>
> >  [...]
> >     ERROR: Couldn't insert module
> '/tmp/stapNEjJDX/stap_4f013e7562b546a0316af840de9f0713_8509.ko': Operation not
> permitted
> > [...] 
I bet
   # setenforce 0
makes it work for you.  As per audit.log:

type=AVC msg=audit(1507222590.683:7940): avc:  denied  { module_load }
for  pid=7595 comm="staprun"
scontext=system_u:system_r:container_t:s0:c534,c921
tcontext=system_u:system_r:container_t:s0:c534,c921 tclass=system permissive=1


- FChE

[atomic-devel] tools and systemtap containers are available in Fedora

[Daniel Walsh]

On 10/05/2017 01:00 PM, Frank Ch. Eigler wrote:
>  wcohen forwarded:
>
> > [...]
> >>    [root@dhcp23-91 ~]# atomic run --spc
> candidate-registry.fedoraproject.org/f26/systemtap
> <http://candidate-registry.fedoraproject.org/f26/systemtap>
> >>      docker run --cap-add SYS_MODULE -v /sys/kernel/debug:/sys/kernel/debug -v
> /usr/src/kernels:/usr/src/kernels -v /usr/lib/modules/:/usr/lib/modules/ -v
> /usr/lib/debug:/usr/lib/debug -t -i --name systemtap-spc
> candidate-registry.fedoraproject.org/f26/systemtap
> <http://candidate-registry.fedoraproject.org/f26/systemtap>
> >>   [...]
> >>      ERROR: Couldn't insert module
> '/tmp/stapNEjJDX/stap_4f013e7562b546a0316af840de9f0713_8509.ko': Operation not
> permitted
> >> [...]
>  I bet
>      # setenforce 0
>  makes it work for you.  As per audit.log:
>
>  type=AVC msg=audit(1507222590.683:7940): avc:  denied  { module_load }
>  for  pid=7595 comm="staprun"
> scontext=system_u:system_r:container_t:s0:c534,c921
>  tcontext=system_u:system_r:container_t:s0:c534,c921 tclass=system permissive=1
>
>
>  - FChE
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org 
Rather then putting the system into permissive mode, you should run a 
privileged container or at least disable SELinux protections.


docker run -ti --security-opt label:disable ...

[atomic-devel] tools and systemtap containers are available in Fedora

[Frank Ch. Eigler]

Hi, Dan -

>  [...]
>  Rather then putting the system into permissive mode, you should run
>  a privileged container  
"atomic run --spc ...." fails similarly on f26, despite its
underlying "docker run --cap-add SYS_MODULE ..." parts.

>  or at least disable SELinux protections.
>
>  docker run -ti --security-opt label:disable ... 
Is there an atomic(1) command line equivalent for this?  Or would
one have to put the security-option bits into the Dockerfile LABEL?


- FChE

[atomic-devel] tools and systemtap containers are available in Fedora

[Daniel Walsh]

On 10/05/2017 01:11 PM, Frank Ch. Eigler wrote:
>  Hi, Dan -
>
> > [...]
> > Rather then putting the system into permissive mode, you should run
> > a privileged container
>  "atomic run --spc ...." fails similarly on f26, despite its
>  underlying "docker run --cap-add SYS_MODULE ..." parts.
>
> > or at least disable SELinux protections.
> >
> > docker run -ti --security-opt label:disable ...
>  Is there an atomic(1) command line equivalent for this?  Or would
>  one have to put the security-option bits into the Dockerfile LABEL?
>
>
>  - FChE
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org 
Could you show the docker line that atomic run is executing?  The LABEL 
would be the

preferred way.

[atomic-devel] tools and systemtap containers are available in Fedora

[Frank Ch. Eigler]

Hi, Dan -


>  Could you show the docker line that atomic run is executing?  
% atomic run --spc candidate-registry.fedoraproject.org/f26/systemtap
/usr/share/systemtap/examples/io/iotop.stp
docker run --cap-add SYS_MODULE -v /sys/kernel/debug:/sys/kernel/debug -v
/usr/src/kernels:/usr/src/kernels -v /usr/lib/modules/:/usr/lib/modules/ -v
/usr/lib/debug:/usr/lib/debug -t -i --name systemtap-spc
candidate-registry.fedoraproject.org/f26/systemtap
/usr/share/systemtap/examples/io/iotop.stp

... which fails.  But a hand-run % docker run, with "--security-opt
label:disable" added in the front works for me.


>  The LABEL would be the preferred way. 
Sure, just someone(tm) needs to find the Dockerfile in git.  I
couldn't find it from a dozen minutes reading
https://fedoraproject.org/wiki/Changes/Layered_Docker_Image_Build_Service
and pals.


- FChE

[atomic-devel] tools and systemtap containers are available in Fedora

[Daniel Walsh]

On 10/05/2017 01:47 PM, Frank Ch. Eigler wrote:
>  Hi, Dan -
>
>
> > Could you show the docker line that atomic run is executing?
>  % atomic run --spc candidate-registry.fedoraproject.org/f26/systemtap
> /usr/share/systemtap/examples/io/iotop.stp
>  docker run --cap-add SYS_MODULE -v /sys/kernel/debug:/sys/kernel/debug -v
> /usr/src/kernels:/usr/src/kernels -v /usr/lib/modules/:/usr/lib/modules/ -v
> /usr/lib/debug:/usr/lib/debug -t -i --name systemtap-spc
> candidate-registry.fedoraproject.org/f26/systemtap
> /usr/share/systemtap/examples/io/iotop.stp
>
>  ... which fails.  But a hand-run % docker run, with "--security-opt
>  label:disable" added in the front works for me.
>
>
> > The LABEL would be the preferred way.
>  Sure, just someone(tm) needs to find the Dockerfile in git.  I
>  couldn't find it from a dozen minutes reading
>  https://fedoraproject.org/wiki/Changes/Layered_Docker_Image_Build_Service
>  and pals.
>
>
>  - FChE 
But really for something like this, it would be better to just run it 
--privileged.  There is on security confinement present in what you are 
doing.

[atomic-devel] tools and systemtap containers are available in Fedora

[Frank Ch. Eigler]

Hi, Dan -

On Thu, Oct 05, 2017 at 01:49:48PM -0400, Daniel Walsh wrote:
>  [...]
>  But really for something like this, it would be better to just run
>  it --privileged.  There is [no] security confinement present in what
>  you are doing. 
Yup.  I thought "atomic run --spc" would imply "docker run
--privileged"
but it doesn't seem to.

- FChE

[atomic-devel] tools and systemtap containers are available in Fedora

[Daniel Walsh]

On 10/05/2017 01:55 PM, Frank Ch. Eigler wrote:
>  Hi, Dan -
>
>  On Thu, Oct 05, 2017 at 01:49:48PM -0400, Daniel Walsh wrote:
> > [...]
> > But really for something like this, it would be better to just run
> > it --privileged.  There is [no] security confinement present in what
> > you are doing.
>  Yup.  I thought "atomic run --spc" would imply "docker run
> --privileged"
>  but it doesn't seem to.
>
>  - FChE
>  _______________________________________________
>  devel mailing list -- devel(a)lists.fedoraproject.org
>  To unsubscribe send an email to devel-leave(a)lists.fedoraproject.org 
No it looks like it is just running the label that is in the container 
image.