Sabayon/Gentoo distribution of 389org

[Roberto Polli]

Hi all,
our company decided to sponsor a Sabayon/Gentoo distribution of 389org.

It seems that there are some issues in admin-server, which have been deeply 
investigated by the Sabayon maintainer.

Those issues could be related to the latest versions of the libraries used in 
Gentoo; so the question:
- did some of you test 389org on Fedora Linux or RHEL6?


Peace,
R.
-- 

Roberto Polli
Babel S.r.l. - http://www.babel.it
Tel. [phone] - fax [phone]
Tel. cel +39.340.6522736
P.zza S.Benedetto da Norcia, 33 - 00040 Pomezia (Roma)

"Il seguente messaggio contiene informazioni riservate. Qualora questo 
messaggio fosse da Voi ricevuto per errore, Vogliate cortesemente darcene 
notizia a mezzo e-mail. Vi sollecitiamo altresì a distruggere il messaggio 
erroneamente ricevuto. Quanto precede Vi viene chiesto ai fini del rispetto 
della legge in materia di protezione dei dati personali."

[Rich Megginson]

Roberto Polli wrote:
>  Hi all,
>  our company decided to sponsor a Sabayon/Gentoo distribution of 389org.
>
>  It seems that there are some issues in admin-server, which have been deeply 
>  investigated by the Sabayon maintainer.
>
>  Those issues could be related to the latest versions of the libraries used in 
>  Gentoo; so the question:
>  - did some of you test 389org on Fedora Linux or RHEL6?
>     
Yes.  I've just tested the latest 389-admin-1.1.11.rc1 and 
389-ds-base-1.2.6.rc3 on Fedora rawhide (F-14)
This system uses httpd version 2.2.15 and apr 1.3.9
* yum install 389-ds
* setup-ds-admin.pl - use defaults
* launched the web browser - http://localhost:9830/ - everything works 
fine - main page, topology page, log viewer, etc. (including the CGI the 
maintainer was having problems with, or at least I'm assuming this is 
the same problem I was working on IRC with lxnay last week)

The problem seems specific to Gentoo
>  Peace,
>  R.
>

[Fabio Erculiani]

On Tue, Jul 6, 2010 at 5:58 PM, Rich Megginson <rmeggins(a)redhat.com&gt; wrote:
>  Roberto Polli wrote:
> >
> > Hi all,
> > our company decided to sponsor a Sabayon/Gentoo distribution of 389org.
> >
> > It seems that there are some issues in admin-server, which have been
> > deeply investigated by the Sabayon maintainer.
> >
> > Those issues could be related to the latest versions of the libraries used
> > in Gentoo; so the question:
> > - did some of you test 389org on Fedora Linux or RHEL6?
> >
>
>  Yes.  I've just tested the latest 389-admin-1.1.11.rc1 and
>  389-ds-base-1.2.6.rc3 on Fedora rawhide (F-14)
>  This system uses httpd version 2.2.15 and apr 1.3.9
>  * yum install 389-ds
>  * setup-ds-admin.pl - use defaults
>  * launched the web browser - http://localhost:9830/ - everything works fine
>  - main page, topology page, log viewer, etc. (including the CGI the
>  maintainer was having problems with, or at least I'm assuming this is the
>  same problem I was working on IRC with lxnay last week)
>
>  The problem seems specific to Gentoo 
I've been able to track down the problem, also thanks to (you) and
people from httpd-dev.
It's either redhat/fedora packagers doing something at .spec level in
some library (it wouldn't be the first time) or there is something
error-prone in 389 codebase. Let me explain: httpd-devs suggested the
following change in password_pipe(): set PASSWORD_PIPE always to 1
(since the fd that PASSWORD_PIPE is pointing to is closed by mod_cgid
after a dup2() here http://pastebin.com/2yTS4MU5 at line 98. This
makes libadminutil able to read from PASSWORD_PIPE fd (and not getting
EBADF). Unfortunately, at that point of the execution "stack", the fd
which libadminutil (distadm.c ADM_InitializePermissions()) expects to
gather data from returns "", like if something reset the read buffer.
Looking at strace output, I am unable to locate where the pipe leaks,
so I'm going to divide the code into chunks and analyze each chunk
separately. The ultimate issue is just that: a leaking pipe.

It can be caused by several reasons, maybe some newer library
introduced a behavior change. I'm running almost "bleeding edge" here,
Gentoo ~amd64 arch on the test machine: I have to in order to commit
389-ds packages in the unstable "branch" of Portage.
I produced an insecure patch, that makes (almost, still have to work
out AdminUtil.pm) htmladmin working properly, just to make sure that
the problem stands in the pipe() handling code.
See:
http://gitweb.sabayon.org/?p=overlay.git;a=commit;h=8d483936b737314edcb83...
look for *PASSWORD_PIPE*.patch

There are two options for such leak to take place: either O_CLOEXEC is
set or some error after the fork() that could stand between mod_cgid.c
and distadm.c.

Here I have:
apache-2.2.15 (worker MPM) + mod_cgid
apr-1.4.2
apr-util-1.3.9
glibc-2.10.1
linux-2.6.34 (and also tested on 2.6.33)
latest 389 packages (including rc releases)

> >
> > Peace,
> > R.
> >
>
>  


-- 
Fabio Erculiani
http://www.sabayon.org
http://www.gentoo.org

[Rich Megginson]

Fabio Erculiani wrote:
>  On Tue, Jul 6, 2010 at 5:58 PM, Rich Megginson
> <rmeggins(a)redhat.com&gt; wrote:
>    
> > Roberto Polli wrote:
> >     
> >> Hi all,
> >> our company decided to sponsor a Sabayon/Gentoo distribution of 389org.
> >>
> >> It seems that there are some issues in admin-server, which have been
> >> deeply investigated by the Sabayon maintainer.
> >>
> >> Those issues could be related to the latest versions of the libraries used
> >> in Gentoo; so the question:
> >> - did some of you test 389org on Fedora Linux or RHEL6?
> >>
> >>       
> > Yes.  I've just tested the latest 389-admin-1.1.11.rc1 and
> > 389-ds-base-1.2.6.rc3 on Fedora rawhide (F-14)
> > This system uses httpd version 2.2.15 and apr 1.3.9
> > * yum install 389-ds
> > * setup-ds-admin.pl - use defaults
> > * launched the web browser - http://localhost:9830/ - everything works fine
> > - main page, topology page, log viewer, etc. (including the CGI the
> > maintainer was having problems with, or at least I'm assuming this is the
> > same problem I was working on IRC with lxnay last week)
> >
> > The problem seems specific to Gentoo
> >     
>
>  I've been able to track down the problem, also thanks to (you) and
>  people from httpd-dev.
>  It's either redhat/fedora packagers doing something at .spec level in
>  some library (it wouldn't be the first time) or there is something
>  error-prone in 389 codebase. Let me explain: httpd-devs suggested the
>  following change in password_pipe(): set PASSWORD_PIPE always to 1
>  (since the fd that PASSWORD_PIPE is pointing to is closed by mod_cgid
>  after a dup2() here http://pastebin.com/2yTS4MU5 at line 98. This
>  makes libadminutil able to read from PASSWORD_PIPE fd (and not getting
>  EBADF). Unfortunately, at that point of the execution "stack", the fd
>  which libadminutil (distadm.c ADM_InitializePermissions()) expects to
>  gather data from returns "", like if something reset the read buffer.
>  Looking at strace output, I am unable to locate where the pipe leaks,
>  so I'm going to divide the code into chunks and analyze each chunk
>  separately. The ultimate issue is just that: a leaking pipe.
>     
I'm not sure what the problem is.  If we look at lines 96-98 of 
http://pastebin.com/2yTS4MU5:
        else if (attr->child_out) {
           dup2(attr->child_out->filedes, STDOUT_FILENO); // dup2 9, 1
           apr_file_close(attr->child_out); // close 9

attr->child_out->filedes is 9 - this value is set at line 760 of mod_cgid.c:
            ((rc = apr_procattr_child_out_set(procattr, inout, NULL)) != 
APR_SUCCESS) ||
inout is set at line 733:
        apr_os_file_put(&inout, &sd2, 0, r->pool);
sd2 is set at line 683:
        sd2 = accept(sd, (struct sockaddr *)&unix_addr, &len);

Perhaps the httpd developers can tell me how it is possible for the fd 
returned by accept() to be the same fd as returned by line 623 in 
mod_admserv.c:
    rv = apr_file_pipe_create(&readp, &writep, r->pool);
?

The only way I know is that the pipe fds are close()d before the call to 
accept.  So why is this happening?  As you say, one way is that the fds 
are set to FD_CLOEXEC by default on Gentoo but not on RHEL/Fedora.  Not 
sure where/how that happens.  I've looked at the Fedora rawhide spec and 
patches for httpd and apr - I don't see where this could be happening.  
Note that there is no O_CLOEXEC on RHEL/Fedora, only FD_CLOEXEC (at 
least, when I do this
find /usr/include -type f -exec grep CLOEXEC {} /dev/null \;
I only see FD_CLOEXEC)

Another option is that
    apr_pool_cleanup_register(r->pool, (void *)((intptr_t)fd), close_pipe,
                                      apr_pool_cleanup_null);
makes the close_pipe callback happen before mod_cgid has a chance to 
execute the CGI.  Could the httpd devs explain how this could happen?

The main thing is that password_pipe creates a pipe - no one else should 
be closing that pipe.
>  It can be caused by several reasons, maybe some newer library
>  introduced a behavior change. I'm running almost "bleeding edge" here,
>  Gentoo ~amd64 arch on the test machine: I have to in order to commit
>  389-ds packages in the unstable "branch" of Portage.
>  I produced an insecure patch, that makes (almost, still have to work
>  out AdminUtil.pm) htmladmin working properly, just to make sure that
>  the problem stands in the pipe() handling code.
>  See:
> http://gitweb.sabayon.org/?p=overlay.git;a=commit;h=8d483936b737314edcb83...
>  look for *PASSWORD_PIPE*.patch
>
>  There are two options for such leak to take place: either O_CLOEXEC is
>  set or some error after the fork() that could stand between mod_cgid.c
>  and distadm.c.
>
>  Here I have:
>  apache-2.2.15 (worker MPM) + mod_cgid
>  apr-1.4.2
>  apr-util-1.3.9
>  glibc-2.10.1
>  linux-2.6.34 (and also tested on 2.6.33)
>  latest 389 packages (including rc releases)
>
>    
> >> Peace,
> >> R.
> >>
> >>       
> >     
>
>
>
>

[Fabio Erculiani]

At last, I've been able to figure out where the pipe() problem stands.
389-admin is improperly using the cgi module. When using a
multithreaded MPM (worker), mod_cgid had to be used. It happens that
389-admin needs apache (with worker MPM) and mod_cgi (and not
mod_cgid).
mod_cgid fork() itself from the main httpd process while mod_cgi
doesn't. That's why password_pipe() works fine with the latter.
Unfortunately, when you tell the build system that you want to use the
worker mpm, it completely ignores your request (via --enable-cgi
switch) of using mod_cgi (instead of mod_cgid) and always builds
mod_cgid (despite the .so is called mod_cgi.so).
In binary distributions that over-split packages, both apache#worker
and apache#prefork are built separately and so their modules. This
makes possible to install mod_cgi into a system that is using
apache#worker, otherwise impossible (when building from sources). So,
this masks the issue above in a perfect and (sadly) elegant way.

Besides, this also explains why I was getting Scriptsock socket path
conflict between mod_cgid and mod_restartd. See
/etc/dirsrv/admin-serv/httpd.conf around line 395
(http://pastebin.com/f0K49cL5). Only mod_cgid uses that directive.
Unfortunately (or fortunately) the potential directive error was
masked by the fact that mod_restartd uses the same directive name, so
if mod_cgid is not loaded, the line, from a config POV, is still
valid.

I think 389-ds should either do:
- fix the pipe() issue (finding an alternative way of sending password
data over) and let use apache#worker in a proper way (with mod_cgid as
per docs -- http://httpd.apache.org/docs/2.2/mod/mod_cgid.html): best
option from code quality POV
- find a way to force building apache#worker with mod_cgi (worse option)

Yeah, that is the final puzzle :-)

Regards,
-- 
Fabio Erculiani
http://www.sabayon.org
http://www.gentoo.org

[Fabio Erculiani]

I just tried to change how our (Gentoo) build script works [1] in
regard to mod_cgi[d] specifying both --enable-cgi=shared
--enable-cgid=shared and it seems to work. Apache build system is
producing both .so. So from my POV, it's all fine now. Maybe the
port389.org project should make 389-admin working with the appropriate
cgi module (mod_cgid) in any case or document this case on port389.org
wiki.

[1]
http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/eclass/apache-2.e...
line 249


-- 
Fabio Erculiani
http://www.sabayon.org
http://www.gentoo.org

[Rich Megginson]

Fabio Erculiani wrote:
>  I just tried to change how our (Gentoo) build script works [1] in
>  regard to mod_cgi[d] specifying both --enable-cgi=shared
>  --enable-cgid=shared and it seems to work. Apache build system is
>  producing both .so. So from my POV, it's all fine now. Maybe the
>  port389.org project should make 389-admin working with the appropriate
>  cgi module (mod_cgid) in any case or document this case on port389.org
>  wiki.
>
>  [1]
> http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/eclass/apache-2.e...
>  line 249
>
>     
Please file bugs/RFEs at 
https://bugzilla.redhat.com/enter_bug.cgi?product=389

[Fabio Erculiani]

On Wed, Jul 7, 2010 at 4:45 PM, Rich Megginson <rmeggins(a)redhat.com&gt; wrote:
>  Please file bugs/RFEs at
>  https://bugzilla.redhat.com/enter_bug.cgi?product=389 
Ok will do, thanks.

-- 
Fabio Erculiani
http://www.sabayon.org
http://www.gentoo.org