apitrace: undefined reference to `__libc_dlopen_mode', `__libc_dlsym'

[Sandro Mani]

Hi

I'd need help with the following issue with apitrace, which failed the 
mass rebuild with:

apitrace-9d42f667e2a36a6624d92b9bd697de097cc4e619/wrappers/dlsym.cpp:70: 
undefined reference to `__libc_dlopen_mode'
apitrace-9d42f667e2a36a6624d92b9bd697de097cc4e619/wrappers/dlsym.cpp:72: 
undefined reference to `__libc_dlsym'

The code triggering this is below. Have these symbols disappeared from 
libc resp is there any alternative?

Thanks
Sandro

-----

/*
  * Protect against dlsym interception.
  *
  * We implement the whole API, so we don't need to intercept dlsym -- 
dlopen is
  * enough. However we need to protect against other dynamic libraries
  * intercepting dlsym, to prevent infinite recursion,
  *
  * In particular the Steam Community Overlay exports dlsym.  See also
  * http://lists.freedesktop.org/archives/apitrace/2013-March/000573.html
  */
PRIVATE
void *
dlsym(void * handle, const char * symbol)
{
     /*
      * We rely on glibc's internal __libc_dlsym.  See also
      * http://www.linuxforu.com/2011/08/lets-hook-a-library-function/
      *
      * Use use it to obtain the true dlsym.  We don't use __libc_dlsym 
directly
      * because it does not support things such as RTLD_NEXT.
      */
     typedef void * (*PFN_DLSYM)(void *, const char *);
     static PFN_DLSYM dlsym_ptr = NULL;
     if (!dlsym_ptr) {
         void *libdl_handle = __libc_dlopen_mode("libdl.so.2", 
RTLD_LOCAL | RTLD_NOW);
         if (libdl_handle) {
             dlsym_ptr = (PFN_DLSYM)__libc_dlsym(libdl_handle, "dlsym");
         }
         if (!dlsym_ptr) {
             os::log("apitrace: error: failed to look up real dlsym\n");
             return NULL;
         }
     }

     return dlsym_ptr(handle, symbol);
}

[Florian Weimer]

* Sandro Mani:

>  I'd need help with the following issue with apitrace, which
> failed the
>  mass rebuild with:
>
>  apitrace-9d42f667e2a36a6624d92b9bd697de097cc4e619/wrappers/dlsym.cpp:70:
>  undefined reference to `__libc_dlopen_mode'
>  apitrace-9d42f667e2a36a6624d92b9bd697de097cc4e619/wrappers/dlsym.cpp:72:
>  undefined reference to `__libc_dlsym'
>
>  The code triggering this is below. Have these symbols disappeared from
>  libc resp is there any alternative? 
These internal GLIBC_PRIVATE symbols are gone and won't come back.

Not sure why this code isn't using RTLD_NEXT.

We are working on improving LD_AUDIT support, which is the plugin API of
the dynamic loader.  It can in theory be used to implement such things.
But API tracing is probably best implemented within the traced libraries
themselves, rather than on the side.

Thanks,
Florian

[John Reiser]

>  We are working on improving LD_AUDIT support, which is the plugin API
> of
>  the dynamic loader.  It can in theory be used to implement such things. 
Depending on the threat model, then DT_BIND_NOW and/or LD_PRELOAD
can be used to find the desired instance of dlsym().  If the threats include
"just in time" modification of the instruction stream, then you must control
the system calls execve, mmap, mprotect, and ptrace.

--

[John Reiser]

>  Depending on the threat model, then DT_BIND_NOW and/or LD_PRELOAD
>  can be used to find the desired instance of dlsym(). 
Also there is dl_iterate_phdr() (declared in /usr/include/link.h)
which enables examination of all the symbols known to the runtime linker.

[Nicholas Miell]

On 7/25/21 4:28 AM, Florian Weimer wrote:
>  * Sandro Mani:
>  
> > I'd need help with the following issue with apitrace, which failed the
> > mass rebuild with:
> >
> > apitrace-9d42f667e2a36a6624d92b9bd697de097cc4e619/wrappers/dlsym.cpp:70:
> > undefined reference to `__libc_dlopen_mode'
> > apitrace-9d42f667e2a36a6624d92b9bd697de097cc4e619/wrappers/dlsym.cpp:72:
> > undefined reference to `__libc_dlsym'
> >
> > The code triggering this is below. Have these symbols disappeared from
> > libc resp is there any alternative?
>  
>  These internal GLIBC_PRIVATE symbols are gone and won't come back.
>  
>  Not sure why this code isn't using RTLD_NEXT.
>   
Because they're interposing dlsym itself and attempting to call dlsym(RTLD_NEXT, ...)
resolves to the interposed dlsym function and dies due to infinite recursion.

>  We are working on improving LD_AUDIT support, which is the plugin API
> of
>  the dynamic loader.  It can in theory be used to implement such things.
>  But API tracing is probably best implemented within the traced libraries
>  themselves, rather than on the side.
>   
This isn't viable when you have multiple libraries from different vendors, some
or all of which are beyond your control.

[Florian Weimer]

* Nicholas Miell:

>  On 7/25/21 4:28 AM, Florian Weimer wrote:
> > * Sandro Mani:
> > 
> >> I'd need help with the following issue with apitrace, which failed the
> >> mass rebuild with:
> >>
> >> apitrace-9d42f667e2a36a6624d92b9bd697de097cc4e619/wrappers/dlsym.cpp:70:
> >> undefined reference to `__libc_dlopen_mode'
> >> apitrace-9d42f667e2a36a6624d92b9bd697de097cc4e619/wrappers/dlsym.cpp:72:
> >> undefined reference to `__libc_dlsym'
> >>
> >> The code triggering this is below. Have these symbols disappeared from
> >> libc resp is there any alternative?
> > 
> > These internal GLIBC_PRIVATE symbols are gone and won't come back.
> > 
> > Not sure why this code isn't using RTLD_NEXT.
> > 
>
>  Because they're interposing dlsym itself and attempting to call
>  dlsym(RTLD_NEXT, ...)  resolves to the interposed dlsym function and
>  dies due to infinite recursion. 
Oops, I see.

> > We are working on improving LD_AUDIT support, which is the plugin
> API of
> > the dynamic loader.  It can in theory be used to implement such things.
> > But API tracing is probably best implemented within the traced libraries
> > themselves, rather than on the side.
> > 
>
>  This isn't viable when you have multiple libraries from different
>  vendors, some or all of which are beyond your control. 
Which part?  The tracing-in-libraries aspect?

LD_AUDIT shouldn't have this problem.

Dyninst and Systemtap might alos be worth investigation.

Thanks,
Florian

[Sandro Mani]

On 25.07.21 12:51, Sandro Mani wrote:
>  Hi
>
>  I'd need help with the following issue with apitrace, which failed the 
>  mass rebuild with:
>
>  apitrace-9d42f667e2a36a6624d92b9bd697de097cc4e619/wrappers/dlsym.cpp:70: 
>  undefined reference to `__libc_dlopen_mode'
>  apitrace-9d42f667e2a36a6624d92b9bd697de097cc4e619/wrappers/dlsym.cpp:72: 
>  undefined reference to `__libc_dlsym'
>
>  The code triggering this is below. Have these symbols disappeared from 
>  libc resp is there any alternative? 
I've raised this upstream but so far no activity [1]. I'm not really 
knowledgeable enough in this area to judge the best way to fix this. Is 
anyone able to help with this? Otherwise I'll have to orphain/retire 
apitrace for F35+.

Thanks
Sandro

[1] https://github.com/apitrace/apitrace/issues/756

[John Reiser]

> > I'd need help with the following issue with apitrace, which
> failed the mass rebuild with:
> >
> > apitrace-9d42f667e2a36a6624d92b9bd697de097cc4e619/wrappers/dlsym.cpp:70: undefined
> reference to `__libc_dlopen_mode'
> > apitrace-9d42f667e2a36a6624d92b9bd697de097cc4e619/wrappers/dlsym.cpp:72: undefined
> reference to `__libc_dlsym'
> >
> > The code triggering this is below. Have these symbols disappeared from libc resp is
> there any alternative?
>  
>  I've raised this upstream but so far no activity [1]. I'm not really
> knowledgeable enough in this area to judge the best way to fix this. Is anyone able to
> help with this? Otherwise I'll have to orphain/retire apitrace for F35+. 
According to the comments in https://github.com/apitrace/apitrace.git file
wrappers/dlsym.cpp ,
the purpose is "to obtain the true dlsym" by explicit lookup in libdl.so.2,
which is a library
that no longer exists in glibc-2.34, having been combined into libc.so itself.

The only legitimate way to find "the true dlsym" is to trust dl_iterate_phdr
(/usr/include/link.h)
and call it, dig through all the Dynamic sections to find all the symbols named
'dlsym',
then choose the one you want: perhaps by being defined in a file whose DT_SONAME is
"libc.so"
and having symbol version GLIBC_2.2.5 .

Because such code has not been contributed, then apitrace should be orphaned/retired in
F35+.

[Sandro Mani]

On 16.08.21 05:14, John Reiser wrote:
> >> I'd need help with the following issue with apitrace,
> which failed 
> >> the mass rebuild with:
> >>
> >> apitrace-9d42f667e2a36a6624d92b9bd697de097cc4e619/wrappers/dlsym.cpp:70: 
> >> undefined reference to `__libc_dlopen_mode'
> >> apitrace-9d42f667e2a36a6624d92b9bd697de097cc4e619/wrappers/dlsym.cpp:72: 
> >> undefined reference to `__libc_dlsym'
> >>
> >> The code triggering this is below. Have these symbols disappeared 
> >> from libc resp is there any alternative?
> >
> > I've raised this upstream but so far no activity [1]. I'm not really 
> > knowledgeable enough in this area to judge the best way to fix this. 
> > Is anyone able to help with this? Otherwise I'll have to 
> > orphain/retire apitrace for F35+.
>
>  According to the comments in https://github.com/apitrace/apitrace.git 
>  file wrappers/dlsym.cpp ,
>  the purpose is "to obtain the true dlsym" by explicit lookup in 
>  libdl.so.2, which is a library
>  that no longer exists in glibc-2.34, having been combined into libc.so 
>  itself.
>
>  The only legitimate way to find "the true dlsym" is to trust 
>  dl_iterate_phdr (/usr/include/link.h)
>  and call it, dig through all the Dynamic sections to find all the 
>  symbols named 'dlsym',
>  then choose the one you want: perhaps by being defined in a file whose 
>  DT_SONAME is "libc.so"
>  and having symbol version GLIBC_2.2.5 .
>
>  Because such code has not been contributed, then apitrace should be 
>  orphaned/retired in F35+. 
Thanks for your reply - I've forwarded your comment upstream. The 
package will be retired in ca 5 weeks if no solution presents itself 
until then.

Sandro