What are _nocancel() system calls in linux, and is there a way to use LD_PRELOAD to intercept them

Question

Please let me know what are the _nocancel() system calls (eg __pwrite_nocancel(), and whether there is a way to create an LD_PRELOAD library to intercept those calls. Here's a bit of the background:

I'm investigating functionality of an Oracle database, and would like to add a small shim layer using LD_PRELOAD to capture information about the call in user space. I'm aware of other ways of capturing this information using system tap, but using LD_PRELOAD is a hard requirement from the customer. strace shows that this particular process is calling pwrite() repeatedly; similarly, the pstack stack trace shows that __pwrite_nocancel() is being called as the last entry on the stack. I tried reproducing my own __libc_pwrite() function, and declaring extern ssize_t pwrite(int fd, const void *buf, size_t numBytes, off_t offset)__attribute__((weak, alias ( "__libc_pwrite"))); but when I link the library and run nm -a |grep pwrite, I get this:

000000000006c190 T __libc_pwrite
000000000006c190 W pwrite

in contrast, nm -a /lib64/libpthread.so.0 |grep pwrite gives the following:

000000000000eaf0 t __libc_pwrite
000000000000eaf0 t __libc_pwrite64
000000000000eaf0 W pwrite
000000000000eaf0 t __pwrite
000000000000eaf0 W pwrite64
000000000000eaf0 W __pwrite64
0000000000000000 a pwrite64.c
000000000000eaf9 t __pwrite_nocancel

I've noticed that the _nocancel version is only 9 bytes ahead of the __pwrite, but looking at the source code, I'm unsure as to where it's being created:

/* Copyright (C) 1997, 1998, 2000, 2002, 2003, 2004, 2006
   Free Software Foundation, Inc.
   This file is part of the GNU C Library.
   Contributed by Ulrich Drepper <drepper@cygnus.com>, 1997.
   The GNU C Library is free software; you can redistribute it and/or
   modify it under the terms of the GNU Lesser General Public
   License as published by the Free Software Foundation; either
   version 2.1 of the License, or (at your option) any later version.
   The GNU C Library is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
   Lesser General Public License for more details.
   You should have received a copy of the GNU Lesser General Public
   License along with the GNU C Library; if not, see
   <http://www.gnu.org/licenses/>.  */

#include <assert.h>
#include <errno.h>
#include <unistd.h>
#include <endian.h>

#include <sysdep-cancel.h>
#include <sys/syscall.h>
#include <bp-checks.h>

#include <kernel-features.h>

#ifdef __NR_pwrite64            /* Newer kernels renamed but it's the same.  */
# ifdef __NR_pwrite
#  error "__NR_pwrite and __NR_pwrite64 both defined???"
# endif
# define __NR_pwrite __NR_pwrite64
#endif

#if defined __NR_pwrite || __ASSUME_PWRITE_SYSCALL > 0

# if __ASSUME_PWRITE_SYSCALL == 0
static ssize_t __emulate_pwrite (int fd, const void *buf, size_t count,
                 off_t offset) internal_function;
# endif

ssize_t
__libc_pwrite (fd, buf, count, offset)
     int fd;
     const void *buf;
     size_t count;
     off_t offset;
{
  ssize_t result;

  if (SINGLE_THREAD_P)
    {
      /* First try the syscall.  */
      result = INLINE_SYSCALL (pwrite, 6, fd, CHECK_N (buf, count), count, 0,
                   __LONG_LONG_PAIR (offset >> 31, offset));
# if __ASSUME_PWRITE_SYSCALL == 0
      if (result == -1 && errno == ENOSYS)
        /* No system call available.  Use the emulation.  */
        result = __emulate_pwrite (fd, buf, count, offset);
# endif
      return result;
    }

  int oldtype = LIBC_CANCEL_ASYNC ();

  /* First try the syscall.  */
  result = INLINE_SYSCALL (pwrite, 6, fd, CHECK_N (buf, count), count, 0,
               __LONG_LONG_PAIR (offset >> 31, offset));
# if __ASSUME_PWRITE_SYSCALL == 0
  if (result == -1 && errno == ENOSYS)
    /* No system call available.  Use the emulation.  */
    result = __emulate_pwrite (fd, buf, count, offset);
# endif

  LIBC_CANCEL_RESET (oldtype);

  return result;
}

strong_alias (__libc_pwrite, __pwrite)
weak_alias (__libc_pwrite, pwrite)

# define __libc_pwrite(fd, buf, count, offset) \
     static internal_function __emulate_pwrite (fd, buf, count, offset)
#endif

#if __ASSUME_PWRITE_SYSCALL == 0
# include <sysdeps/posix/pwrite.c>
#endif

Any help is appreciated.

Answer 1

The pwrite_nocancel() et cetera are not syscalls in Linux. They are functions internal to the C library, and tightly coupled with pthreads and thread cancellation.

The _nocancel() versions behave exactly the same as the original functions, except that these versions are not thread cancellation points.

Most I/O functions are cancellation points. That is, if the thread cancellation type is deferred and cancellation state is enabled , and another thread in the process has requested the thread to be cancelled, the thread will cancel (exit) when entering a cancellation point. See man 3 pthread_cancel , man 3 pthread_setcancelstate , and man 3 pthread_setcanceltype for further details.

Unfortunately, the pwrite_nocancel() and other _nocancel() functions are internal (local) to the pthreads library, and as such, are quite difficult to interpose; they're not dynamic symbols, so the dynamic linker cannot override them. At this point, I suspect, but am not sure, that the way to interpose them would involve rewriting the beginning of the library code with a direct jump to your own code.

If they were exported (global) functions, they could be interposed just like any other library function (these are provided by the pthread library, libpthread ), using the normal approach. (Of my own answers here, you might find this , this , and this informative. Otherwise, just search for LD_PRELOAD example .)

Answer 2

Given that you want to interpose via LD_PRELOAD , you need to look at the same symbols rtld is using. Therefore, instead of nm -a , use objdump -T . If the symbol you want is not listed there, you cannot interpose it. All exported glibc symbols have a "version" string that is compared by rtld and allows having multiple versions of the same symbol (this is why even nm -D is not useful -- it doesn't show the versions). To interpose something, you should have the same version string. This can be done using a version script (see info ld ), the .symver asm directive or a macro that invokes that directive. Note that symbols with a "private" version may change at any new revision of the libc, and that other symbols may be superseded by new ones at any new version (the symbols still work, but applications compiled against the new version don't use them anymore).