Openblas, OpenMP, and R is there a decent test?

Question

I am trying to setup a multithreaded R with Openblas and OpenMP. I am using OpenSuSE 12.2 with an AMD fx-8230 8-core processor. After fighting awhile with ATLAS it was suggested that I bag it and try openblas, which I have.

First. There was some reports of opensuse 12.2 gcc having a broken openmp, so I figured I should test it. I went to http://openmp.org/wp/openmp-compilers/ and compiled and executed the example file hello.c with all threads responding.

Second. I set up a git clone of openblas. I read the instructions and executed 'make USE_OPENMP=1' followed by 'make PREFIX=/usr/lib64/OpenBLAS install' The program installed and passed all checks with no problem.

Third. Setting up R. I downloaded R as a tar file and unpacked. I will eventually check out the subversion R. I used this configure command ../configure --prefix=/usr/lib64/R --enable-openmp --enable-R-shlib --with-blas="-L/opt/maths/OpenBLAS -lopenblas" --with-tcl-config=/usr/lib64/tclConfig.sh --with-tk-config=/usr/lib64/tkConfig.sh --with-x

The prefix and -with-blas information doesn't make it into the Makeconf file. Why is that? I changed the Makeconf file to:

PACKAGE = R
VERSION = 2.15.3

abs_top_srcdir = /opt/maths/R-2.15.3/gserver/..
abs_top_builddir = /opt/maths/R-2.15.3/gserver

include $(top_srcdir)/share/make/vars.mk

AR = ar
ACLOCAL = aclocal
AUTOCONF = autoconf
AUTOMAKE = automake
AUTOHEADER = autoheader
BLAS_LIBS = -L/opt/maths/OpenBLAS/ -lopenblas
BUILDDIR_IS_SRCDIR = no
## next two for future support of cross-compiling, not actually used
BUILD_CC =
BUILD_R =
CC = gcc -std=gnu99
CFLAGS = -g -O2 -fopenmp
CPICFLAGS = -fpic
CPPFLAGS = -I/usr/include
DEFS = -DHAVE_CONFIG_H
DISTDIR_TAR_EXCLUDE = --exclude=.svn --exclude=Makefile --exclude="*.o" --exclude="*$(SHLIB_EXT)" --exclude="*~"
DYLIB_EXT = .so
DYLIB_LD = gcc -std=gnu99
DYLIB_LDFLAGS = -shared -fopenmp# $(DYLIBS_LTO)
DYLIB_LINK = $(DYLIB_LD) $(DYLIB_LDFLAGS) $(LDFLAGS)
DYLIBS_LTO = $(CFLAGS) $(CPICFLAGS)
ECHO = echo
ECHO_C =
ECHO_N = -n
ECHO_T =
FFLAGS = -g -O2 -fopenmp
FLIBS =  -lgfortran -lm -lquadmath
FLIBS_IN_SO =  -lgfortran -lm -lquadmath
FPICFLAGS = -fpic
F77 = gfortran
GETWD = /usr/bin/pwd
GZIP = --best
INSTALL = /usr/bin/install -c
INSTALL_DATA = ${INSTALL} -m 644
INSTALL_PROGRAM = ${INSTALL}
INSTALL_SCRIPT = ${INSTALL}
INSTALL_DIR = ${INSTALL} -d
LDFLAGS = -L/usr/lib64
LIBINTL=
LIBM = -lm
LIBR = -L$(R_HOME)/lib$(R_ARCH) -lR
LIBS =  -lrt -ldl -lm
LIBnn = lib64
LIBTOOL = $(SHELL) $(top_builddir)/libtool
## AFAICS unused
LN_S = ln -s
MAIN_CFLAGS =
MAIN_FFLAGS =
MAIN_LD = gcc -std=gnu99# $(CFLAGS)
MAIN_LDFLAGS = -Wl,--export-dynamic -fopenmp # # -Wl,-bE:$(top_builddir)/etc/R.exp
MAIN_LINK = $(MAIN_LD) $(MAIN_LDFLAGS) $(LDFLAGS)
## need this for bootstrapping
MKINSTALLDIRS = /bin/sh $(top_srcdir)/src/scripts/mkinstalldirs.in
NOTANGLE = false
R_ARCH =
R_FRAMEWORK_DIR = $(prefix)/R.framework
R_GZIPCMD = /usr/bin/gzip
## needed for AIX only
# R_HOME = $(top_builddir)
R_OPENMP_CFLAGS = -fopenmp
R_OPENMP_FFLAGS = -fopenmp
R_OSTYPE = unix
R_PKGS = $(R_PKGS_BASE)  $(R_PKGS_RECOMMENDED)
R_PLATFORM = x86_64-unknown-linux-gnu
R_XTRA_CFLAGS =
R_XTRA_CPPFLAGS =  -I. -I$(top_builddir)/src/include -I$(top_srcdir)/src/include
R_XTRA_FFLAGS =
R_XTRA_LIBS =
RANLIB = ranlib
READLINE_LIBS = -lreadline
SED = /usr/bin/sed
SHELL = /bin/sh
SHLIB_EXT = .so
SHLIB_CFLAGS =
SHLIB_FFLAGS =
SHLIB_LD = gcc -std=gnu99
SHLIB_LDFLAGS = -shared# $(SHLIB_LTO)
SHLIB_LINK = $(SHLIB_LD) $(SHLIB_LDFLAGS) $(LDFLAGS)
SHLIB_LTO = $(CFLAGS) $(CPICFLAGS)
STRIP_LIBS = strip --strip-unneeded
STRIP_STATIC_LIBS = strip --strip-debug
TAR = /bin/tar
USE_NLS = yes
X_CFLAGS =
X_LIBS =  -lX11 -lXt -lXmu
X_PRE_LIBS =  -lSM -lICE
X_EXTRA_LIBS =
YACC = bison -y

ALL_CFLAGS = $(R_XTRA_CFLAGS) $(R_OPENMP_CFLAGS) $(MAIN_CFLAGS) $(CFLAGS)
ALL_CPPFLAGS = $(R_XTRA_CPPFLAGS) $(CPPFLAGS) $(DEFS)
ALL_FFLAGS = $(R_XTRA_FFLAGS) $(R_OPENMP_FFLAGS) $(MAIN_FFLAGS) $(FFLAGS)
ALL_CFLAGS_LO = $(R_XTRA_CFLAGS) $(R_OPENMP_CFLAGS) $(CPICFLAGS) $(SHLIB_CFLAGS) $(CFLAGS)
ALL_FFLAGS_LO = $(R_XTRA_FFLAGS) $(R_OPENMP_FFLAGS) $(FPICFLAGS) $(SHLIB_FFLAGS) $(FFLAGS)

.SUFFIXES:
.SUFFIXES: .c .f .m .d .o

.c.o:
        $(CC) $(ALL_CPPFLAGS) $(ALL_CFLAGS) -c $< -o $@
.c.d:
        @echo "making $@ from $<"
        @gcc -std=gnu99 -MM $(ALL_CPPFLAGS) $< > $@
.m.d:
        @echo > $@
.f.o:
        $(F77) $(ALL_FFLAGS) -c $< -o $@

prefix = /usr/lib64/R
exec_prefix = ${prefix}
datarootdir = ${prefix}/share
## only used for installing 'R'.
bindir = ${exec_prefix}/bin
## not used
datadir = ${datarootdir}
## used for 'rhome' and installation of standalone Rmath
libdir = ${exec_prefix}/${LIBnn}
## used for man page
mandir = ${datarootdir}/man
## used for installation of standalone Rmath headers
includedir = ${prefix}/include

rhome = ${libdir}/R
rsharedir = ${rhome}/share
rincludedir = ${rhome}/include
rdocdir = ${rhome}/doc

## Overrides for installing R as a framework (MacOS X).
#FW_VERSION =
#rhome = $(R_FRAMEWORK_DIR)/Versions/$(FW_VERSION)/Resources
#bindir = $(rhome)
#mandir = $(rhome)

Rexecbindir = $(rhome)/bin
Rexecbindir2 = $(rhome)/bin/exec$(R_ARCH)
Rexeclibdir = $(rhome)/lib$(R_ARCH)
## FIXME:
## Alternatively, we could try to set pkglibdir = $(rhome)/lib when
## switching to automake.
## </FIXME>
Rexecmodulesdir = $(rhome)/modules$(R_ARCH)

Which still couldn't find open blas. I made a symbolic link into /usr/lib64 of libopenblas.so.0 and then openblas was found. Is there a better way than -I to set include? Maybe I am not properly setting include and it is not finding the right files?

Using the information here I checked the libraries being found by R in /bin/exec/R

gserver/bin/exec # ldd R
    linux-vdso.so.1 (0x00007fff4b278000)
    libR.so => not found
    libRblas.so => not found
    libgomp.so.1 => /usr/lib64/libgomp.so.1 (0x00007f2cb3e00000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f2cb3be0000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f2cb3830000)
    librt.so.1 => /lib64/librt.so.1 (0x00007f2cb3628000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f2cb4010000)

I expected that it would not find libRblas if it isn't using it. So I found libR.so and

gserver/lib # ldd libR.so
    linux-vdso.so.1 (0x00007fffaf360000)
    libopenblas.so.0 => /usr/lib64/libopenblas.so.0 (0x00007fe131158000)
    libgfortran.so.3 => /usr/lib64/libgfortran.so.3 (0x00007fe130e40000)
    libm.so.6 => /lib64/libm.so.6 (0x00007fe130b40000)
    libquadmath.so.0 => /usr/lib64/libquadmath.so.0 (0x00007fe130908000)
    libreadline.so.6 => /lib64/libreadline.so.6 (0x00007fe1306c0000)
    librt.so.1 => /lib64/librt.so.1 (0x00007fe1304b8000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007fe1302b0000)
    libgomp.so.1 => /usr/lib64/libgomp.so.1 (0x00007fe1300a0000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe12fe80000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fe12fad0000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fe132610000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fe12f8b8000)
    libtinfo.so.5 => /lib64/libtinfo.so.5 (0x00007fe12f680000)

Following the rest of the R installation instructions I did make check (passed) make check-all and it passed.

Ok, so now's the kicker. How can I test to see that R is indeed using more than one core. I was able to find a scheme for testing in R:

a = matrix(rnorm(5000*5000), 5000, 5000)
b = matrix(rnorm(5000*5000), 5000, 5000)
c = a%*%b

Then check cpu with 'top'. I did this, but it never used more than 100.1% of a cpu! This made me pretty sad. I did see when I was running make-all that when the examples were being tested that there was sometimes more than 250% cpu usage which makes me think that it is working and I am forgetting something. In my profile I set 'export OMP_NUM_THREADS=8' and 'export OPEN_BLAS_NUM_THREADS=8' Does this mean that openblas is not working with openmp and how can I fix it?

This server doesn't really have anything set, if I echo $LD_LIBRARY_PATH nothing comes up, so maybe there is something that I need to set there in order for everyone to work together? My eventual goal is to install the boost libraries, snow and spp. Does anyone have experience setting up openmp with snow in the R package, will it work?

Answer 1

The first thing to find out would be if

• the optimized BLAS is used at all
  OpenBLAS is already with NUM_TREADS=1 much faster than the default BLAS. Check the times of the m %*% m multiplication.

• Once you know that OpenBLAS is used, check the number of threads spawned ( top or htop )

• If the optimized BLAS is used, NUM_TREADS threads are spawned but get all executed on the same core, see here: Parallel processing in R limited

Answer 2

The problem with not seeing the expected performance gains from switching to OpenBLAS, may be related to processor affinity. From OpenBLAS Github:

"On Linux, OpenBLAS sets the processor affinity by default. This may cause the conflict with R parallel. You can build the library with NO_AFFINITY=1."

Compiling with NO_AFFINITY=1 flag, disables processor affinity.

https://stat.ethz.ch/pipermail/r-sig-hpc/2012-April/001348.html