C ++ runtime_error捕获在群集节点之间不一致

Question

Problem

I am trying to set up a c++ program to run on a Redhat scientific linux (v5.11) cluster. I have been able to compile the software and it runs flawlessly on the head node, however it crashes when run on any worker nodes.

I have traced the issue to part of the code where if some conditions return false, a std::runtime_error is thrown. This is deliberate as when the software is running correctly, this is caught and it continues to iterate. On the worker node the software aborts when the error is first thrown. The abort code and backtrace is shown below.

As it works on one node but not the others my guess is that this is an issue of gcc versions. To compile I had to yum install devtoolset-2 and built the software using gcc 4.8.2 (Red Hat 4.8.2-15) as the system gcc 4.1.2 (Red Hat 4.1.2-55) was too old to correctly compile. When I launch the application on both nodes I have the following:

which gcc > /opt/rh/devtoolset-2/root/usr/bin/gcc
which c++ > /opt/rh/devtoolset-2/root/usr/bin/c++
which g++ > /opt/rh/devtoolset-2/root/usr/bin/g++
which gfortran > /opt/rh/devtoolset-2/root/usr/bin/gfortran
$LD_LIBRARY_PATH > /opt/rh/devtoolset-2/root/usr/lib64:/opt/rh/devtoolset-2/root/usr/lib

In terms of the differences between head and worker nodes, they are identical except in their kernel version:

• Head: Linux address.com 2.6.18-419.el5 #1 SMP x86_64 x86_64 x86_64 GNU/Linux
• Worker: Linux address.com 2.6.18-164.11.1.el5 #1 SMP x86_64 x86_64 x86_64 GNU/Linux

Things I have tired:

• Running on worker nodes using SGE queue submission (with -V to pass environment)
• Running directly on worker nodes via ssh worker and exporting all the same environment variables as on the head node
• Compiling and running on the worker nodes

Any help would be greatly appreciated! Here are a couple questions that I think getting answers to would help me narrow down the cause:

1. Is it work pursuing the kernel version difference?
2. Does this look like an issue with libraries and paths rather than the code?
3. Has the way in which c++ error handling changed between library versions?
4. Are there more debugging methods I could try to find the cause of this?

Extra info

The abort is as follows:

terminate called after throwing an instance of 'std::runtime_error'
  what():  'custom error message'

Program received signal SIGABRT, Aborted.
0x00000038b6830265 in raise () from /lib64/libc.so.6

The backtrace is as follows:

#0  0x00000038b6830265 in raise () from /lib64/libc.so.6
#1  0x00000038b6831d10 in abort () from /lib64/libc.so.6
#2  0x00000038bb0bec44 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib64/libstdc++.so.6
#3  0x00000038bb0bcdb6 in ?? () from /usr/lib64/libstdc++.so.6
#4  0x00000038bb0bcde3 in std::terminate() () from /usr/lib64/libstdc++.so.6
#5  0x00000038bb0bceca in __cxa_throw () from /usr/lib64/libstdc++.so.6
#6  0x00002aaaab074bdc in Some::Function::Name() () from path/to/file.so

I must admit my knowledge of c++ is pretty limited, though iv been trying to improve over the last two days iv spent battling with this problem. Here is a simplified example of the code that throws and catches the error (This is obviously part of a much larger process that repeatedly calls Func1 ):

double Func1(int a, double b, int c)
  {

  for (bool OK = true ; OK && d > e && f < a ; f++)
    {
    try
      {
      for (d = 0, g = 1 ; g < 10 ; g *= 2)
        {
        Func2() ;
        }
      }
    catch (runtime_error problem)
      {
      *log << problem.what() ;
      OK = false ;
      }
    if (c > 1)
      {
      *log << f << d;
      }
    }

void Func2()
  {
  for (int j = 0 ; j < ny && (x & 5) > 0 ; j++)
    {
    if (Func3(j) <= 0.0)
      {
      throw runtime_error("custom error message") ;
      }
    Func4[j] = j ;
    }
  }

Running ldd on the compiled program (run on head node, line 1 missing on worker node):

linux-vdso.so.1 =>  (0x00007fff2b6e7000)
/users/username/software/version/Part1/Part1Extra.so (0x00002b3543587000)
libgfortran.so.3 => /usr/lib64/libgfortran.so.3 (0x00002b354385b000)
libm.so.6 => /lib64/libm.so.6 (0x0000003cc2000000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x000000315f800000)
libc.so.6 => /lib64/libc.so.6 (0x0000003cc1c00000)
/users/username/software/version/Part2/Part2.so (0x00002b3543b4f000)
/users/username/software/version/Part3/Part3.so (0x00002b3543d9b000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x0000003160000000)
/lib64/ld-linux-x86-64.so.2 (0x0000003cc1800000)
/users/username/software/version/Part3/Part3Extra.so (0x00002b3543fb2000)

Answer 1

Finally managed to work out what was going on...

For those that find unexpected errors related to c++ code that runs normally on one part of a multi-node system but not on other parts even though the system shares a common file structure, this last point can be very misleading, although i'm sure rather obvious to those more clued up with system administration.

Initially I was under the impression that the head and worker/computation nodes shared the entire file structure. This is only partly true as the worker nodes had access to certain parts of the file system, but importantly not the core files, such as /lib and /lib64. Essentially all packages installed via yum were independent to each computation node. Having updated the head node to the correct gcc version ( devtoolset-2 in this instance), I was under the impression that each worker node was also updated. This was not the case.

The Underlying Problem

c++ code compiled using gcc 4.8.2 (Red Hat 4.8.2-15) worked on the head-node which had libstdc++.x86_64 (v4.1.2-55.el5) in catching a std::runtime_error error thrown. This error was not properly caught when running on worker nodes.

The problem was that the worker node system libstdc++.x86_64 version was old (unfortunately I can't remember what version exactly) which meant errors were not being caught. It appears that a certain version of libstdc++ is unable to catch errors from code compiled using gcc 4.8.2 .

The Solution

Each worker node had to be manually updated using yum so that its libstdc++ version was high enough to resolve this issue ( v4.1.2-55.el5 in our case). Updating libstdc++ fixed the problem.

Extra Info

In our case the worker nodes were unable to connect to the internet directly so yum had to be done via proxy. Our version of yum was also too old to use the socks5h automatic ssh tunnelling proxy method. Because of this we had to use the squid package on the head node to allow a connection.

Finally, this took a while to figure out as the file structure differences were very misleading. While the key folders are not shared between nodes, there contents looks exactly the same as the old/new versions of the packages had the exact same file structure, just with modified contents.

Once again, this is probably all very obvious to a system admin, but there you go.