g++ exhaustive memory usage compiling auto-generated code

Question

I'm working on a project that requires the calculation of a Jacobian and Hessian matrix of a complex function. To generate the correct mathematical expression, I use symbolic differentiation and Python's SymPy module to generate the corresponding C++ code. It results in a single C++ Header-File with three functions (the function itself, jacobian, and hessian). The hessian, for example, has 11x11 expressions of the following form:

hessian(0,0) = <long-long-mathematical-function-with-abs-and-tanh-etc>;

Furthermore, these functions/expressions are template-functions as I need to pass different types (Scalars, Intervals) to them.

When I compile the program, it requires up to 70 GB RAM with g++ (GCC) 10.2.1 20201016 (Red Hat 10.2.1-6). Initially, I got a warning g++ note: variable tracking size limit exceeded

I added the -fno-var-tracking flag, but it did not reduce the required memory in any significant way.

Could the problem be originated in the size of the right-hand-side of the individual expressions?

Is there any way (maybe compiler flags or sorts of) that might reduce the memory usage during compilation?

I use Eigen3 with custom scalar types to perform mathematical operations.

Answer 1

This question is not really related to eigen3, and in fact there's no way to answer it definitively, given the lack of details. That said, here are some suggestions:

1. Split the header into smaller headers, eg three separate headers for the respective functions, and only include the one(s) you need at any given point. (I suspect this won't help much.)
2. Manually rewrite the header to declare the function templates, and move the definitions into a CPP file, along with explicit template instantiations . This assumes that there are a fixed number of instantiations (eg for double and some interval class) that you need.
3. Rewrite the CPP files that include these headers to be smaller, by splitting them into components.

Ultimately, each invocation of g++ is on a single translation unit (traditionally a .cpp file), so at some point you need to figure out which translation unit is causing g++ to run out of memory, and split it up.

On the system side, you could increase the amount of swap available .