Segmentation Fault with Large Input

Question

I know segmentation fault means that the process has attempted to access certain memory that it's not allowed to.

I'm running some program written by others using C++. And when my input is large (around 1GB), there'll be a segmentation fault even though I requested 30GB memory; while when input size is quite small, it goes well.

So what shall I do? Is it because there's not enough memory? I'm really kind of a newbie without much knowledge of C++. I don't even know which part of the code controls memory allocation.

Thanks to BLender, line from the debugging is:

Program received signal SIGSEGV, Segmentation fault. 0x0000003fbd653174 in _IO_vfscanf_internal () from /share/bin/intel/cc/10.1.015/lib/tls/x86_64/libc.so.6

Answer 1

even I request 30GB memory

Do you have 30GB of memory? I really doubt it.

The answer depends on the function of the program. If the program reads and processes data without memory (ie, the data that was read before doesn't influence the processing of the data being read), you can load the file in chunks.

But without details, I can't say much more.

---

Debug your program. When you compile it, enable debugging:

g++ -g -o program -Wall program.cpp 

And use gdb to debug it:

gdb program
(gdb) run

And the line number and function that caused the segfault should show up.

Answer 2

Your code calls malloc several times, but never free , so it uses quite a lot of memory. And it never checks for an out-of-memory condition...

My suggestion is that you change all the calls to malloc to something like:

size_t total_memory = 0;
void *my_malloc(size_t sz)
{
    void *res = malloc(sz);
    total_memory += sz;
    if (res == NULL)
    {
        printf("Too much memory eaten: %zu\n", total_memory);
        abort();
    }
    return res;
}
#define malloc(x) my_malloc(x)

And see what happens.

Answer 3

Most likely they have a fixed-sized buffer in there, and your "large input"s are too large to fit inside it. C++ does not check these things by default (if they had used nice checked data structures from the STL perhaps, but clearly they didn't do that).

Answer 4

Try running the program in a debugger, but first make sure it is compiled with debugging information (-g). Run it with the data it segvaults on. It may be that your program tries to allocate large amount of memory with a call to malloc or new and assign it to a pointer, doesn't check if that was successful and then tries to access the memory (via the pointer) which was supposed to be allocated. You would see where it happens by examining stack trace after the segmentation violation in the debugger. That should give you a clue which part of the program should be modified so eg it doesn't read in the whole input file but only a chunk of it, in a loop.