Using more memory than available

Question

I have written a program that expands a database of prime numbers. This program is written in python and runs on windows 10 (x64) with 8GB RAM.

The program stores all primes it has found in a list of integers for further calculations and uses approximately 6-7GB of RAM while running. During some runs however, this figure has dropped to below 100MB . The memory usage then stays low for the duration of the run, though increasing as expected as more numbers are added to the prime array. Note that not all runs result in a memory drop.

Memory usage measured with task manager

These, seemingly random, drops has led me the following theories:

1. There's a bug in my code, making it drop critical data and messing up the results (most likely but not supported by the results)
2. Python just happens to optimize my code extremely well after a while.
3. Python or Windows is compensating for my over-usage of the RAM by cleaning out portions of my prime-number array that aren't used that much. (eventually resulting in incorrect calculations)
4. Python or Windows is compensating for my over-usage of the RAM by allocating disk space instead of ram.

Questions

1. What could be the reason(s) for this memory drop?
2. How does python handle programs that use more than available RAM?
3. How does Windows handle programs that use more than available RAM?

Answer 1

1, 2, and 3 are incorrect theories.

4 is correct. Windows (not Python) is moving some of your process memory to swap space. This is almost totally transparent to your application - you don't need to do anything special to respond to or handle this situation. The only thing you will notice is your application may get slower as information is written to and read from disk. But it all happens transparently. See https://en.wikipedia.org/wiki/Virtual_memory for more information.

Answer 2

Have you heard of paging? Windows dumps some ram (that hasn't been used in a while) to your hard drive to keep your computer from running out or ram and ultimately crashing.

Only Windows deals with memory management. Although, if you use Windows 10, it will also compress your memory, somewhat like a zip file.