Why doesn't the OS crash if I dereference a null pointer?

Question

Dereferencing a null pointer results in undefined behavior. In practice it usually means that my program will crash. But why doesn't the OS crash? Because if my program dereferences a null pointer, and my program is run by the OS, then, according to the rules of logical transitivity, this means the OS tried to dereference a null pointer. Why doesn't the OS enter a state of "undefined behavior"?

Answer 1

The C++ standard doesn't define the behaviour, either to guarantee a crash, or to do anything else. That doesn't prevent the OS from defining the behaviour - it's not a C++ program, so it doesn't have to abide by the "rules" ^[1] of C++ programs. Even so, the OS won't dereference the pointer itself.

On most modern platforms, accessing the target of the dereferenced pointer will cause the memory-management hardware to raise an exception (often called a "segmentation fault" or "protection fault"). This is caught by the kernel, which can determine which process did it, and either kill the process, or send it a signal.

So, on such a platform, the default behaviour of a process that dereferences a null pointer will be to crash; there is no reason whatsoever for the OS itself to crash.

^[1] _{By which I mean the informal "rules" that a program should be well-formed and avoid undefined behaviour - not to be confused with the formal "rules" for C++ implementations specified by the language standard.}

Answer 2

Memory access is protected in every major OS. You cannot simply write a program that manipulates memory that was not allocated for it (assuming a pointer is not initialized for example, it could be ANY address). So, every time a program tries to access some address space that does not belong to it, the OS will send a signal to terminate the program (resulting in the ultimate famous "Segmentation fault", familiar to any C/C++ programmer).

Answer 3

Because the OS has to do something , and crashing would make for a rather bad user experience.

The OS isn't being written to run on the abstract machine of the C standard. It's being written for real hardware that behaves in real ways to different situations that the standard calls "undefined," so it can (and really must) take those real behaviors into account. If it didn't, the operating system would perform differently on different hardware, which kind of defeats the purpose of having an OS, doesn't it?

And before you say "undefined behavior is undefined, let the user of bad code wreck what they want," imagine the security problems of a single accidental buffer overrun being able to segfault an entire server.

Answer 4

First of all, UB means "anything can happen". In practice however modern OSes offer memory protection - when a program tries to dereference a null pointer that attempt triggers an interrupt inside the CPU which is caught and handled by the OS and the OS then stops the program and then continues running as if nothing bad happened.

Answer 5

There are no rules of logical transitivity when it comes to UB. Your assumption is wrong.

UB does mean that anything can happen, so on a poorly written OS, your program might actually crash the OS. Don't rule it out.

Also, your program doesn't crash because you dereference a NULL pointer. It crashes because the OS tells it to crash.

Answer 6

The OS sets up a fault handler which is called if a memory access violates rules imposed by the OS - such as an access to the null address. If your program is about to dereference a null pointer this fault handler is called and the program will be ended before it accesses the disallowed memory region. So your program does actually never dereference a null pointer, it is catched while trying.

The mechanism for detecting forbidden memory accesses is often done with hardware support like page tables or memory segmentation.

If the OS kernel itself dereferences a null pointer, it usually halted while trying to do so. You will get a blue screen, kernel oops or similar. If it keeps going, that may actually result in "undefined behaviour".

Note that the term "undefined behaviour" is only exactly defined in C or similar languages, the processor doesn't really care - usually what happens if you try to access a memory region for which you don't have sufficient rights is very well defined in the context of the architecture.

Answer 7

Because most of the programs run in user mode , and the OS runs in kernel mode . The Kernel mode is near to the physical hardware (they say close to the metal ). Kernel mode programs (OS, some services, drivers etc) runs in ring 0 of CPU. User mode programs runs on higher ring. User mode programs running on ring N of CPU, cannot access programs or memory running on anything less than N. If they try to, they wont be allowed!

All programs get their logical address, and OS assigns it. OS does the logical to physical addressing when program tries to read or to write some memory. If program tries to access the address, which it doesn't have permission, the OS will throw the exception. This exception may be handled by program itself (a local exception-handler, in same thread). If not, any attached global exception handler. Debugger may also come into picture, if local EH doesn't handle it. It depends on OS, how/when to route exception to debugger and/or to the global exception handler. It also depends on type of exception (like null-pointer access), if OS allows local/global/debugger to handle it or not. If no one handles it, the OS would terminate the process (and possibly creating crash dump, segmentation fault core dump).

If the process it not being debugged (Windows specific), and some debugger is installed, OS may allow user to debug it.

If the kernel mode program does something nasty, it would take down the OS. I'm not Linux guy, so don't know behavior of Linux. But, in case of Windows, BSOD would brighten your monitor with blue color!

Answer 8

Because if my program dereferences a null pointer, and my program is run by the OS, then, according to the rules of logical transitivity, this means the OS tried to dereference a null pointer. Why doesn't the OS enter a state of "undefined behavior"?

This is wrong. There's something called memory protection and thats WHY your program is terminated. Is the OS that is protecting itself (in the terms of memory use).

Answer 9

Sorry, what rules of 'logical transitivity'? One of the things an operating system is designed to do is to protect programs from the misbehaviour of other programs. In particular, the O/S should not crash just because your program tries to do something silly.

On operating systems without memory protection, accessing via a null (or any invalid) pointer could indeed cause the O/S to crash (if the O/S happened to use location 0 for something interesting).

But that has nothing to do with logical transitivity. That has to do with your program accessing memory that belongs to another program. Either programs could crash in those circumstances.