How to monitor processes on linux

Question

When an executable is running on Linux, it generates processes, threads, I/O... etc, and uses libraries from languages like C/C++, sometimes there might be timers in question, is it possible to monitor this? how can I get a deep dive into these software and processes and what is going on in the background?

I know this stuff is abstracted from me because I shouldn't be worrying about it as a regular user, but I'm curious to what would I see.

What I need to see are:

1. System calls for this process/thread.
2. Open/closed sockets.
3. Memory management and utilization, what block is being accessed.
4. Memory instructions.
5. If a process is depending on the results of another one.
6. If a process/thread terminates, why, and was it successful?
7. I/O operations and DB read/write if any.

Answer 1

How about top ?

On that page are more than one linux/Unix command, which might come in handy.

Answer 2

The different things you wanted to monitor may require different tools. All tools I will mention below have extensive manual pages where you can find exactly how to use them.

System calls for this process/thread.

The strace command does exactly this - it lists exactly which system calls are invoked by your program. The ltrace tool is similar, but focuses on calls to library functions - not just system calls (which involve the kernel).

Open/closed sockets.

The strace/ltrace commands will list among other things socket creation, but if you want to know which sockets are open - connected, listening, and so on - right now, there is the netstat utility, which lists all the connected (or with "-a", also listening) sockets in the system, and which process they belong to.

Memory management and utilization, what block is being accessed. Memory instructions.

Again ltrace will let you see all malloc()/free() calls, but to see exactly what memory is being access where, you'll need a debugger, like gdb . The thing is that almost everything your program does will be a "memory instruction" so you'll need to know exactly what you are looking for, with breakpoints, tracepoints, single-stepping, and so on, and usually don't just want to see every memory access in your program.

If you don't want to find all memory accesses but rather are searching for bugs in this area - like accessing memory after it's freed and so on, there are tools that help you find those more easily. One of them called ASAN ("Address Sanitizer") is built into the C++ compiler, so you can build with it enabled and get messages on bad access patterns. Another one you can use is valgrind .

Finally, if by "memory utilization" you meant to just check how much memory your process or thread is using, well, both ps and top can tell you that.

If a process is depending on the results of another one. If a process/thread terminates, why, and was it successful?

Various tools I mentioned like strace/ltrace will let you know when the process they follow exits. Any process can print the exit code of one of its sub-processes, but I'm not aware of a tool which can print the exit status of all processes in the system.

I/O operations

There is iostat that can give you periodic summaries of how much IO was done to each disk. netstat -s gives you network statistics so you can see how many network operations were done. vmstat gives you, among other things, statistics on IO caused by swap in/out (in case this is a problem in your case).

and DB read/write if any.

This depends on your DB, I guess, and how you monitor it.