How can I create a long running task that listens to requests from the terminal? (Not just asking for input)

Question

I am trying something new to me, so I don't have the vocabulary to express my questions in any sort of domain specific language.

I am currently mind mapping a tool I would like to build. The function of this tool is to execute many long running tasks and log results to an remote database. This is most similar to Jenkins build and test functionality. Unfortunately, I don't think I can use Jenkins since these are tests executed on real live custom hardware with a lot of IO operations with other hardware resources.

It would almost certainly be run on a server, so it's headless. I generally build desktop tools with a UI to execute similar tasks in a windows desktop environment. When I want to communicate to the user what the tool is doing I simply create some UI elements to display status.

In this environment there would rarely if ever be a person looking at it work. If I DO need to debug something while running, or just want to check the status my immediate thought is log files. However, they are pretty cumbersome to watch in real time.

I would like to be able to make requests of the task runner from the command line in the same way I git status . My current thought is to register my command (like git) on PATH. I could have that command connect to a named pipe that my long process is connected to/monitoring and relay the user's request. (I have never used named pipes before, but it seems like a standard way to have processes communicate?)

This solution requires three "layers":

1. The command that is on PATH that can parse/accept/reject the user's request and then forward it along.
2. The long process manager that listens for user requests and monitors long task execution.
3. The task executers themselves.

Are there other approaches? Am I reinventing the wheel? Links and resources are greatly appreciated!

Answer 1

How can I create a long running task that listens to requests from the terminal? (Not just asking for input)

You can use sockets for inter-process communication or pipes .

Am I reinventing the wheel?

It depends on the use of that program. ie what are the tasks that are running. For ex: If you are creating a version control system then yes, git already exists

If not then no. You are simply creating a program or application or website that use multiple threads. Multiple threads are provided by the hardware (CPU) and Operating system. You Interact with these threads using the programming languages. Applications or programs can be single or multi threaded.
For example github: When you vote on an answer you are basically creating a request that is received by a program running on a server which creates a thread to update the vote count in the database. In the meanwhile if someone else also vote on an answer a similar request is sent and handled maybe by the same server on another thread. This server is still listening for request and in the same time executing tasks (updating the vote count in the database)

Note : This example is simplified, just as an example

For running tasks in the background and still having a functioning UI or request listener, you require multiple threads. One thread for UI, one or more for handling requests, and the rest for running those background tasks and writing to the logs.

Note : when dealing with threads thread safety and shared resources between threads are important.
Also programs can also use more than multiple processes, but multithreading was introduced to fix some of the overhead of using multiple processes (related to thread safety and shared resources).

In your case, If the log files are in a database, then concurrency is access is handled by that database.
But in case the log file is a local file on the system, then writing and reading from the log requires handling the multiple threads access to that one log file.
That is where thread safety and shared resources is used.

Note that the below is a simplified comparision:

In case of a local application which uses multiple processes:

This application is like a.network of processes, with each process running a certain program.

Process 1 might render the UI, Process 2 might write to a log file, etc...

Pipes, signals, or other operating system technologies are used to send messages and communicate between these processes (Programs).
Ex: when a user press "Write log" button, process 1 communicates with process 2 (using the communication methods from above) to tell process2 to write to that log file.

In the case of a web app:

processes becomes computers connected over the.network(or servers running on a host)
Operating system technologies becomes Network communication like HTTP request and responses, JSON, xml, etc...
So
This application is like a.network of computers (servers) located in different locations, with each computer running a certain program.

Computer 1 might render the UI, computer 2 might write to the database, etc...

Network communication technologies are used to communicate between the computers or servers over the inte.net.
Ex: when a user press "Write log" button, Computer 1 communicates with Computer 2 (using the communication methods from above) to tell Computer to write to that database.

A host is a computer hosting (running) a program. This program is usually called a server. (The host provides services)