How to create a kubernetes job from a pod

Question

I'm working on a cluster in which I'm performing a lot scraping on Instagram to find valuable accounts and then message them to ask if they're interested in selling their account. This is what my application consists of:

1. Finding Instagram accounts by scraping for them with a lot of different accounts
2. Refine the accounts retrieved and sort out the bad ones
3. Message the chosen accounts

In addition to this, I'm thinking of uploading every data of each step to a database (the whole chunk of accounts gathered in step 1, the refined accounts gathered in step 2, and the messaged users from step 3) in separate collections. I'm also thinking of developing a slack bot that handles errors by messaging me a report of the error and eventually so it can message me whenever a user responds. As you can see, there are a lot of different parts of this application and that is the reason why I figured that using Kubernetes for this would be a good idea.

My initial approach to this was by making every pod in my node a rest API. Then I could send a request to each pod, each time I wanted them to run. But if figured that this would not be an optimal solution and not in any way a Kubernetes-way approach.

Answer 1

The only way to achieve it in way you describe it is to communicate with Kubernetes API server from inside of your pod. This requires several thing (adding service account and role binding, using kubernetes client etc) and I would not recommended it as regular application flow (unless you are a devops trying to provide some generic/utility solution).

From another angle - sharing a volumes between pods and jobs should be avoided if possible (it adds complexity and restrictions)

You can dit more on this here - https://kubernetes.io/docs/tasks/administer-cluster/access-cluster-api/#accessing-the-api-from-within-a-pod - as a starter.

If I can suggest some solutions:

• you can share S3 volume and have Cronjob scheduled to run every some time. If cronjob finds data - it process it. Therefore you do not need to trigger job from inside a pod.

• Two services, sending data via http (if feasible) - second service don't do anything when it is not requested from it.

If you share your usecase with some details probably better answers could be provided.

Cheers

Answer 2

There is out of the box support in kubectl to run a job from a cronjob ( kubectl create job test-job --from=cronjob/a-cronjob ), but there is no official support for running a job straight from a pod. You will need to get the pod resource from the cluster and then create a job by using the pod specification as part of the job specification.