How to KILL coordinator Leader node in Apache Druid from a python program (AWS lambda function)?

Question

How to KILL coordinator Leader node in Apache Druid from a python program (AWS lambda function)?

I am working on a druid system. Some of the segments get stuck in rebalancing because of problems in metadata. To prevent this from occurring, I am thinking of killing the coordinator leader node. The Zookeeper will restart the node.

I have following queries.

1. How do I kill druid coordinator leader node from a Python AWS lambda function?

2. How to identify if segment(s) are stuck during rebalancing from a python AWS lambda function?

I know how to identify coordinator leader node from Python program. There is a druid API end point for that [ GET on "/druid/coordinator/v1/leader" ]. This is listed in api-reference.html document.

But I am not able to get the druid POST endpoint for killing a Druid leader node. I went through the API reference documents and posts in inte.net. But not able to get a answers to my queries.

Answer 1

How do I kill druid coordinator leader node from a Python AWS lambda function?

Rarely do databases expose endpoints to kill the services. Druid is also one such example.

I think the question is more about whats causing your druid cluster to be unstable.

How to identify if segment(s) are stuck during rebalancing from a python AWS lambda function?

This is one of the symptoms.

I would first check if I have enough space in the historicals. You can open the services tab from the druid console and see how free are your historicals.

I would also check if your historicals are configured correctly to pull from deep storage. Seems like you are on aws please configure it using s3.

https://druid.apache.org/docs/latest/tutorials/cluster.html#deep-storage

If you have more doubts please feel free to open an issue on github: https://github.com/apache/druid/issues or find us on the druid slack channel https://druid.apache.org/community/