Salt request timed out - Minion did not return

Question

I have on AWS 6 t2.micro and 1 t2.small instances running with Ubuntu. All these instances are running in the same VPC.

After accepting the keys

Accepted Keys:
minion-bastion
minion-esc-kibana
minion-esc-logs-s3
minion-esn-1
minion-esn-2
minion-esn-3

, I run the salt '*' test.ping command. Sometimes I get no response from any of the minions:

minion-bastion:
    Minion did not return. [No response]
minion-esn-1:
    Minion did not return. [No response]
minion-esn-2:
    Minion did not return. [No response]
minion-esn-3:
    Minion did not return. [No response]
minion-esc-logs-s3:
    Minion did not return. [No response]
minion-esc-kibana:
    Minion did not return. [No response]

Sometimes some minions respond and others not.

minion-esc-kibana:
    True
minion-esc-logs-s3:
    True
minion-esn-3:
    True
minion-esn-2:
    True
minion-bastion:
    Minion did not return. [Not connected]
minion-esn-1:
    Minion did not return. [No response]

And again sometimes I get this message:

Salt request timed out. The master is not responding. If this error persists after verifying the master is up, worker_threads may need to be increased.

Everytime I run the test command, I get another responses. One minion which answered before, is not reponding now. And so on.

ubuntu@ip-10-2-1-100 ~> sudo salt-run manage.status
down:
    - minion-esn-1
up:
    - minion-bastion
    - minion-esc-kibana
    - minion-esc-logs-s3
    - minion-esn-2
    - minion-esn-3
ubuntu@ip-10-2-1-100 ~> sudo service salt-master status
salt-master start/running, process 11390
ubuntu@ip-10-2-1-100 ~> sudo salt-run manage.status
down:
    - minion-esc-logs-s3
up:
    - minion-bastion
    - minion-esc-kibana
    - minion-esn-1
    - minion-esn-2
    - minion-esn-3
ubuntu@ip-10-2-1-100 ~>

In log files there are no relevant entries.

I use the original config file. I just changed the minion ID and the master ID or IP.

For the instance with the salt master running on it, I use a t2.small instance with 2GB RAM. Before I was using a t2.micro with 1GB RAM. There were even more errors.

I don't know what I can do. OR what I'm doing wrong. Has anybody any suggestions?

EDIT:

1 of 6 minions is alwas down. Which one, changes with every manage.status command.

Answer 1

When you run test.ping the Master tells Minions to run commands/functions, and listens for the return data, printing it to the screen when it is received. If it doesn't receive anything back, it doesn't have anything to display for that Minion.

There are a couple options for getting information on Minions that are not responding. One is to use the verbose (-v) option when you run salt commands, as it will display "Minion did not return" for any Minions which time out.

salt -v '*' pkg.install zsh

Another option is to use the manage.down runner:

salt-run manage.down

Also, if the Master is under heavy load, it is possible that the CLI will exit without displaying return data for all targeted Minions. However, this doesn't mean that the Minions did not return; this only means that the Salt CLI timed out waiting for a response. Minions will still send their return data back to the Master once the job completes. If any expected Minions are missing from the CLI output, the jobs.list_jobs runner can be used to show the job IDs of the jobs that have been run, and the jobs.lookup_jid runner can be used to get the return data for that job.

salt-run jobs.list_jobs
salt-run jobs.lookup_jid 20130916125524463507

If you find that you are often missing Minion return data on the CLI, only to find it with the jobs runners, then this may be a sign that the worker_threads value may need to be increased in the master config file. Additionally, running your Salt CLI commands with the -t option will make Salt wait longer for the return data before the CLI command exits. For instance, the below command will wait up to 60 seconds for the Minions to return:

salt -t 60 '*' test.ping