服务器重启后 Nova 服务调度程序和指挥器无法启动

Question

I have installed allinone setup using packstack in a server. I have generated one windows server instance and it was working fine. The system was left idle and later it was rebooted. After reboot, I'm not able to see the keystone service. I'm not able to start keystone service as well. It says the unit is not found but earlier keystone service was working fine and there are no logs of keystone after reboot. I'm not able ot start nova-conductor and nova-scheduler services. I'm attaching the log files for reference. Can anyone please share a way ahead to eliminate the error.

Thanks in advance.

Nova-scheduler Log:

2021-12-28 14:02:36.568 44587 ERROR nova keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://172.26.6.238:5000/v3/auth/tokens : HTTPConnectionPool(host='172.26.6.238', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f5f470529e8>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2021-12-28 14:02:36.568 44587 ERROR nova 2021-12-28 14:02:38.790 44597 INFO oslo_service.periodic_task [-] Skipping periodic task _discover_hosts_in_cells because its interval is negative 2021-12-28 14:02:38.799 44597 WARNING keystoneauth.identity.generic.base [-] Failed to discover available identity versions when contacting http://172.26.6.238:5000/v3 . Attempting to parse version from URL.: keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://172.26.6.238:5000/v3 : HTTPConnectionPool(host='172.26.6.238', port=5000): Max retries exceeded with url: /v3 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f53f3575a20>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2021-12-28 14:02:38.803 44597 CRITICAL nova [-] Unhandled error: keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://172.26.6.238:5000/v3/auth/tokens : HTTPConnectionPool(host='172.26.6.238', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f53f358e9e8>: Fail ed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2021-12-28 14:02:38.803 44597 ERROR nova Traceback (most recent call last): 2021-12-28 14:02:38.803 44597 ERROR nova File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 159, in _new_conn 2021-12-28 14:02:38.803 44597 ERROR nova (self._dns_host, self.port), self.timeout, **extra_kw) 2021-12-28 14:02:38.803 44597 ERROR nova File "/usr/lib/python3.6/site-packages/urllib3/util/connection.py", line 80, in create_connection 2021-12-28 14:02:38.803 44597 ERROR nova raise err 2021-12-28 14:02:38.803 44597 ERROR nova File "/usr/lib/python3.6/site-packages/urllib3/util/connection.py", line 70, in create_connection 2021-12-28 14:02:38.803 44597 ERROR nova sock.connect(sa) 2021-12-28 14:02:38.803 44597 ERROR nova File "/usr/lib/python3.6/site-packages/eventlet/greenio/base.py", line 253, in connect 2021-12-28 14:02:38.803 44597 ERROR nova socket_checkerr(fd) 2021-12-28 14:02:38.803 44597 ERROR nova File "/usr/lib/python 3.6/site-packages/eventlet/greenio/base.py", line 51, in socket_checkerr 2021-12-28 14:02:38.803 44597 ERROR nova raise socket.error(err, errno.errorcode[err]) 2021-12-28 14:02:38.803 44597 ERROR nova ConnectionRefusedError: [Errno 111] ECONNREFUSED 2021-12-28 14:02:38.803 44597 ERROR nova 2021-12-28 14:02:38.803 44597 ERROR nova During handling of the above exception, another exception occurred: 2021-12-28 14:02:38.803 44597 ERROR nova 2021-12-28 14:02:38.803 44597 ERROR nova Traceback (most recent call last): 2021-12-28 14:02:38.803 44597 ERROR nova File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen 2021-12-28 14:02:38.803 44597 ERROR nova chunked=chunked) 2021-12-28 14:02:38.803 44597 ERROR nova File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 354, in _make_request 2021-12-28 14:02:38.803 44597 ERROR nova conn.request(method, url, **httplib_request_kw) 2021-12-28 14:02:38.803 44597 ERROR nova File "/usr/lib64/python3.6/http/client.py", line 1254, in request 2021-12-28 14:02:38.803 44597 ERROR nova self._send_request(method, url, body, headers, encode_chunked) 2021-12-28 14:02:38.803 44597 ERROR nova File "/usr/lib64/python3.6/http/client.py", line 1300, in _send_request 2021-12-28 14:02:38.803 44597 ERROR nova self.endheaders(body, encode_chunked=encode_chunked) 2021-12-28 14:02:38.803 44597 ERROR nova File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders 2021-12-28 14:02:38.803 44597 ERROR nova self._send_output(message_body, encode_chunked=encode_chunked) 2021-12-28 14:02:38.803 44597 ERROR nova File "/usr/lib64/python3.6/http/client.py", line 1036, in _send_output 2021-12-28 14:02:38.803 44597 ERROR nova self.send(msg) 2021-12-28 14:02:38.803 44597 ERROR nova File "/usr/lib64/python3.6/http/client.py", line 974, in send 2021-12-28 14:02:38.803 44597 ERROR nova self.connect() 2021-12-28 14:02:38.803 44597 ERROR nova File "/usr /lib/python3.6/site-packages/urllib3/connection.py", line 181, in connect 2021-12-28 14:02:38.803 44597 ERROR nova conn = self._new_conn() 2021-12-28 14:02:38.803 44597 ERROR nova File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 168, in _new_conn 2021-12-28 14:02:38.803 44597 ERROR nova self, "Failed to establish a new connection: %s" % e) 2021-12-28 14:02:38.803 44597 ERROR nova urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f53f358e9e8>: Failed to establish a new connection: [Errno 111] ECONNREFUSED 2021-12-28 14:02:38.803 44597 ERROR nova 2021-12-28 14:02:38.803 44597 ERROR nova During handling of the above exception, another exception occurred: 2021-12-28 14:02:38.803 44597 ERROR nova 2021-12-28 14:02:38.803 44597 ERROR nova Traceback (most recent call last): 2021-12-28 14:02:38.803 44597 ERROR nova File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send 2021-12-28 14:02:3 8.803 44597 ERROR nova timeout=timeout 2021-12-28 14:02:38.803 44597 ERROR nova File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen 2021-12-28 14:02:38.803 44597 ERROR nova _stacktrace=sys.exc_info()[2]) 2021-12-28 14:02:38.803 44597 ERROR nova File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 399, in increment 2021-12-28 14:02:38.803 44597 ERROR nova raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2021-12-28 14:02:38.803 44597 ERROR nova urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='172.26.6.238', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f53f358e9e8>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2021-12-28 14:02:38.803 44597 ERROR nova 2021-12-28 14:02:38.803 44597 ERROR nova During handling of the above exception, another exception occurred: 2021-12-28 14:02:38.803 44597 ERROR nova 2021-12-28 14:02:38.803 44597 ERROR nova Traceback (most recent call last): 2021-12-28 14:02:38.803 44597 ERROR nova File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1004, in _send_request 2021-12-28 14:02:38.803 44597 ERROR nova resp = self.session.request(method, url, **kwargs)

Nova-Conductor Log

2021-12-28 15:08:32.128 82721 ERROR nova ConnectionRefusedError: [Errno 111] ECONNREFUSED 2021-12-28 15:08:32.128 82721 ERROR nova 2021-12-28 15:08:32.128 82721 ERROR nova During handling of the above exception, another exception occurred: 2021-12-28 15:08:32.128 82721 ERROR nova 2021-12-28 15:08:32.128 82721 ERROR nova Traceback (most recent call last): 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen 2021-12-28 15:08:32.128 82721 ERROR nova chunked=chunked) 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 354, in _make_request 2021-12-28 15:08:32.128 82721 ERROR nova conn.request(method, url, **httplib_request_kw) 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib64/python3.6/http/client.py", line 1254, in request 2021-12-28 15:08:32.128 82721 ERROR nova self._send_request(method, Z572D4E421E5E6B9BC11D815E8 A027112Z, body, headers, encode_chunked) 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib64/python3.6/http/client.py", line 1300, in _send_request 2021-12-28 15:08:32.128 82721 ERROR nova self.endheaders(body, encode_chunked=encode_chunked) 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib64/python3.6/http/client.py", line 1249, in endheaders 2021-12-28 15:08:32.128 82721 ERROR nova self._send_output(message_body, encode_chunked=encode_chunked) 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib64/python3.6/http/client.py", line 1036, in _send_output 2021-12-28 15:08:32.128 82721 ERROR nova self.send(msg) 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib64/python3.6/http/client.py", line 974, in send 2021-12-28 15:08:32.128 82721 ERROR nova self.connect() 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 181, in connect 2021-12-28 15:08:32.128 82721 ERROR nova conn = self._new_conn() 2021-12-28 15:08:32.1 28 82721 ERROR nova File "/usr/lib/python3.6/site-packages/urllib3/connection.py", line 168, in _new_conn 2021-12-28 15:08:32.128 82721 ERROR nova self, "Failed to establish a new connection: %s" % e) 2021-12-28 15:08:32.128 82721 ERROR nova urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fc0e5be6978>: Failed to establish a new connection: [Errno 111] ECONNREFUSED 2021-12-28 15:08:32.128 82721 ERROR nova 2021-12-28 15:08:32.128 82721 ERROR nova During handling of the above exception, another exception occurred: 2021-12-28 15:08:32.128 82721 ERROR nova 2021-12-28 15:08:32.128 82721 ERROR nova Traceback (most recent call last): 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send 2021-12-28 15:08:32.128 82721 ERROR nova timeout=timeout 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen 2021-12-28 15:08:32.128 82721 ERROR nova _stacktrace=sys.exc_info()[2]) 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 399, in increment 2021-12-28 15:08:32.128 82721 ERROR nova raise MaxRetryError(_pool, url, error or ResponseError(cause)) 2021-12-28 15:08:32.128 82721 ERROR nova urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='172.26.6.238', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc0e5be6978>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2021-12-28 15:08:32.128 82721 ERROR nova 2021-12-28 15:08:32.128 82721 ERROR nova During handling of the above exception, another exception occurred: 2021-12-28 15:08:32.128 82721 ERROR nova 2021-12-28 15:08:32.128 82721 ERROR nova Traceback (most recent call last) : 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1004, in _send_request 2021-12-28 15:08:32.128 82721 ERROR nova resp = self.session.request(method, url, **kwargs) 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 533, in request 2021-12-28 15:08:32.128 82721 ERROR nova resp = self.send(prep, **send_kwargs) 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 646, in send 2021-12-28 15:08:32.128 82721 ERROR nova r = adapter.send(request, **kwargs) 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 516, in send 2021-12-28 15:08:32.128 82721 ERROR nova raise ConnectionError(e, request=request) 2021-12-28 15:08:32.128 82721 ERROR nova requests.exceptions.ConnectionErro r: HTTPConnectionPool(host='172.26.6.238', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc0e5be6978>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2021-12-28 15:08:32.128 82721 ERROR nova 2021-12-28 15:08:32.128 82721 ERROR nova During handling of the above exception, another exception occurred: 2021-12-28 15:08:32.128 82721 ERROR nova 2021-12-28 15:08:32.128 82721 ERROR nova Traceback (most recent call last): 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/bin/nova-conductor", line 10, in 2021-12-28 15:08:32.128 82721 ERROR nova sys.exit(main()) 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib/python3.6/site-packages/nova/cmd/conductor.py", line 44, in main 2021-12-28 15:08:32.128 82721 ERROR nova topic=rpcapi.RPC_TOPIC) 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib/python3.6/site-packages/nov a/service.py", line 270, in create 2021-12-28 15:08:32.128 82721 ERROR nova periodic_interval_max=periodic_interval_max) 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib/python3.6/site-packages/nova/service.py", line 128, in init 2021-12-28 15:08:32.128 82721 ERROR nova self.manager = manager_class(host=self.host, *args, **kwargs) 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 121, in init 2021-12-28 15:08:32.128 82721 ERROR nova self.compute_task_mgr = ComputeTaskManager() 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 247, in init 2021-12-28 15:08:32.128 82721 ERROR nova self.report_client = report.SchedulerReportClient() 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 186, in init 2021-12-28 15:08:32.128 82721 ERROR nova self._client = self._create_client() 2021-12-28 15:08:32 .128 82721 ERROR nova File "/usr/lib/python3.6/site-packages/nova/scheduler/client/report.py", line 229, in _create_client 2021-12-28 15:08:32.128 82721 ERROR nova client = self._adapter or utils.get_sdk_adapter('placement') 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib/python3.6/site-packages/nova/utils.py", line 1079, in get_sdk_adapter 2021-12-28 15:08:32.128 82721 ERROR nova return getattr(conn, service_type) 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib/python3.6/site-packages/openstack/service_description.py", line 93, in get 2021-12-28 15:08:32.128 82721 ERROR nova endpoint = proxy_mod.Proxy.get_endpoint(proxy) 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 282, in get_endpoint 2021-12-28 15:08:32.128 82721 ERROR nova return self.session.get_endpoint(auth or self.auth, **kwargs) 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib/python3.6/site-packages/keystoneaut h1/session.py", line 1225, in get_endpoint 2021-12-28 15:08:32.128 82721 ERROR nova return auth.get_endpoint(self, **kwargs) 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint 2021-12-28 15:08:32.128 82721 ERROR nova allow_version_hack=allow_version_hack, **kwargs) 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data 2021-12-28 15:08:32.128 82721 ERROR nova service_catalog = self.get_access(session).service_catalog 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 134, in get_access 2021-12-28 15:08:32.128 82721 ERROR nova self.auth_ref = self.get_auth_ref(session) 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref 2021-12-28 15:08:32.128 82721 ERRO R nova return self._plugin.get_auth_ref(session, **kwargs) 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/v3/base.py", line 184, in get_auth_ref 2021-12-28 15:08:32.128 82721 ERROR nova authenticated=False, log=False, **rkwargs) 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1131, in post 2021-12-28 15:08:32.128 82721 ERROR nova return self.request(url, 'POST', **kwargs) 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 913, in request 2021-12-28 15:08:32.128 82721 ERROR nova resp = send(**kwargs) 2021-12-28 15:08:32.128 82721 ERROR nova File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1020, in _send_request 2021-12-28 15:08:32.128 82721 ERROR nova raise exceptions.ConnectFailure(msg) 2021-12-28 15:08:32.128 82721 ERROR nova keystoneauth1.exceptions.connection.ConnectFailure: Unable t o establish connection to http://172.26.6.238:5000/v3/auth/tokens : HTTPConnectionPool(host='172.26.6.238', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc0e5be6978>: Failed to establish a new connection: [Errno 111] ECONNREFUSED',)) 2021-12-28 15:08:32.128 82721 ERROR nova

Answer 1

The error messages in the logfile say that it cannot connect to http://172.26.6.238:5000/v3/auth/tokens . That will be your Keystone service.

You need to investigate why your Keystone service is not listening for connections.

I'm not able to start Keystone service as well.

Solve that problem first.

It says the unit is not found...

Suspicious! Systemd unit files don't just spontaneously disappear. You need to figure out where they have gone.