Openstack: Message: No valid host was found. There are not enough hosts available., Code: 500"

Question

I am installing openstack in all-in-one mode and when try to create instances the operation halts with mentioned error. My nova-compute.log file shows following exception.

2022-03-31 11:45:50.435 7 ERROR oslo_service.service libvirt.libvirtError: unable to connect to server at '192.168.211.133:16509': Connection refused
2022-03-31 11:45:50.435 7 ERROR oslo_service.service 
2022-03-31 11:45:50.435 7 ERROR oslo_service.service During handling of the above exception, another exception occurred:
2022-03-31 11:45:50.435 7 ERROR oslo_service.service 
2022-03-31 11:45:50.435 7 ERROR oslo_service.service Traceback (most recent call last):
2022-03-31 11:45:50.435 7 ERROR oslo_service.service   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_service/service.py", line 806, in run_service
2022-03-31 11:45:50.435 7 ERROR oslo_service.service     service.start()
2022-03-31 11:45:50.435 7 ERROR oslo_service.service   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/service.py", line 159, in start
2022-03-31 11:45:50.435 7 ERROR oslo_service.service     self.manager.init_host()
2022-03-31 11:45:50.435 7 ERROR oslo_service.service   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/compute/manager.py", line 1416, in init_host
2022-03-31 11:45:50.435 7 ERROR oslo_service.service     self.driver.init_host(host=self.host)
2022-03-31 11:45:50.435 7 ERROR oslo_service.service   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 709, in init_host
2022-03-31 11:45:50.435 7 ERROR oslo_service.service     self._update_host_specific_capabilities()
2022-03-31 11:45:50.435 7 ERROR oslo_service.service   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 813, in _update_host_specific_capabilities
2022-03-31 11:45:50.435 7 ERROR oslo_service.service     'supports_secure_boot': self._host.supports_secure_boot,
2022-03-31 11:45:50.435 7 ERROR oslo_service.service   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/host.py", line 1550, in supports_secure_boot
2022-03-31 11:45:50.435 7 ERROR oslo_service.service     arch = self.get_capabilities().host.cpu.arch
2022-03-31 11:45:50.435 7 ERROR oslo_service.service   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/host.py", line 781, in get_capabilities
2022-03-31 11:45:50.435 7 ERROR oslo_service.service     xmlstr = self.get_connection().getCapabilities()
2022-03-31 11:45:50.435 7 ERROR oslo_service.service   File "/var/lib/kolla/venv/lib/python3.8/site-packages/nova/virt/libvirt/host.py", line 596, in get_connection
2022-03-31 11:45:50.435 7 ERROR oslo_service.service     raise exception.HypervisorUnavailable()
2022-03-31 11:45:50.435 7 ERROR oslo_service.service nova.exception.HypervisorUnavailable: Connection to the hypervisor is broken on host
2022-03-31 11:45:50.435 7 ERROR oslo_service.service 
2022-03-31 11:45:50.497 7 INFO nova.virt.libvirt.driver [-] Connection event '0' reason 'Failed to connect to libvirt: unable to connect to server at '192.168.211.133:16509': Connection refused'
 2022-03-31 11:46:05.487 7 INFO os_vif [-] Loaded VIF plugins: linux_bridge, noop, ovs
 2022-03-31 11:46:11.219 7 INFO nova.virt.driver [req-85c2604c-936a-4e8c-a168-de17800dbbbb - - - - -] Loading compute driver 'libvirt.LibvirtDriver'

I am using ubuntu 20.04 VM on vmware with 4 VCPUs and 16 GB RAM. KVM is also installed with version EMU emulator version 4.2.1 (Debian 1:4.2-3ubuntu6.21)

What is this URL nova shows connection to in log file and why HypervisorUnavailable exception being raised?

Any help will be appreciated as its been days I am facing this.

Edited: Following commands may give more information:

openstack hypervisor list
+----+----------------------+-----------------+-----------------+-------+
| ID | Hypervisor Hostname  | Hypervisor Type | Host IP         | State |
+----+----------------------+-----------------+-----------------+-------+
|  1 | kypo-virtual-machine | QEMU            | 192.168.211.133 | down  |
+----+----------------------+-----------------+-----------------+-------+

nova service-list
+--------------------------------------+----------------+---------------------    -+----------+----------+-------+----------------------------+-----------------     ------------------------------------------------------------------------------    -----------------+-------------+
| Id                                   | Binary         | Host                     | Zone     | Status   | State | Updated_at                 | Disabled Reason                                                                                                     | Forced down |
+--------------------------------------+----------------+----------------------+----------+----------+-------+----------------------------+----------------------------------------------------------------------------------------------------------------+-------------+
| 5aad1db3-7d9d-4991-811a-1a1ba0621725 | nova-scheduler | kypo-virtual-machine | internal | enabled  | up    | 2022-04-04T03:53:34.000000 | -                                                                                                                 | False       |
| 8ae129c9-c1e8-4eb8-9626-176a856d1217 | nova-conductor | kypo-virtual-machine | internal | enabled  | up    | 2022-04-04T03:53:36.000000 | -                                                                                                               | False       |
| e935f63b-a55b-4ac9-a9e2-38e7c431e19a | nova-compute   | kypo-virtual-machine | nova     | disabled | down  | 2022-03-31T11:47:57.000000 | AUTO: Failed to connect to libvirt: unable to connect to server at '192.168.211.133:16509': Connection refused | False       |
+--------------------------------------+----------------+----------------------+----------+----------+-------+----------------------------+----------------------------------------------------------------------------------------------------------------+-------------+

Status of libvirtd

systemctl status libvirtd
● libvirtd.service - Virtualization daemon
     Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor   preset: enabled)
     Active: active (running) since Mon 2022-04-04 03:40:41 GMT; 17min ago
TriggeredBy: ● libvirtd-admin.socket
         ● libvirtd.socket
         ● libvirtd-ro.socket
   Docs: man:libvirtd(8)
         https://libvirt.org
   Main PID: 995 (libvirtd)
   Tasks: 19 (limit: 32768)
   Memory: 31.2M
   CGroup: /system.slice/libvirtd.service
             ├─ 995 /usr/sbin/libvirtd
             ├─1256 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq /default.conf --leasefile-ro >
         └─1257 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro >

Apr 04 03:45:11 kypo-virtual-machine libvirtd[995]: ethtool ioctl error: No such device
Apr 04 03:45:11 kypo-virtual-machine libvirtd[995]: ethtool ioctl error: No such device
Apr 04 03:45:11 kypo-virtual-machine libvirtd[995]: ethtool ioctl error: No such device
Apr 04 03:45:11 kypo-virtual-machine libvirtd[995]: ethtool ioctl error: No such device

I am creating two instances with following commands:

openstack stack create --wait -e kypo-base-params.yml -t heat/kypo-head.yml kypo-head-stack
openstack stack create --wait -e kypo-base-params.yml -t heat/kypo-proxy-jump.yml kypo-proxy-jump-stack

cat kypo-head.yml 
heat_template_version: "2018-08-31"

description: "KYPO Head server."

parameters:
  kypo: {type: json}

resources:
  kypo-head:
    type: "OS::Nova::Server"
     properties:
      flavor: { get_param: [kypo, kypo_base_head_flavor] }
      image: { get_param: [kypo, kypo_base_head_image] }
      name: "kypo-head"
      networks:
        - port: { get_resource: kypo-head-port }
      key_name: { get_param: [kypo, kypo_base_head_keypair_name] }
    depends_on:
      - kypo-head-port
  kypo-head-port:
    type: "OS::Neutron::Port"
    properties:
      admin_state_up: true
      allowed_address_pairs:
        - ip_address: "0.0.0.0/0"
      network_id: kypo-base-net
      security_groups:
        - kypo-base-head-sg
  kypo-head-floating-ip:
    type: "OS::Neutron::FloatingIPAssociation"
    properties:
      floatingip_id: { get_param: [kypo, kypo_base_head_floating_ip_id] }
      port_id: { get_resource: kypo-head-port }
    depends_on:
      - kypo-head-port

(kypo-crp-openstack-base) root@kypo-virtual-machine:~/kypo-crp-openstack-base# cat kypo-base-params.yml 
parameters:
  kypo:
    kypo_base_network_cidr: 192.168.64.0/18
    kypo_base_public_router_ip: 192.168.64.1
    kypo_base_private_router_ip: 192.168.64.2
    kypo_base_network_allocation_pools: [{start: 192.168.64.10, end: 192.168.127.254}]
    kypo_base_external_public_network_name: public
    kypo_base_dns_nameservers: [1.1.1.1,1.0.0.1]

    kypo_base_head_keypair_name: admin_kypo-base-key
    kypo_base_head_flavor: standard.small
    kypo_base_head_image: ubuntu-focal-x86_64
    kypo_base_head_floating_ip_id: 65569d91-5d76-4b72-95cf-eaaeaab65740

    kypo_base_proxy_keypair_name: admin_kypo-base-key
    kypo_base_proxy_flavor: standard.small
    kypo_base_proxy_image: ubuntu-focal-x86_64
    kypo_base_proxy_floating_ip_id: 25601de5-6c84-4ec4-9821-8c02ba5548ed

Following are images and flavors I have for those instances:

(kypo-crp-openstack-base) root@kypo-virtual-machine:~/kypo-crp-openstack-base/heat# openstack image list
+--------------------------------------+---------------------+--------+
| ID                                   | Name                | Status |
+--------------------------------------+---------------------+--------+
| 6e7467cf-44e6-4fed-9846-3cd4b59bd7c7 | debian-9-x86_64     | active |
| 0e4c0582-9d46-4e60-995b-323884976578 | ubuntu-focal-x86_64 | active |
+--------------------------------------+---------------------+--------+
(kypo-crp-openstack-base) root@kypo-virtual-machine:~/kypo-crp-openstack-base/heat# openstack flavor list
+----+-----------------+------+------+-----------+-------+-----------+
| ID | Name            |  RAM | Disk | Ephemeral | VCPUs | Is Public |
+----+-----------------+------+------+-----------+-------+-----------+
| 1  | csirtmu.tiny1x2 | 2048 |   10 |         0 |     1 | True      |
| 2  | standard.small  | 2048 |   10 |         0 |     1 | True      |
| 3  | standard.medium | 2048 |   10 |         0 |     1 | True      |
| 4  | standard.large  | 2048 |   10 |         0 |     1 | True      |
+----+-----------------+------+------+-----------+-------+-----------+

Answer 1

The root cause was mentioned in libvirtd.log file which said: Failed to acquire pid file '/run/libvirtd.pid': Resource temporarily unavailable

I found a libvirtd.pid file already existed in /var/run folder. I renamed it with command mv libvird.pid libvird.pid.old and created instances again. Every thing went fine.