Octavia：尝试删除不可变负载均衡器

Question

I have a loadbalancer (see status below) that I want to delete. I already deleted the instances in its pool. Full disclosure: This is on a Devstack which I rebooted, and where I recreated the lb-mgmt-network routing manually. I may have overlooked a detail after the reboot. The loadbalancer worked before the reboot.

The first step to delete the loadbalancer is to delete its pool members. This fails as follows:

$ alias olb='openstack loadbalancer'
$ olb member delete website-pool 08f55..
Load Balancer 1ff... is immutable and cannot be updated. (HTTP 409)

What can I do to make it mutable?

Below, see the loadbalancer's status after recreating the o-hm0 route and restarting the amphora. Its provisioning status is ERROR, but according to the API , this should enable me to delete it:

$ olb status show kubelb
{
    "loadbalancer": {
        "id": "1ff7682b-3989-444d-a1a8-6c91aac69c45",
        "name": "kubelb",
        "operating_status": "ONLINE",
        "provisioning_status": "ERROR",
        "listeners": [
            {
                "id": "d3c3eb7f-345f-4ded-a7f8-7d97e3af0fd4",
                "name": "weblistener",
                "operating_status": "ONLINE",
                "provisioning_status": "ACTIVE",
                "pools": [
                    {
                        "id": "9b0875e0-7d16-4ebc-9e8d-d1b90d4264a6",
                        "name": "website-pool",
                        "provisioning_status": "ACTIVE",
                        "operating_status": "ONLINE",
                        "members": [
                            {
                                "id": "08f55bba-260a-4b83-ad6d-f9d6b44f0e2c",
                                "name": "",
                                "operating_status": "NO_MONITOR",
                                "provisioning_status": "ACTIVE",
                                "address": "172.16.0.21",
                                "protocol_port": 80
                            },
                            {
                                "id": "f7665e90-dad0-480e-8ef4-65e0a042b9fa",
                                "name": "",
                                "operating_status": "NO_MONITOR",
                                "provisioning_status": "ACTIVE",
                                "address": "172.16.0.22",
                                "protocol_port": 80
                            }
                        ]
                    }
                ]
            }
        ]
    }
}

Answer 1

When you have a load balancer in ERROR state you have two options:

1. Delete the load balancer using the cascade delete option (--cascade on the cli).
2. Use the failover API to tell Octavia to repair the load balancer once your cloud is fixed.

In Octavia, operating status is a measured/observed status. If they don't go ONLINE it is likely that there is a network configuration issue with the lb-mgmt-net and the health heartbeat messages (UDP 5555) are not making it back to the health manager controller.

That said, devstack is not setup to work after a reboot. Specifically neutron and the network interfaces will be in an improper state. As you have found, you can manually reconfigure those and usually get things working again.

Answer 2

If I understand documentation and source code right, a loadbalancer in provisioning status ERROR can be deleted but not modified . Unfortunately, it can only be deleted after its pools and listeners have been deleted, which would modify the loadbalancer. Looks like a chicken and egg problem to me. I "solved" this by recreating the cloud from scratch. I guess I could also have cleaned up the database.

An analysis of the stack.sh log file revealed that a few additional steps were needed to make the Devstack cloud reboot-proof. To make Octavia ready:

• Create /var/run/octavia , owned by the stack user
• Ensure o-hm0 is up
• Give o-hm0 the correct MAC and IP addresses, both found in the details of Neutron port octavia-health-manager-standalone-listen-port
• add netfilter rules for traffic coming from o-hm0

At this point, I feel I can reboot Devstack and still have functioning load balancers. Strangely, all load balancers' operating_status (as well as their listeners' and pools' operating_status ) is OFFLINE. However, that doesn't prevent them from working. I have not found out how to make that ONLINE.

In case anybody is interested, below is the script I use after rebooting Devstack. In addition, I also changed the Netplan configuration so that br-ex gets the server's IP address (further below).

restore-devstack script:

$ cat restore-devstack

source ~/devstack/openrc admin admin

if losetup -a | grep -q /opt/stack/data/stack-volumes
then echo loop devices are already set up
else
    sudo losetup -f --show --direct-io=on /opt/stack/data/stack-volumes-default-backing-file
    sudo losetup -f --show --direct-io=on /opt/stack/data/stack-volumes-lvmdriver-1-backing-file
    echo restarting Cinder Volume service
    sudo systemctl restart devstack@c-vol
fi
sudo lvs
openstack volume service list
echo
echo recreating /var/run/octavia
sudo mkdir /var/run/octavia
sudo chown stack /var/run/octavia
echo
echo setting up the o-hm0 interface
if ip l show o-hm0 | grep -q 'state DOWN'
then sudo ip l set o-hm0 up
else echo o-hm0 interface is not DOWN
fi

HEALTH_IP=$(openstack port show octavia-health-manager-standalone-listen-port -c fixed_ips -f yaml | grep ip_address | cut -d' ' -f3)
echo health monitor IP is $HEALTH_IP
if ip a show dev o-hm0 | grep -q $HEALTH_IP
then echo o-hm0 interface has IP address
else sudo ip a add ${HEALTH_IP}/24 dev o-hm0
fi
HEALTH_MAC=$(openstack port show octavia-health-manager-standalone-listen-port -c mac_address -f value)
echo health monitor MAC is $HEALTH_MAC
sudo ip link set dev o-hm0 address $HEALTH_MAC
echo o-hm0 MAC address set to $HEALTH_MAC
echo route to loadbalancer network:
ip r show 192.168.0.0/24
echo
echo fix netfilter for Octavia
sudo iptables -A INPUT -i o-hm0 -p udp -m udp --dport 20514 -j ACCEPT
sudo iptables -A INPUT -i o-hm0 -p udp -m udp --dport 10514 -j ACCEPT
sudo iptables -A INPUT -i o-hm0 -p udp -m udp --dport 5555 -j ACCEPT
echo fix netfilter for Magnum
sudo iptables -A INPUT -d 192.168.1.200/32 -p tcp -m tcp --dport 443 -j ACCEPT
sudo iptables -A INPUT -d 192.168.1.200/32 -p tcp -m tcp --dport 80 -j ACCEPT
sudo iptables -A INPUT -d 192.168.1.200/32 -p tcp -m tcp --dport 9511 -j ACCEPT

Netplan config:

$ cat /etc/netplan/00-installer-config.yaml
# This is the network config written by 'subiquity'
network:
  ethernets:
    enp1s0:
      dhcp4: no
    br-ex:
      addresses: [192.168.1.200/24]
      nameservers: { addresses: [192.168.1.16,1.1.1.1] }
      gateway4: 192.168.1.1
  version: 2