- OpenStack services are no longer able to establish a connection to the database.
- The following error can be observed in the respective OpenStack component logs (Cinder, Nova, Glance, etc).
SQL connection failed. 3 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
- HAProxy shows the status of the bk-mysql backend to be "DOWN".
Note: If you're uncertain how to check the status of the backend in HAProxy, see: How to Check HAProxy Statistics.
- Platform9 Managed OpenStack - v3.11
The PMO upgrade to version 3.11 made the change from Switcher to HAProxy as the component managing the connections between the hosts and the Management Plane. HAProxy has a health check in place for the MySQL (database) backend. If the health check fails then it attempts to find a backup DB for the service, if it doesn't exist then that backend goes down. There is an upstream bug that causes HAProxy to stay in this state and not attempt the health check again.
- Reload the haproxy service.
# systemctl reload haproxy
- Verify the normal operation of the OpenStack components has resumed.