OpenStack虚拟机迁移失败及数据库连接问题排查
一、问题现象
在进行虚拟机迁移过程中，目标计算节点的nova-compute日志和cinder-volume日志均报错，导致迁移失败。
二、错误日志分析
1. nova-compute日志
2018-11-28 10:12:01.120 7 ERROR nova.volume.cinder [req-8d60395f-5c21-4699-bd5e-31b921acb378 90dea4aefe0c44e19cea9b3cbf0a37c2 3c4c8987959d465285c6e5e1db72e5a5 - default default] Delete attachment failed for attachment e6b83198-8286-4925-b1d8-1b09b0ae9f79. Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-9f348d0a-6afd-4ab9-a11f-de2aae655ca6) Code: 500: ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-9f348d0a-6afd-4ab9-a11f-de2aae655ca6)
...
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-9f348d0a-6afd-4ab9-a11f-de2aae655ca6)
2. cinder-volume日志
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server [req-9f348d0a-6afd-4ab9-a11f-de2aae655ca6 90dea4aefe0c44e19cea9b3cbf0a37c2 3c4c8987959d465285c6e5e1db72e5a5 - - -] Exception during message handling: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8)
...
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8)
3. 日志对比分析
通过对比正常迁移和异常迁移的日志，发现问题集中在Cinder卷的卸载流程中。具体表现为在尝试终止卷连接时发生数据库连接中断。
三、Cinder卷卸载流程
上述流程图显示，卷在卸载过程中发生错误，具体在detach() -> get vol info by id -> terminate connection(call)环节失败。
四、代码分析
在cinder-volume容器中，查看相关代码发现：
def _receive_data(self, num_bytes):
self._sock.settimeout(self._read_timeout)
while True:
try:
data = self._rfile.receive(num_bytes)
break
except (IOError, OSError) as e:
if e.errno == errno.EINTR:
continue
self._force_close()
raise err.OperationalError(
CR.CR_SERVER_LOST,
"Lost connection to MySQL server during query (%s)" % (e,))
if len(data) < num_bytes:
self._force_close()
raise err.OperationalError(
CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
return data
五、解决方案
调整MySQL数据库的max_allowed_packet参数以允许更大的数据包传输：
[root@hci-test01 templates]# vi /usr/share/kolla-ansible/ansible/roles/mariadb/templates/galera.cnf.j2
[server]
# 增加以下配置
max_allowed_packet=32M
六、相关知识链接
- 数据库基础：数据库基础知识
- 数据库问题分析：数据库连接问题排查
- Cinder流程图：Cinder卷管理流程
