I'm having an absolute bitch of a time standing up a multi-node lab deployment of Kolla Ansible OpenStack (KAO). All steps of deployment work great until the deployment step, which always fails at the same spot: "TASK [nova-cell : Waiting for nova-compute services to register themselves]".
I investigated what was failing and how and I don't think this is an error with nova-compute. Having dug into the logs, it looks more like an authentication issue involving libvirt, SASL, and possibly the OpenStack Keystone service.
Kolla1: \var\lib\docker\volumes\kolla_logs\_data\libvirt\libvirtd.log
Kolla1: \var\lib\docker\volumes\kolla_logs\_data\keystone\keystone-apache-public-error.log
Kolla1: \var\lib\docker\volumes\kolla_logs\_data\mariadb\mariadb.log
I am baffled as to why this is happening. I've not seen any reported bugs that mirror my errors, so I'm thinking there is something wrong with my KAO global.yml file, or possibly the way I prepared the VMs for KAO + Ceph.
At this point, you are probably looking for the globals.yml file. This file is inside the Beachhead.zip file.
A little bit about my lab environment:
I investigated what was failing and how and I don't think this is an error with nova-compute. Having dug into the logs, it looks more like an authentication issue involving libvirt, SASL, and possibly the OpenStack Keystone service.
Kolla1: \var\lib\docker\volumes\kolla_logs\_data\libvirt\libvirtd.log
Code:
2022-08-30 20:51:59.832+0000: 58177: error : virNetSASLSessionServerStep:603 : authentication failed: Failed to start SASL negotiation: -20 (SASL(-13): user not found: unable to canonify user and get auxprops)
2022-08-30 20:51:59.832+0000: 58177: error : remoteDispatchAuthSaslStep:3957 : authentication failed: authentication failed
2022-08-30 20:51:59.833+0000: 58156: error : virNetSocketReadWire:1793 : End of file while reading data: Input/output error
Code:
2022-08-30 20:47:55.800752 2022-08-30 20:47:55.800 21 WARNING keystone.server.flask.application [req-098b3bd8-1649-4f1a-b450-d34d31f41b4b 6adce8f2889f4d25b179d38d8b3e9a2e - - default -] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin.\x1b[00m
2022-08-30 20:47:56.253671 2022-08-30 20:47:56.253 22 WARNING keystone.server.flask.application [req-bd8cb03c-1e7d-4a8a-8a81-32ef63ac3fd3 6adce8f2889f4d25b179d38d8b3e9a2e - - default -] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder.\x1b[00m
2022-08-30 20:48:54.093718 2022-08-30 20:48:54.093 22 WARNING keystone.server.flask.application [req-21595d95-6230-4626-a57c-4382c89af802 6adce8f2889f4d25b179d38d8b3e9a2e - - default -] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service.\x1b[00m
2022-08-30 20:48:58.704259 2022-08-30 20:48:58.702 21 WARNING keystone.server.flask.application [req-72852ef4-b6a1-4ecb-b4f1-b9053714939d 6adce8f2889f4d25b179d38d8b3e9a2e - - default -] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service.\x1b[00m
2022-08-30 20:50:29.298592 2022-08-30 20:50:29.298 22 WARNING keystone.server.flask.application [req-591cab45-60ea-4e7a-bb4c-c0d1ac9a07f0 6adce8f2889f4d25b179d38d8b3e9a2e - - default -] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin.\x1b[00m
2022-08-30 20:50:29.582448 2022-08-30 20:50:29.582 21 WARNING keystone.server.flask.application [req-38301601-3955-426a-b557-0e398ab79f29 6adce8f2889f4d25b179d38d8b3e9a2e - - default -] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova.\x1b[00m
Code:
2022-08-30 20:43:59 12 [Warning] Aborted connection 12 to db: 'unconnected' user: 'unauthenticated' host: 'kolla1.openstack.cg' (This connection closed normally without authentication)
2022-08-30 20:44:04 14 [Warning] Access denied for user 'haproxy'@'kolla1.openstack.cg' (using password: NO)
2022-08-30 20:46:05 202 [Warning] Aborted connection 202 to db: 'keystone' user: 'keystone' host: 'kolla1.openstack.cg' (Got an error reading communication packets)
2022-08-30 20:47:30 357 [Warning] Aborted connection 357 to db: 'glance' user: 'glance' host: 'kolla1.openstack.cg' (Got an error reading communication packets)
2022-08-30 20:49:11 604 [Warning] Aborted connection 604 to db: 'placement' user: 'placement' host: 'kolla1.openstack.cg' (Got an error reading communication packets)
2022-08-30 20:49:28 646 [Warning] Aborted connection 646 to db: 'nova_api' user: 'nova_api' host: 'kolla1.openstack.cg' (Got an error reading communication packets)
2022-08-30 20:49:45 706 [Warning] Aborted connection 706 to db: 'nova_cell0' user: 'nova' host: 'kolla1.openstack.cg' (Got an error reading communication packets)
2022-08-30 20:50:03 737 [Warning] Aborted connection 737 to db: 'nova' user: 'nova' host: 'kolla1.openstack.cg' (Got an error reading communication packets)
2022-08-30 20:50:06 759 [Warning] Aborted connection 759 to db: 'nova_api' user: 'nova_api' host: 'kolla1.openstack.cg' (Got an error reading communication packets)
2022-08-30 21:48:44 515 [Warning] Aborted connection 515 to db: 'cinder' user: 'cinder' host: 'kolla1.openstack.cg' (Got an error reading communication packets)
At this point, you are probably looking for the globals.yml file. This file is inside the Beachhead.zip file.
A little bit about my lab environment:
- Kolla nodes are VMs in Hyper-V
- Beachhead
- Kolla1
- Kolla2
- Kolla3
- Beachhead Roles:
- Deployment orchestrator
- DNS Server (Bind9)
- Kolla Ansible OpenStack (Yoga 14.3.0)
- Ceph (Quincy 17.1.0)
- Ubuntu 20.04 LTS
- Python version 3.8.10
- Ansible version 2.12.8 (core) or 5.x (community)
Attachments
-
605.7 KB Views: 1
-
841.4 KB Views: 0
-
16.3 KB Views: 1