Kolla Ansible OpenStack deployment help

The Gecko

Active Member
Jan 4, 2015
107
81
28
43
I'm having an absolute bitch of a time standing up a multi-node lab deployment of Kolla Ansible OpenStack (KAO). All steps of deployment work great until the deployment step, which always fails at the same spot: "TASK [nova-cell : Waiting for nova-compute services to register themselves]".

1661948161181.png

I investigated what was failing and how and I don't think this is an error with nova-compute. Having dug into the logs, it looks more like an authentication issue involving libvirt, SASL, and possibly the OpenStack Keystone service.

Kolla1: \var\lib\docker\volumes\kolla_logs\_data\libvirt\libvirtd.log
Code:
2022-08-30 20:51:59.832+0000: 58177: error : virNetSASLSessionServerStep:603 : authentication failed: Failed to start SASL negotiation: -20 (SASL(-13): user not found: unable to canonify user and get auxprops)
2022-08-30 20:51:59.832+0000: 58177: error : remoteDispatchAuthSaslStep:3957 : authentication failed: authentication failed
2022-08-30 20:51:59.833+0000: 58156: error : virNetSocketReadWire:1793 : End of file while reading data: Input/output error
Kolla1: \var\lib\docker\volumes\kolla_logs\_data\keystone\keystone-apache-public-error.log
Code:
2022-08-30 20:47:55.800752 2022-08-30 20:47:55.800 21 WARNING keystone.server.flask.application [req-098b3bd8-1649-4f1a-b450-d34d31f41b4b 6adce8f2889f4d25b179d38d8b3e9a2e - - default -] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin.\x1b[00m
2022-08-30 20:47:56.253671 2022-08-30 20:47:56.253 22 WARNING keystone.server.flask.application [req-bd8cb03c-1e7d-4a8a-8a81-32ef63ac3fd3 6adce8f2889f4d25b179d38d8b3e9a2e - - default -] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder.\x1b[00m
2022-08-30 20:48:54.093718 2022-08-30 20:48:54.093 22 WARNING keystone.server.flask.application [req-21595d95-6230-4626-a57c-4382c89af802 6adce8f2889f4d25b179d38d8b3e9a2e - - default -] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service.\x1b[00m
2022-08-30 20:48:58.704259 2022-08-30 20:48:58.702 21 WARNING keystone.server.flask.application [req-72852ef4-b6a1-4ecb-b4f1-b9053714939d 6adce8f2889f4d25b179d38d8b3e9a2e - - default -] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service.\x1b[00m
2022-08-30 20:50:29.298592 2022-08-30 20:50:29.298 22 WARNING keystone.server.flask.application [req-591cab45-60ea-4e7a-bb4c-c0d1ac9a07f0 6adce8f2889f4d25b179d38d8b3e9a2e - - default -] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin.\x1b[00m
2022-08-30 20:50:29.582448 2022-08-30 20:50:29.582 21 WARNING keystone.server.flask.application [req-38301601-3955-426a-b557-0e398ab79f29 6adce8f2889f4d25b179d38d8b3e9a2e - - default -] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova.\x1b[00m
Kolla1: \var\lib\docker\volumes\kolla_logs\_data\mariadb\mariadb.log
Code:
2022-08-30 20:43:59 12 [Warning] Aborted connection 12 to db: 'unconnected' user: 'unauthenticated' host: 'kolla1.openstack.cg' (This connection closed normally without authentication)
2022-08-30 20:44:04 14 [Warning] Access denied for user 'haproxy'@'kolla1.openstack.cg' (using password: NO)
2022-08-30 20:46:05 202 [Warning] Aborted connection 202 to db: 'keystone' user: 'keystone' host: 'kolla1.openstack.cg' (Got an error reading communication packets)
2022-08-30 20:47:30 357 [Warning] Aborted connection 357 to db: 'glance' user: 'glance' host: 'kolla1.openstack.cg' (Got an error reading communication packets)
2022-08-30 20:49:11 604 [Warning] Aborted connection 604 to db: 'placement' user: 'placement' host: 'kolla1.openstack.cg' (Got an error reading communication packets)
2022-08-30 20:49:28 646 [Warning] Aborted connection 646 to db: 'nova_api' user: 'nova_api' host: 'kolla1.openstack.cg' (Got an error reading communication packets)
2022-08-30 20:49:45 706 [Warning] Aborted connection 706 to db: 'nova_cell0' user: 'nova' host: 'kolla1.openstack.cg' (Got an error reading communication packets)
2022-08-30 20:50:03 737 [Warning] Aborted connection 737 to db: 'nova' user: 'nova' host: 'kolla1.openstack.cg' (Got an error reading communication packets)
2022-08-30 20:50:06 759 [Warning] Aborted connection 759 to db: 'nova_api' user: 'nova_api' host: 'kolla1.openstack.cg' (Got an error reading communication packets)
2022-08-30 21:48:44 515 [Warning] Aborted connection 515 to db: 'cinder' user: 'cinder' host: 'kolla1.openstack.cg' (Got an error reading communication packets)
I am baffled as to why this is happening. I've not seen any reported bugs that mirror my errors, so I'm thinking there is something wrong with my KAO global.yml file, or possibly the way I prepared the VMs for KAO + Ceph.

At this point, you are probably looking for the globals.yml file. This file is inside the Beachhead.zip file.

A little bit about my lab environment:
  • Kolla nodes are VMs in Hyper-V
    • Beachhead
    • Kolla1
    • Kolla2
    • Kolla3
  • Beachhead Roles:
    • Deployment orchestrator
    • DNS Server (Bind9)
  • Kolla Ansible OpenStack (Yoga 14.3.0)
  • Ceph (Quincy 17.1.0)
  • Ubuntu 20.04 LTS
  • Python version 3.8.10
  • Ansible version 2.12.8 (core) or 5.x (community)
Hosting everything in Hyper-V lets me roll back VMs to a previous state and try again, saving massive amounts of time. Plus, I can use PowerShell to script the creation of Hyper-V virtual switches, and all VMs.
 

Attachments

lungj

New Member
Sep 21, 2022
1
0
1
Not sure if you're still stuck or if you're the one that filed this bug report, but this might resolve your issue: change your `/etc/hosts` so that your node's FQDN appears last on the host's line. I ran into this issue when deploying from the same Kolla config files that were working in another situation (literally unchanged -- I destroyed an experimental instance that I was doing nested virtualization in and tried to deploy OpenStack onto bare metal).
 

The Gecko

Active Member
Jan 4, 2015
107
81
28
43
I will absolutely look into this. Thank you.

Also, I've decided to share my (broken) deployment steps in hope that someone could see where I was taking a wrong turn. It's posted here on GitHub.
 

The Gecko

Active Member
Jan 4, 2015
107
81
28
43
I looked at the hosts file from one of the Kolla Ansible nodes. It meets the qualifications for the bug post you linked. I'll have to work up some new steps in my deployment instructions to reconfigure the hosts file, then try another deployment.