Upgrade of vCenter 6.0 to 6.5

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

dswartz

Active Member
Jul 14, 2011
610
79
28
Somewhat off topic, but I tried several times to upgrade my 6.0 appliance. Failed every time with some 'an internal error has occurred' message (or words to that effect.) Google revealed nothing similar to this. I ended up deploying a 6.5 appliance from scratch, but haven't yet rolled it out to replace the 6.0 appliance - mainly since I would need to do some hacking on my Veeam B&R appliance to replace the Vcenter server it is referencing...
 

BSDguy

Member
Sep 22, 2014
168
7
18
53
Power off and delete the vCSA 6.5 VM. Power on the old 6.0 vCSA.
Host 2 might connect to the vCenter ok but if not, disconnect and reconnect it. I suspect it will think it is connected to a differect vCenter and not connect to the 6.0 instance.
Host 1 will need to be disconnected or removed from vCenter before you connect the freshly built 6.0u2. Certificate etc will be different.
I'm busy powering up the vCSA 6 appliance so let's see what happens. I'll report back.

Thanks for the help.
 

BSDguy

Member
Sep 22, 2014
168
7
18
53
Somewhat off topic, but I tried several times to upgrade my 6.0 appliance. Failed every time with some 'an internal error has occurred' message (or words to that effect.) Google revealed nothing similar to this. I ended up deploying a 6.5 appliance from scratch, but haven't yet rolled it out to replace the 6.0 appliance - mainly since I would need to do some hacking on my Veeam B&R appliance to replace the Vcenter server it is referencing...
Not off topic at all. I have to say, this upgrade has been a nightmare from the very beginning. I've done a few upgrades in the past with no issues but this has just been terrible. I really don't want to rebuild my vSphere setup again to get on version 6.5.

I also came back empty handed after y research of the errors/issues I experienced.

My Veeam backups are now broken too which is so odd considering they worked after the upgrade to vCSA 6.5.
 
  • Like
Reactions: dswartz

BSDguy

Member
Sep 22, 2014
168
7
18
53
So I powered off my vCSA 6.5 appliance and turned on the vCSA 6.0 appliance. It seems to be working again! My second host was connected and seems to be fine. I have since reinstalled ESXi 6.0U2 on my first host (just a basic install) and added it back into my cluster and it hasn't disconnected in 20min. So I have absolutlely no idea why this kept happening with the vCSA 6.5 appliance?!

I'm not even sure how to proceed with upgrading to 6.5.

FYI: If you use IPv6, remove the address from the vCSA 6.0 appliance AND remove and IPv6 DNS records BEFORE the migration
 

BSDguy

Member
Sep 22, 2014
168
7
18
53
Heh, you can say that again. The funny thing is I read up quite a bit about it before attempting to upgrade and it still went horribly wrong.
 

RyC

Active Member
Oct 17, 2013
359
88
28
Shot in the dark, you don't have any VMs with host USB devices attached do you?
 

dswartz

Active Member
Jul 14, 2011
610
79
28
Whoa, I do, actually. A windows 7 pro VM with USB controller. Had totally forgotten, since we never actually used it. How does that play here?
 

RyC

Active Member
Oct 17, 2013
359
88
28
Hmm this is more for the vCSA 6.5 not starting the vpxd service (resulting in the "could not connect to one or more vCenter systems" message), rather than updating. And I'm not sure if a controller triggers it or if you have to actually have a USB device attached to a VM (like a UPS, for example). There's a nasty bug in 6.5 where vpxd service won't start if you have any VMs with a host USB device attached. The solution is to log into the host web client, remove the offending VM(s) from inventory, and start/restart the vCenter appliance. I think your issue may be something different if it couldn't upgrade in the first place :(
 
  • Like
Reactions: Tom5051

dswartz

Active Member
Jul 14, 2011
610
79
28
bummer, well, i deleted the usb controller from the windows guest. i'll try the upgrade again sometime soon... thanks!
 

BSDguy

Member
Sep 22, 2014
168
7
18
53
Shot in the dark, you don't have any VMs with host USB devices attached do you?
I have a VM with a USB UPS attached!!

upload_2017-1-24_8-28-12.png

Could this be the cause of all my issues? The funny thing is, vCSA 6.5 died on me when I tried to shut this VM down. I have the USB UPS physically plugged into host 1 but the VM was running on host 2 after the upgrade.
 
Last edited:

RyC

Active Member
Oct 17, 2013
359
88
28
I have a VM with a USB UPS attached!!

View attachment 4366

Could this be the cause of all my issues? The funny thing is, vCSA 6.5 died on me when I tried to shut this VM down. I have the USB UPS physically plugged into host 1 but the VM was running on host 2 after the upgrade.
Shut down the vCSA and remove the VM from the host inventory (using the host web client if necessary, you can add the VM back later if vCenter comes up). Make sure you remove any VM with a USB device from all hosts, just removing the USB device from the VM but leaving the VM registered will still trigger the bug.

Boot the vCSA and try logging in to either web client and hopefully you won't get the "could not find one or more vCenter systems" message.
 

BSDguy

Member
Sep 22, 2014
168
7
18
53
Shut down the vCSA and remove the VM from the host inventory (using the host web client if necessary, you can add the VM back later if vCenter comes up). Make sure you remove any VM with a USB device from all hosts, just removing the USB device from the VM but leaving the VM registered will still trigger the bug.

Boot the vCSA and try logging in to either web client and hopefully you won't get the "could not find one or more vCenter systems" message.
Were you referring to shutting down the vCSA 6.5 appliance? I've rolled back to the vCSA 6 appliance and deleted the vCSA 6.5.

So I'm assuming moving forward, BEFORE upgrading vCenter to 6.5 I should unregister the VM with the USB device? And then upgrade/migrate to 6.5?
 

RyC

Active Member
Oct 17, 2013
359
88
28
Were you referring to shutting down the vCSA 6.5 appliance? I've rolled back to the vCSA 6 appliance and deleted the vCSA 6.5.

So I'm assuming moving forward, BEFORE upgrading vCenter to 6.5 I should unregister the VM with the USB device? And then upgrade/migrate to 6.5?
Yes, although I successfully upgraded to 6.5 with the VM with USB attached in the host inventory. It was when I shut down/restarted the vCSA 6.5 VM for the very first time that it never came back up until I removed the VM from the host. It's probably safer to remove the VM and upgrade anyway.
 

BSDguy

Member
Sep 22, 2014
168
7
18
53
Ok, thanks. I'll unregister the VM with the USB UPS in vCenter before trying the upgrade again!

It was so weird because the upgrade to vCSA 6.5 was working for a day and then when I logged in the following day it was all screwed (even a reboot couldn't save it).
 

BSDguy

Member
Sep 22, 2014
168
7
18
53
So, I attempted an upgrade to vCenter 6.5 again today and so far it is working. Wanna let it run for a couple days before I get excited!

I changed the following when I did the upgrade this time:

1) Unregistered VM with USB device before upgrading

2) Removed IPv6 address from vCSA 6 before upgrading

3) Completely disabled HA/DRS

I've also rebooted the upgraded vCSA a few times and it has been fine so far but still want to give it a couple days before upgrading my hosts. Veeam is happy too so backups are working (moved to ReFS for backup repository, damn its fast!)

The only odd thing I have noticed so far is (and maybe this is a GUI bug), here is my HA settings on the cluster in the HTML5 client:

upload_2017-1-25_19-47-14.png

Looks good right? Now check it out in the Flash client:

upload_2017-1-25_19-47-51.png

Is this a bug in the Flash client maybe?

One other thing I have noticed which is odd, when I set an IPv6 address again in the vCSA 6 appliance it loses it after a reboot?

Also, I don't have an option to upgrade my VMs VMware Tools. Normally after an upgrade if you right click a VM you get the option to update the tools but mine keeps saying I am up to date even though I know for sure I am two versions behind. Will this option come after updating my hosts maybe?
 

Attachments

Tom5051

Active Member
Jan 18, 2017
359
79
28
46
The tools are host version dependant. Have you upgraded the hosts to 6.5 yet?
 

BSDguy

Member
Sep 22, 2014
168
7
18
53
Further update:

So as a test I shutdown the first host in my cluster and removed the bootable ESXi USB key and replaced it with a temporary one that I installed ESXi 6.5 on. After the install I proceeded to add the 6.5 host to the 6.5 vCenter. And guess what?? Straight away I had disconnects between vCenter 6.5 and ESXi 6.5!

I tried unplugging the USB UPS altogether and checking forward/reverse DNS, firewall rules etc but nothing made a difference until...

IPv6!!

I run a dual stack IPv4/IPv6 network and my ESXi host and vCenter has an IPv4 AND IPv6 address. As soon as I removed the IPv6 address (and the relevant IPv6 AAAA/PTR records from DNS) from my ESXi 6.5 host ALL the disconnect errors disappeared instantly.

So then I tried to assign ONLY an IPv6 address to the ESXi 6.5 host and this failed with an error saying the default gateway was incorrect (even though it WAS correct).

So I'm wondering if IPv6 is broken in vSphere 6.5?

I also noticed that whenver I reboot the vCSA 6.5, it loses the IPv6 address.

Has anyone else had any experience with running dual stack IPv4/IPv6 with vSphere 6.5? I've been using dual stack with vSphere 6.0U2 for 4 months without a single issue so I am surprised/shocked that this is/may be the case with 6.5.

Here's the error when setting the static IPv6 address on the ESXi 6.5 host:

upload_2017-1-27_20-5-6.png
 
Last edited: