Upgrade of vCenter 6.0 to 6.5

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

BSDguy

Member
Sep 22, 2014
168
7
18
53
Stupid question but what OS did you run your upgrade from? I'm doing it from a Windows Server 2016 VM but have also tried a patched Windows 10 Enterprise LSTB and both fail during the upgrade.
 

RyC

Active Member
Oct 17, 2013
359
88
28
I ran into this issue and what ended up being the solution was ssh-ing into the appliance and resetting the administrator password and/or password expiration. The error message the installer gives is terrible.
 

BSDguy

Member
Sep 22, 2014
168
7
18
53
How does one (correctly) change the root password and check the password expiration once you have SSH'd into the vCSA?

PS: I just ran "chage -l root" while SSH'd into the vCSA and this is the output:

Minimum: 0
Maximum: 99999
Warning: 7
Inactive: -1
Last Change: Jan 21, 2017
Password Expires: Never
Password Inactive: Never
Account Expires: Never

I did change the password earlier but did this using the https://vcsa.domain.com:5480 interface and setting the password to not expire.

Do I still need to change the password using SSH rather?

Also, you mentioned the "administrator" account. Did you mean this? Or did you mean the root account?
 
Last edited:

RyC

Active Member
Oct 17, 2013
359
88
28
Sorry, I got the root account and SSO administrator account mixed up. I think it's the root account that needed to be changed, so I'm not sure now...
 

Doozer

New Member
Jan 21, 2017
1
0
1
54
@BSDguy
I had this problem too a few minutes ago with our second vCenter Server Appliance. At the first appliance I could do the upgrade 6.0 -> 6.5 a few hours ago without any problems. But the second Appliance always had the phenomenon that the upgrade installer said "Invalid Appliance (OS) root password."

Cause: The password for the root user in the second appliance seems to be to complex: E5a82W@s3Knr1H
After changing (via the console --> F2 --> Configure Root Password) the root password to a simple word, e.g. "relation", the upgade installer did his job.

Strange...
Again a stupid problem, that wasted me hours of my life :-(
 

BSDguy

Member
Sep 22, 2014
168
7
18
53
Thanks for all the suggestions! I FINALLY managed to get past this screen by leaving it for a bit and trying again later in the evening and I got past the authentication screen. Not sure what fixed it thought, resetting the password possibly?

Anyway, onto the next issue! My upgrade to vSphere 6.5 is not going well so far ;-( After upgrading to vCenter 6.5 I left it running overnight and it was looking good this morning. Veeam backups worked since I have Veeam 9.5 Update 1 installed but then I decided to upgrade the first ESXi host to 6.5...

The upgrade went fine with no errors but after the host rebooted it kept saying it was not responding in vCenter. No amount of disconnecting/reconnecting helped so I eventually disconnected it and removed it from the inventory.

After readding the host back into vCenter I now get this:

The distributed switch corresponding to the proxy switches "ca 55 39 50 06 2d 31 76-5f 7c d4 56 52 55 0a c0"; "40 7f 39 50 66 d2 b0 f2-6f 09 cb f0 6d c9 04 91"; "da 70 39 50 2d 44 40 cb-c9 48 f4 d1 14 5f 94 c0" on the host does not exist in vCenter Server or does not contain this host.

Go to Host > Configure > Settings > Virtual Switches to manually remove the invalid proxy switches if vCenter Server is not able to automatically remove them.

So I readded the host to the VDS and this *seemed* to fix the issue. I did this for the 3 VDS switches I use.

But now when I try to do a compute vMotion I get this:
Unable to prepare migration.
Network addresses '192.168.30.8' and '2001:xxx:xxxx:30::7' are from different address families.

So the .30 range is my management network. For vMotion I use the .70 range and a dedicated VDS switch and vmkernel for vMotion traffic.

How do I get past this odd error? Why is it trying to use my mgt network to vMotion after the upgrade? My mgr vmkernel doesn't have vMotion enabled on it.

So currently I have vCenter 6.5 running and one of the two hosts is running ESXi 6.5 (the other is still on 6.0 Update 2).

Oh and just to make things even weirder, when I am in vCenter I can see the upgraded host occassionally saying it's not responding??

PS: Here is the events happening on the upgraded ESXi 6.5 host that keeps disconnecting:

upload_2017-1-22_13-21-46.png

I think this has to do with vCenter not being able to "see" the host on UDP port 902 for the heartbeat but I'm just not sure why it's doing this as the networking side hasn't changed at all and the host that hasn't been upgraded (still running 6.0) is still rock solid and hasn't disconnected once, even since the vCenter upgrade.
 
Last edited:

BSDguy

Member
Sep 22, 2014
168
7
18
53
So after a bit more investigation I discovered two things:

1) After the host was upgraded it set ALL my NICs to 1000Mb/Full duplex (they were set to auto negotiate before the upgrade)

2) I have two Supermicro 5028D-TN4T servers which have two 1Gb Intel i350 NICs onboard and on 6.0 it used the "igb" driver. After upgrading to 6.5 it is now using the "igbn" driver (these two NICs are used for my management traffic and VM traffic)

So I set all the NICs to auto negotiate again but this didn't help the continuous disconnecting of the host in vCenter. I then checked my switch and sure enough, every few minutes I can see the Intel i350 ports connected to the switch disconnecting every few minutes.

So what can I do here? Is this a driver issue? I couldn't find an updated driver and am using the ESXi 6.5 driver that comes built in. It's listed as supported on the VMware HCL site.

As a test I set the i350 ports to 1000Mb in vCenter AND on the switch but this didn't help.

Is there anything else I can try? The other ESXi 6.0 host (that I haven't upgraded) has never disconnected (and neither did the upgraded host when running ESXi 6.0).

I'm baffled. I can only think the new driver is problematic or I need to change something?
 
Last edited:

cheezehead

Active Member
Sep 23, 2012
731
176
43
Midwest, US
Duplex mismatching will cause all sorts of fun. Try both sides at auto if it doesn't work then try both sides locked to 1000-full....sometimes there's a 1000-auto option which I would test as well just make sure. If none of those combinations work, you could always manually load the older nic driver on the host.
 

BSDguy

Member
Sep 22, 2014
168
7
18
53
I tried auto on both sides as well as setting it to 1000Mb/Full but I still get the disconnects.

How do I enable/load the old driver so it's the same as before?

It's funny because I can see in the switch logs that as soon as I rebooted the host after it was upgraded to 6.5 I can see all the errors start. There were none before the upgrade.
 

BSDguy

Member
Sep 22, 2014
168
7
18
53
Thanks but I tried installing the older driver but it didn't help. I am rebuilding the host as we speak.
 

BSDguy

Member
Sep 22, 2014
168
7
18
53
Host rebuilt with ESXi 6.5 and re-added to cluster and almost straight away the host would start to disconnect from vCenter every few mintutes.

If I add the host by IP only then it works fine. Add it by FQDN and the host disconnects from vCenter. All forward and reverse DNS are in place for vCenter/ESXi and working.

Huh??? Why is ESXi 6.0 working 100% with FQDN but not ESXi 6.5? I thought it may be IPv6 so I removed the IPv6 address from the ESXi 6.5 management interface and temporarily deleted the IPv6 records but this didn't help.

Any ideas? About to give up with vSphere 6.5 at this rate.
 

nitrobass24

Moderator
Dec 26, 2010
1,087
131
63
TX
Well that is definitely odd (and undesirable), but as it relates to your VCSA upgrade I am confused. Dont you have a 6.0u2 host available?
 

Tom5051

Active Member
Jan 18, 2017
359
79
28
46
Not sure if it is helpful but I ran into a strange issue with an upgrade from 6.0 to 6.5. Web Client would stop working after 30 seconds or so after rebooting the Windows 2012R2 server and the web browser would throw up "Page cannot be displayed".
This server had a GPO set to enforce the 'log on as batch job' security settings.
The vCenter installer adds the accounts to the local GPO but this gets over written when the GPO is refreshed.
Seems to only be an issue if there is a GPO specifically set for 'log on as batch job' and of course doesn't apply to the VC Server Appliance instance.

GPO Path: Computer Configuration -> Windows Settings ->Security Settings ->Local Policies -> User Rights Assignment ->Log on as a batch job

Accounts:
cm
content-library
eam
imagebuilder
mbcs
netdumper
perfcharts
rbd
vapiEndpoint
vmware-vpostgres
vsan-health
vsm
vsphere-client
vsphere-ui
 

BSDguy

Member
Sep 22, 2014
168
7
18
53
Well that is definitely odd (and undesirable), but as it relates to your VCSA upgrade I am confused. Dont you have a 6.0u2 host available?
I currently have my second host still running 6.0 but vCenter is 6.5 still. I have since powered off the first ESXi 6.5 host until I can think of something different to do to get this working!

Maybe a bad arp entry somewhere? I dunno. Really baffled!
 

BSDguy

Member
Sep 22, 2014
168
7
18
53
Not sure if it is helpful but I ran into a strange issue with an upgrade from 6.0 to 6.5. Web Client would stop working after 30 seconds or so after rebooting the Windows 2012R2 server and the web browser would throw up "Page cannot be displayed".
This server had a GPO set to enforce the 'log on as batch job' security settings.
The vCenter installer adds the accounts to the local GPO but this gets over written when the GPO is refreshed.
Seems to only be an issue if there is a GPO specifically set for 'log on as batch job' and of course doesn't apply to the VC Server Appliance instance.

GPO Path: Computer Configuration -> Windows Settings ->Security Settings ->Local Policies -> User Rights Assignment ->Log on as a batch job
Thanks I did come across this somewhere but I am using the appliance so I guess it doesn't apply. Willing to try almost anything at this stage...hah!
 

BSDguy

Member
Sep 22, 2014
168
7
18
53
Just seems to be getting worse and worse this. I just logged into vCenter 6.5 Web client and how get this:

org.apache.http.conn.HttpHostConnectException: Connect to localhost:8085 [localhost/127.0.0.1] failed: Connection refused

Is rolling back as simple as powering the vCenter 6.0 VM?
 

Tom5051

Active Member
Jan 18, 2017
359
79
28
46
Just seems to be getting worse and worse this. I just logged into vCenter 6.5 Web client and how get this:

org.apache.http.conn.HttpHostConnectException: Connect to localhost:8085 [localhost/127.0.0.1] failed: Connection refused

Is rolling back as simple as powering the vCenter 6.0 VM?
You will need to remove the host from the vCenter 6 server as it most likely won't like a 6.5 host, blow the host away and fresh install ESXi 6.0u2 back on it, configure it, connect the fresh host back to VC6.
There might be a better way but that is how I would go about it.
 

BSDguy

Member
Sep 22, 2014
168
7
18
53
So currently I have this in my setup:

Host1 - not running anything currently, was formatted yesterday
Host2 - running ESXi 6.0U2

I then have a vCenter 6.0 appliance that is powered off but is still registered on host 2.

I also have vCenter 6.5 appliance which is also running on hosts 2.

The only host registered in the vCSA 6.5 appliance currently is host 2

The old vCSA 6 appliance has host1 and host2 registered (althought host1 has now been formatted).

So do I power down vCSA 6.5 and then power on vCSA 6.0, is it this simple? Or do I have to disconnect/reconnect host to from vCSA 6.0 or 6.5?

My web client is screwed now even though it was fine last night (a reboot of vCSA 6.5 didn't help):

upload_2017-1-23_13-39-30.png
 

Tom5051

Active Member
Jan 18, 2017
359
79
28
46
Power off and delete the vCSA 6.5 VM. Power on the old 6.0 vCSA.
Host 2 might connect to the vCenter ok but if not, disconnect and reconnect it. I suspect it will think it is connected to a differect vCenter and not connect to the 6.0 instance.
Host 1 will need to be disconnected or removed from vCenter before you connect the freshly built 6.0u2. Certificate etc will be different.