Two APC SmartUPS SMT1500RM2U suddenly no longer accessible via web interface

[mattlach]

Has anyone ever seen this problem before?

I have two of these SMT1500RM2U UPS:es in my rack I bought used on eBay about 2 years ago. They initially came without network cards, so I ordered a pair of AP9630 "UPS Network Management Card 2" and installed them in the UPS:es. When I set them up I configured them via the web interface, and I still check in on them every once and a while using the web interface.

Everything has been working well lately, so It has probably been a few months since the last time I tried logging in.

In the past it worked just fine to access them in my web browser via https://<ip address>, but this time it doesn't. Both of them refuse to connect. I have tried with both Firefox and Chrome to no avail. Firefox gives me "a secure connection failed" error:



Chrome gives me something similar:



I tried Edge and Internet explorer in a Windows 10 VM as well, and both give similar errors.

I can ping both of them just fine, and both respond via USB (one via APCUPSd in Linux and one via NUT on FreeBSD) but the web interfaces on both of them seem to have dropped dead.

Notably I also can't seem to raise them via SSH (though I can't remember if I ever could) Apparently none of the ciphers are compatible between the units and my Linux machines, giving me the following error message:

Unable to negotiate with 10.0.1.5 port 22: no matching cipher found. Their offer: aes256-cbc,3des-cbc

It seems like a little much of a coincidence that both of them have gone down at the same time.

Could this be a TLS version deprecation issue? Or maybe some sort of crazy APC licensing scheme that I never caught on to that expired?

I'd appreciate any suggestions regarding what I can do to get back into the web interface on these things!

 

[oneplane]

It is indeed cipher deprecation and subsequent removal from most (of not all) clients. Sometimes it's just the browsers, sometimes the SSL/TLS libraries as well at which point anything that uses those libraries no longer works. The same applies to SSH as you've found.

For SSH you might have some luck with self-contained SSH clients (I think ZOC8 specifically supports ancient cryptography and other broken/skanky stuff that really shouldn't exist).

Most of the protocol/cipher removals are for good reasons, and as everything keeps moving forward, the things that tend to break are the ones that never got any upgrades. The embedded software in the AP9630 does still have somewhat recent versions available, and you might be able to reset it to factory settings and either upgrade over the console or TFTP. Their client libraries for TLS does support 1.2 which suggests that they may also be fixing their server-side crypto if that is using the same software. Keep in mind that a full reset on the card takes a 20-second push.

Edit: I just checked some release notes and they do state that ECDHE is supported and TLS 1.0 and SSL 3 and older are removed, so that actually sounds like a good thing.

 

[mattlach]

It is indeed cipher deprecation and subsequent removal from most (of not all) clients. Sometimes it's just the browsers, sometimes the SSL/TLS libraries as well at which point anything that uses those libraries no longer works. The same applies to SSH as you've found.

For SSH you might have some luck with self-contained SSH clients (I think ZOC8 specifically supports ancient cryptography and other broken/skanky stuff that really shouldn't exist).

Most of the protocol/cipher removals are for good reasons, and as everything keeps moving forward, the things that tend to break are the ones that never got any upgrades. The embedded software in the AP9630 does still have somewhat recent versions available, and you might be able to reset it to factory settings and either upgrade over the console or TFTP. Their client libraries for TLS does support 1.2 which suggests that they may also be fixing their server-side crypto if that is using the same software. Keep in mind that a full reset on the card takes a 20-second push.

Edit: I just checked some release notes and they do state that ECDHE is supported and TLS 1.0 and SSL 3 and older are removed, so that actually sounds like a good thing.

Thank you, I appreciate your input here.

Do you know if it is possible to reset the network cards without interrupting power to the attached servers?

Honestly, these APC's are on their own VLAN that does not reach the WAN, so I'd even be happy to run them without TLS if I can just get them operating.

Maybe I should try to get them to connect somehow via self contained or older revision SSH as you mention, and see if there is any way I can enable non-TLS web mode via SSH...

 

[klui]

There's no reason to use an older version of SSH. Run SSH using -c aes256-cbc. There are other options you can use to re-enable deprecated KEXes and ciphers. Whatever error message you get, just search on the web and you should hit solutions for others who ran into that before.

On the webUI front, download an older version of Firefox portable for use with older devices. I use FF 30 and 32 for my APC AP9619 and other ancient appliances. PortableApps.com - Browse /Mozilla Firefox, Portable Ed. at SourceForge.net

 

[oneplane]

There's no reason to use an older version of SSH. Run SSH using -c aes256-cbc. There are other options you can use to re-enable deprecated KEXes and ciphers. Whatever error message you get, just search on the web and you should hit solutions for others who ran into that before.

On the webUI front, download an older version of Firefox portable for use with older devices. I use FF 30 and 32 for my APC AP9619 and other ancient appliances. PortableApps.com - Browse /Mozilla Firefox, Portable Ed. at SourceForge.net

Except that ciphers and exchange methods are actually getting removed and there is no 'enabling' something that doesn't exist in the client. Especially for embedded SSH servers that is problematic as they often get stuck om something useless (rc4 for example) which hasn't been in any reputable SSH clients forever.

On the other hand, having legacy versions in a sandbox does make it a bit easier, same in the iKVM days where it only worked on 32-Bit JRE 1.6... but it's not a good thing to have around on a day-to-day operating system where it is easy to exploit.

 

[Rand__]

Thanks for this post, made me realise I am not the only one and it cant be a network issue.

Contacted APC (thankfully u dont need a support contract) and its a known issue:

NMC2 self-signed certificate expired -

also current firmware : UPS Network Management Card v7.0.8 Firmware for Smart-UPS & Galaxy 3500 with AP9630/31/35 - SFSUMX708 | APC Deutschland

Also the hint - http still works if u turn it on from ssh, just make sure to turn it off again

 

[klui]

Except that ciphers and exchange methods are actually getting removed and there is no 'enabling' something that doesn't exist in the client. Especially for embedded SSH servers that is problematic as they often get stuck om something useless (rc4 for example) which hasn't been in any reputable SSH clients forever.

Oh, I didn't know about them actually removing from the codebase. I don't have any old devices at home that only support it. You're right, -c arcfour and its various varieties don't work under OpenSSH 9.1p1. I suppose when these other ciphers get removed I'll just stick with telnet or use my terminal server.

My biggest roadblock on my APC management module was a certificate issue several years back. Resolved it using a combination of APC Security Wizard and older browser versions.

 

[oneplane]

Oh, I didn't know about them actually removing from the codebase. I don't have any old devices at home that only support it. You're right, -c arcfour and its various varieties don't work under OpenSSH 9.1p1. I suppose when these other ciphers get removed I'll just stick with telnet or use my terminal server.

My biggest roadblock on my APC management module was a certificate issue several years back. Resolved it using a combination of APC Security Wizard and older browser versions.

Yeah, it's getting even weirder sometimes where it gets removed in a minor release without warning, a bunch of enterprise customers get mad, it gets re-added in the next minor, and marked as deprecated, and then in the next major release gets removed again. For OpenSSH I've had similar weird things where some vendors/distributions undo the removal patch in their own supplied versions and then a version that works on one workstation doesn't work on another .

I also had a cert issue (but it wasn't an APC, I think it was some Eaton stuff), it was one where the signature was no longer supported, so even if it is valid it still won't work because TLS 1.2 doesn't support it. And then there is the fun stuff like switches that often tout some cool SSH cipher patch but forget the KEX need to be up-to-date as well (looking at you, HPE and Quanta).

I hope that at some point we're getting better separation between the 'application' and 'device' side of the embedded systems like SONiC or OpenBMC... probably not going to happen but one can dream

 

[mattlach]

Oh, I didn't know about them actually removing from the codebase. I don't have any old devices at home that only support it. You're right, -c arcfour and its various varieties don't work under OpenSSH 9.1p1. I suppose when these other ciphers get removed I'll just stick with telnet or use my terminal server.

My biggest roadblock on my APC management module was a certificate issue several years back. Resolved it using a combination of APC Security Wizard and older browser versions.

This was not the case for me.

I was able to add entries for my two APC units in /etc/ssh/ssh_config as follows:

host <ip>
    Ciphers +aes256-cbc

Then I could SSH in and enable http mode using the command "web -h enable" followed by a reboot using the command "reboot" (which does not restart the UPS itself, only the network management card, so everything stayed online)

Now I am confused though.

It looks like TLS 1.2 is supported by the units:



...but my browsers won't connect.

Did all of the browsers suddenly omit TLS 1.2?

I don't remember reading about that anywhere...

 

[mattlach]

Thanks for this post, made me realise I am not the only one and it cant be a network issue.

Contacted APC (thankfully u dont need a support contract) and its a known issue:

NMC2 self-signed certificate expired -

also current firmware : UPS Network Management Card v7.0.8 Firmware for Smart-UPS & Galaxy 3500 with AP9630/31/35 - SFSUMX708 | APC Deutschland

Also the hint - http still works if u turn it on from ssh, just make sure to turn it off again

Oh, I see.

Thank you for that link.

Yeah, I'm still on 6.8.0, so I guess I'll have to upgrade the firmware.

I'm confused about how the firmware works on these APC units. Is it unified across the entire UPS, or does the UPS have a separate firmware, and the network management card a different firmware.

My about screen shows me this:



In the configuration menu there is a firmware update option, but that appears to want to want me to turn the UPS OFF (which is not good for my servers right now) so I supect it is for the UPS firmware.

Can anyone point me in the direction of how to update the firmware of the network management card?

I feel like I did this a couple of years ago, and I remember it being different from the firmware on the UPS, but it is all hazy. It's been too long and I can't remember a thing.

 

[mattlach]

Oh, I see.

Thank you for that link.

Yeah, I'm still on 6.8.0, so I guess I'll have to upgrade the firmware.

I'm confused about how the firmware works on these APC units. Is it unified across the entire UPS, or does the UPS have a separate firmware, and the network management card a different firmware.

My about screen shows me this:

View attachment 25851

In the configuration menu there is a firmware update option, but that appears to want to want me to turn the UPS OFF (which is not good for my servers right now) so I suspect it is for the UPS firmware.

Can anyone point me in the direction of how to update the firmware of the network management card?

I feel like I did this a couple of years ago, and I remember it being different from the firmware on the UPS, but it is all hazy. It's been too long and I can't remember a thing.

Never mind, I found these instructions:

• Go to APC, a flagship brand of Schneider Electric and search for your network management card model number.
• Select the “UPS Network Management Card 2” link.
• Scroll down to the product documentation tabs.
• Click the Software and Firmware tab.
• Find the firmware file. This one is “Network Management Card v6.5.0 Firmware for Symmetra 3-Phase with AP9630/AP9631/AP9635”
• Verify that you have located firmware appropriate for the host UPS family.
• Click the “Download” Link.
• Then Select the “Download Now” button.
• The file will download to your default download directory.
• Click the file and extract the firmware package.
• Open the extracted firmware package.
• Locate and click the Firmware Upgrade Utility application.
• Enter the IP address
• Enter your Username and Password. The default values are both lowercase “apc”.
• Click the Ping button to assure you have connectivity with the card to be upgraded.
• Click “Upgrade Now” button.
• If successful, the utility will report “Device(s) Successfully Updated”.
• You can now log in to your UPS Network Management Card 2.

They are older than the current firmware, but the process is probably the same.

Finding the firmware the way they suggestied in the download section on their website was bloody impossible, but I was able to use google with a site:apc.com filter to find what I think is the latest.

"7.0.8 for Smart-UPS and Single Phase Symmetra" (link if it helps anyone else)

It looks like the download includes an executable that will send the firmware to the UPS via the network, but it is an .exe file, sigh...

So now I have to find a windows machine somewhere. Everything I have is Linux or BSD...

 

[mattlach]

Hmm.

I tried to repeat everything on the second UPS, which is an identical configuration. It too gave me the identical cipher error message as a few posts up.

The workaround (adding the cipher in the SSH config file) works for the first one, but not for the second one. It gives me a "handshake failed" error, which is weird.

If it worked on the first, I would have expected it to work on the second...

Any suggestions here?

I commented out my edits to /etc/ssh/ssh_config and instead tried connecting using the -c option.

"ssh -c aes256-cbc root@<first_ip>" works just fine.

"ssh -c aes256-cbc root@<second_ip>" fails with a "Handshake Failed" error.

Then I figured I'd be clever and try 3des-cbc. Again, worked on the first IP (surprisingly, good old obsolete triple DES is still there on my linux box), but failed with a handshake error on the second...

Starting to think something else is wrong on the second card.

Maybe if I flash the firmware first, it will work.

I'm going to try that, but not tonight. I'm getting tired. I have a policy of never flashing firmware while tired. It never ends well

 

[klui]

I'd look at both units to see if the base configuration except IP is the same. Most likely you've enabled some option on the second and not the first, or vice versa. Add -vvvv on the client to see if you can see any meaningful messages. From that message many possibilities are documented on the web.

 

[mattlach]

I'd look at both units to see if the base configuration except IP is the same. Most likely you've enabled some option on the second and not the first, or vice versa. Add -vvvv on the client to see if you can see any meaningful messages. From that message many possibilities are documented on the web.

Tough to do when I can't access any way to configure it

They both used to work just fine, so if the configuration on the second unit is different, then somehting weird has happened to that one unit.


So, I got distracted from this and only just got back to it today.

Flashing the firmware on the first unit worked like a charm. Installing the firmware update solved all issues. The unit responds and allows me to log on via the web interface again.

The second unit, not so much.

The remote firmware update software complains that the device is either corrupt or incompatible (its the exact same model as the first one)

I tried taking the other unit offline first just in case they were somehow interfering with each other, but that didn't make any difference.

There is a reset button on the back of the network management card, I could try, but my best guess is that it just resets the network configuration to its default. Since I can ping the thing, that is not my current issue.

I was thinking maybe I should try to reboot the affected UPS. I don't think that would do anything too harmful, as most of the important stuff in the rack has redundant PSU's, with one cored plugged into each UPS. A secondary switch may go offline, but that's not the end of the world.

It struck me - however - that these things usually don't go fully offline. There is a power button on the front, but I forget if just pushing it is sufficient, or if I need to do something like unplug the unit from the wall, AND pull the battery cable disconnect or something like that in order to make sure it gets a complete cold boot.

If that doesn't work, I may just have to replace the network management card. APC doesn't seem to offer any other way to update the firmware than through the remote utility, and if it can't work because things are corrupted, then the NMC may be a goner. Luckily now that they are old, they are only ~$25 buy it now on eBay

Appreciate any thoughts!

 

[oneplane]

Can you detach the card? That will do a true reboot, no matter what APC designed/thought of ;-)

Regarding firmware issues: depending on the hardware revision it might have a boot loader upload feature, a recovery mode of sorts. While not generally explained or documented well in the user manual, it usually follows the same procedure as the 30/30/30 reset straight out of the consumer WRT54G era.

Going deeper, you could take the hardware hacking route and try to find a serial console that shows uboot/redboot on the SoC that is on the APC card, sometimes it's on the external serial port, sometimes on an internal one. It might show both information as to what is actually wrong, as well as give an option to start a TFTP load over the network or serial XMODEM/ZMODEM download. That would then directly unpack into RAM and from there you can do a normal firmware upgrade to write into NVRAM/Flash.

 

[Rand__]

There is a reset button on the back of the network management card, I could try, but my best guess is that it just resets the network configuration to its default. Since I can ping the thing, that is not my current issue.

If you press that button for like 20s you do a reset to default config, which might or might not help in ur case. Not sure if it enables telnet to provide another access path...

 

[Terry Kennedy]

The remote firmware update software complains that the device is either corrupt or incompatible (its the exact same model as the first one)

Can you FTP to the card? You can actually just "put" the AOS and application files (in that order, and the card reboots by itself after each):

(0:30) host:/sysprog/terry# ftp ups1.example.com
Connected to ups1.example.com.
220 AP9631 Network Management Card AOS v7.0.8 FTP server ready.
Name (ups1.example.com:terry): apc
331 User name okay, need password.
Password:
230 User logged in, proceed.
ftp> dir
229 Entering Extended Passive Mode (|||51241|).
150 File status okay; about to open data connection.
--wx-wx-wx  1 admin      admin     3127808 May 27  2022  apc_hw05_aos_708.bin
--wx-wx-wx  1 admin      admin     2224164 Jun 29 18:21  apc_hw05_sy_708.bin
-rw-rw-rw-  1 admin      admin       45000 Dec 12 22:36  config.ini
drwxrwxrwx  1 admin      admin           0 Jan 19  2022  db/
drwxrwxrwx  1 admin      admin           0 Jan 19  2022  ssl/
drwxrwxrwx  1 admin      admin           0 Jan 19  2022  ssh/
drwxrwxrwx  1 admin      admin           0 Jan 19  2022  logs/
drwxrwxrwx  1 admin      admin           0 Jan 19  2022  sec/
drwxrwxrwx  1 admin      admin           0 Jan 19  2022  dbg/
drwxrwxrwx  1 admin      admin           0 Jan 19  2022  fwl/
drwxrwxrwx  1 admin      admin           0 Jan 19  2022  email/
drwxrwxrwx  1 admin      admin           0 Jan 19  2022  eapol/
drwxrwxrwx  1 admin      admin           0 Jan 19  2022  lang/
226 Closing data connection.
ftp> quit
221 Goodbye.

I would re-do the flash with the utility once you get the card updated, in case the boot loader is upgraded.

You can also get the config.ini file (it is ASCII text) to compare between two devices. If you try to put a config.ini from another unit, it will reject it unless you change the "Override=" MAC address to that of the target unit.

AP9631 cards can download firmware from USB drives via the onboard USB ports. AP9630 cards will need to be downloaded via serial port (which, of course, is a different cable than older APC units) if it isn't responding to the network..

There is a reset button on the back of the network management card, I could try, but my best guess is that it just resets the network configuration to its default. Since I can ping the thing, that is not my current issue.

There are a number of resets - password, IP config and full config but as you said, that isn't your problem.

It struck me - however - that these things usually don't go fully offline. There is a power button on the front, but I forget if just pushing it is sufficient, or if I need to do something like unplug the unit from the wall, AND pull the battery cable disconnect or something like that in order to make sure it gets a complete cold boot.

On a classic (non-LCD) APC UPS you "dead" it by turning it off, unplugging the input power and then pressing and holding the power off button until the UPS chirps. There is still a flea power draw, so if you're not going to be using it for a while it is best to disconnect the batteries. There's always a connector inside the battery compartment. Some non-rackmount models had a triangular cap on the back which also serves as a battery disconnect - APC did that while they were negotiating with the IATA about shipping "dead" units with batteries connected.

If that doesn't work, I may just have to replace the network management card. APC doesn't seem to offer any other way to update the firmware than through the remote utility, and if it can't work because things are corrupted, then the NMC may be a goner. Luckily now that they are old, they are only ~$25 buy it now on eBay

See above for a couple of alternate methods. If you're in the US I'll flash it for you if you ship it to me and cover return postage (it should fit in a USPS "small flat rate box").

 

[mattlach]

If you press that button for like 20s you do a reset to default config, which might or might not help in ur case. Not sure if it enables telnet to provide another access path...

Can you FTP to the card? You can actually just "put" the AOS and application files (in that order, and the card reboots by itself after each):

That there is exactly the issue. I disabled everything that was not HTTPS and SSH for the sake of security, figuring I could alsways just sftp in over SSH if I had to.

(0:30) host:/sysprog/terry# ftp ups1.example.com
Connected to ups1.example.com.
220 AP9631 Network Management Card AOS v7.0.8 FTP server ready.
Name (ups1.example.com:terry): apc
331 User name okay, need password.
Password:
230 User logged in, proceed.
ftp> dir
229 Entering Extended Passive Mode (|||51241|).
150 File status okay; about to open data connection.
--wx-wx-wx  1 admin      admin     3127808 May 27  2022  apc_hw05_aos_708.bin
--wx-wx-wx  1 admin      admin     2224164 Jun 29 18:21  apc_hw05_sy_708.bin
-rw-rw-rw-  1 admin      admin       45000 Dec 12 22:36  config.ini
drwxrwxrwx  1 admin      admin           0 Jan 19  2022  db/
drwxrwxrwx  1 admin      admin           0 Jan 19  2022  ssl/
drwxrwxrwx  1 admin      admin           0 Jan 19  2022  ssh/
drwxrwxrwx  1 admin      admin           0 Jan 19  2022  logs/
drwxrwxrwx  1 admin      admin           0 Jan 19  2022  sec/
drwxrwxrwx  1 admin      admin           0 Jan 19  2022  dbg/
drwxrwxrwx  1 admin      admin           0 Jan 19  2022  fwl/
drwxrwxrwx  1 admin      admin           0 Jan 19  2022  email/
drwxrwxrwx  1 admin      admin           0 Jan 19  2022  eapol/
drwxrwxrwx  1 admin      admin           0 Jan 19  2022  lang/
226 Closing data connection.
ftp> quit
221 Goodbye.

I would re-do the flash with the utility once you get the card updated, in case the boot loader is upgraded.

You can also get the config.ini file (it is ASCII text) to compare between two devices. If you try to put a config.ini from another unit, it will reject it unless you change the "Override=" MAC address to that of the target unit.

AP9631 cards can download firmware from USB drives via the onboard USB ports. AP9630 cards will need to be downloaded via serial port (which, of course, is a different cable than older APC units) if it isn't responding to the network..


There are a number of resets - password, IP config and full config but as you said, that isn't your problem.


On a classic (non-LCD) APC UPS you "dead" it by turning it off, unplugging the input power and then pressing and holding the power off button until the UPS chirps. There is still a flea power draw, so if you're not going to be using it for a while it is best to disconnect the batteries. There's always a connector inside the battery compartment. Some non-rackmount models had a triangular cap on the back which also serves as a battery disconnect - APC did that while they were negotiating with the IATA about shipping "dead" units with batteries connected.


See above for a couple of alternate methods. If you're in the US I'll flash it for you if you ship it to me and cover return postage (it should fit in a USPS "small flat rate box").

I might try resetting to default settings, and using a non-encrypted method of connecting and updating the firmware as suggested above.

Does anyone know if holding that reset button interrupts power from the unit? Givne my understanding that these things are hot-pluggable and mostly separate from the operation of the UPS itself I am assuming "no", but I figured I'd make sure before I try.

 

[LodeRunner]

IIRC, the reset button on the NMC is for the NMC, not that chassis.

 

[hmw]

SCP on newer machines won't work with the NMC2 cards - you will have to use the older compatibility mode and then it will work
e.g. (assuming your NMC is at 192.168.0.5)

scp -O -o HostKeyAlgorithms=+ssh-rsa apc_hw05_sumx_708.bin apc@192.168.0.5:apc_hw05_sumx_708.bin