What are you using to connect your servers to your US-16-XG switches? Are you running fiber exclusively? Which modules?The switch will retain the controller settings between reboots if the controller is unavailable. And you can still SSH to it using the Site credentials that you configured in the controller (and then you have to 'telnet localhost' to get into the EdgeOS CLI). It won't persist CLI changes between reboots and the L3 stuff isn't there so there's not much point in getting at the CLI anyways.
If you don't like the controller concept, by all means, UniFi ain't for you and that's why they have the two product lines for switching and routing. I never touched the UniFi stuff before coming across this deal but I've gone all-in since -- USG Pro, APs, and I'll snag a couple of US-8s as soon as they hit retail. Single-pane-of-glass management for my whole network and not having to muck with switch CLIs was too great to resist.
As to reliability... I've had the XGs at the core of my home network since I got them and they haven't let me down yet. I work remote and pretty much my whole world revolves around the network and my servers staying up. So far, so good.