I don't use ceph......
I have a 3 node cluster. I upgraded an E3 Xeon and that was all good. At that point 1 node v5 and 2 v4.4. Cluster working fine.
Then upgraded a dual E5 Xeon, this one died. Won't boot. Not overly serious, it doesn't like the new kernel for some reason. It will boot with an older kernel and is running proxmox 5.
So at this point I have 1 node still at 4.4 (I won't upgrade until I sort out the E5 box) , 1 node running v5 but with an older kernel and one running v5 with the correct kernel.
Cluster seems to be fine.
I wouldn't want a production cluster to be running the mix of versions mine is but as it's a lab it seems fine. Pretty impressive that it's still chugging away really.
Of course your mileage may vary