Looking for feedback and ideas....
BACKGROUND: want to setup vSphere 7.0 and planned to go with a 2 host direct connect cluster. After realizing that I would still need a witness, I decided OK why not just go with a 3 host cluster because a 3rd host is more useful than a witness? And the more I looked at a 3 node cluster, the more a 4 node cluster seemed attractive for extra features like erasure coding and having some slack space for rebuilds.
PROBLEM: Build is balooning and I am drawing the line if I can't do direct connect. Many people say that you shoudn't really **direct connect** more than 2 hosts. I do NOT want to get a separate switch (I want to use 10gbe/40gbe for vSAN, vMotion etc and those speeds are only available to me if I direct connect). I'm building up a Dell C6300/C6320 box with up to 4 nodes and want to keep the whole build inside the chassis.
HYPOTHESIZED SOLUTIONS:
(1) FORGE AHEAD and try to chain 3 or 4 hosts togehter and experience the pain everyone predicts
(2) SCALE BACK to 2 nodes + witness on third (seems like a waste of a node but provides for high speed direct connect)
(3) "STRETCHED" FOUR HOST CLUSTER: Host A<=>B @40gbe, C<=>D @40gbe with slower link between as if stretched between sites (back to needing witness?)
(4) 2X TWO NODE CLUSTERS: similar to #3 but as two distinct two-node clusters (with a witness for each on the opposite cluster)
(5) PFSENSE: somewhere within the cluster or dedicate one of the four server nodes to it (single point of failure?)
(6) PROXMOX: try to do a MESH setup (maybe ditch vShpere): Full Mesh Network for Ceph Server - Proxmox VE
Some options for physcial connections:
(A) QSFP DAISY CHAIN: 3 or 4 nodes + try to avoid a loop with nic teaming and failover? set "routing based on originating virtual port" (?)
(B) QSFP TO 4x SFP+ breakout cables on 2 nodes.... so all 4 nodes meet up on a SINGLE physcial port (mirror on another node for failover)
(C) 1GBE SWITCH MODE: run PFsense bare metal or in another virtualized environment, and maybe have failover instance within the cluster
I'm also hoping to route WAN and private network through all nodes WITHOUT physical uplinks to every node. I'd like to route that traffic througoht vSphere using whatever interconnections I'm setting up for other traffic. Otherwise the back of this server is going to be ridiculous with 4x iDRAC cables, 4x WAN cables, 4x private network cables, and then all the cross/direct connect 40gbe/10gbe cables.
Here is what solution #3 and #4 might look like roughly....
And below is an option for using QSFP to SFP fanout cables. Could run WAN and LAN over one and then vMotion, vSAN over the other. I think the only way this may work well is if there is some huge benefit to concentrating traffic on the fanout ports.... wondering if the NIC could insulate the CPU? I don't know enough about resource costs for packet switchiing.
BACKGROUND: want to setup vSphere 7.0 and planned to go with a 2 host direct connect cluster. After realizing that I would still need a witness, I decided OK why not just go with a 3 host cluster because a 3rd host is more useful than a witness? And the more I looked at a 3 node cluster, the more a 4 node cluster seemed attractive for extra features like erasure coding and having some slack space for rebuilds.
PROBLEM: Build is balooning and I am drawing the line if I can't do direct connect. Many people say that you shoudn't really **direct connect** more than 2 hosts. I do NOT want to get a separate switch (I want to use 10gbe/40gbe for vSAN, vMotion etc and those speeds are only available to me if I direct connect). I'm building up a Dell C6300/C6320 box with up to 4 nodes and want to keep the whole build inside the chassis.
HYPOTHESIZED SOLUTIONS:
(1) FORGE AHEAD and try to chain 3 or 4 hosts togehter and experience the pain everyone predicts
(2) SCALE BACK to 2 nodes + witness on third (seems like a waste of a node but provides for high speed direct connect)
(3) "STRETCHED" FOUR HOST CLUSTER: Host A<=>B @40gbe, C<=>D @40gbe with slower link between as if stretched between sites (back to needing witness?)
(4) 2X TWO NODE CLUSTERS: similar to #3 but as two distinct two-node clusters (with a witness for each on the opposite cluster)
(5) PFSENSE: somewhere within the cluster or dedicate one of the four server nodes to it (single point of failure?)
(6) PROXMOX: try to do a MESH setup (maybe ditch vShpere): Full Mesh Network for Ceph Server - Proxmox VE
Some options for physcial connections:
(A) QSFP DAISY CHAIN: 3 or 4 nodes + try to avoid a loop with nic teaming and failover? set "routing based on originating virtual port" (?)
(B) QSFP TO 4x SFP+ breakout cables on 2 nodes.... so all 4 nodes meet up on a SINGLE physcial port (mirror on another node for failover)
(C) 1GBE SWITCH MODE: run PFsense bare metal or in another virtualized environment, and maybe have failover instance within the cluster
I'm also hoping to route WAN and private network through all nodes WITHOUT physical uplinks to every node. I'd like to route that traffic througoht vSphere using whatever interconnections I'm setting up for other traffic. Otherwise the back of this server is going to be ridiculous with 4x iDRAC cables, 4x WAN cables, 4x private network cables, and then all the cross/direct connect 40gbe/10gbe cables.
Here is what solution #3 and #4 might look like roughly....
And below is an option for using QSFP to SFP fanout cables. Could run WAN and LAN over one and then vMotion, vSAN over the other. I think the only way this may work well is if there is some huge benefit to concentrating traffic on the fanout ports.... wondering if the NIC could insulate the CPU? I don't know enough about resource costs for packet switchiing.
Last edited: