I am currently working on designing our network. I will show you my plans and hopefully you have some feedback/advice 
Goals
- Redundant. One switch should fail without the network going down
- Cheap
- Fast
- Expandable
I plan on starting with 2 Quanta LB4M switches. This sketch takes up to 5 which gives me 228 ports which should be able to substain 8-11 C6100 machines (in other words way more than we will have).
Conceptual sketch over network topology,
Each C6100 will connect to the switches like this,
Design motivations
I am planing to use the Quanta LB4M for switches. Sadly it only has 2x SPF+ so in order to be able to have 10Gbe to the SAN i have to do LACP between switch E and D.
If i can find any 4 (or more) SFP+ port switch i can swap E and D to those. Then i will be able to do 10Gbe between E and D. And i would be able to do 2x10Gbe from SAN to both E and D!
This design gives me a redundant 10Gbe backbone where any one switch can fail and we still have connectivity to all parts of the network (except BMC/IPMI, but we can live with that).
All switches will run RSTP/MSTP. This enables me to build this cyclic topology.
The 10Gbe will be connected with twinax cables which have a lower latency than 10GBASE.T,
"Moreover, this setup offers 15 to 25 times lower transceiver latency than current 10GBASE-T".
So if switch E fails traffic from B will go via A, C, D. The lower latency of twinax should help mitigate the additionally required route while we change/repair switch E.
The SAN uses the Brocade 1020 CNA cards. Tested and working with the Quanta LB4M switch. OmniOS will run IPMP failover on two NICs to give me redundancy. Each NIC connected to different switches (E and D). If i can get two 4 port SFP+ switches i will do 20Gbe to switch E and 20Gbe to D using IPMP (think it should be possible).
The PFsense boxes use CARP if a firewall goes down. Each pfsense machine is connected to two different switches with nic bonding in a failover configuration.
On each C6100 node we will run xenserver. We will populate the free PCI-E slot with either a 2 port or 4 port intel nic. This gives 5 or 7 ports on each C6100 node. I will use xenserver nic bonding in a failover setup for 5 port C6100 nodes. For 7 port C6100 nodes i think it should be possible to run 2x 1Gbit ports in a LACP setup to a switch. I.e. using 4 ports for storage traffic to the SAN. 2x1Gbit LACP to switch A, 2x1Gbit LACP to switch B. I do not yet know if it is possible to do both LACP and use the LACP ports in a failover configuration with xenserver. Since LACP does NOT work between multiple switches i am unsure if this will work.
Conclusion
This design should fulfill my requirements,
- Redundant: Yes we can lose any 1 switch without losing network connectivity.
- Cheap: Yes, the quanta switches, twinax cables, brocade and intel NICs are cheap on ebay.
- Fast; Yes, the 10Gbe backbone gives us 10Gbe to each switch. Each C6100 will have 1Gbit/s for VM traffic and 1 or 2 Gbit/s for storage/management traffic.
- Expandable: Kinda, at some point we cannot expand the switch chain and still have good performance if a switch dies. For our requirements this 5 switch design will be enough the next couple or years (/ decade?).
Any cons/pros or other comments on this design?
Goals
- Redundant. One switch should fail without the network going down
- Cheap
- Fast
- Expandable
I plan on starting with 2 Quanta LB4M switches. This sketch takes up to 5 which gives me 228 ports which should be able to substain 8-11 C6100 machines (in other words way more than we will have).
Conceptual sketch over network topology,
Each C6100 will connect to the switches like this,
Design motivations
I am planing to use the Quanta LB4M for switches. Sadly it only has 2x SPF+ so in order to be able to have 10Gbe to the SAN i have to do LACP between switch E and D.
If i can find any 4 (or more) SFP+ port switch i can swap E and D to those. Then i will be able to do 10Gbe between E and D. And i would be able to do 2x10Gbe from SAN to both E and D
!This design gives me a redundant 10Gbe backbone where any one switch can fail and we still have connectivity to all parts of the network (except BMC/IPMI, but we can live with that).
All switches will run RSTP/MSTP. This enables me to build this cyclic topology.
The 10Gbe will be connected with twinax cables which have a lower latency than 10GBASE.T,
"Moreover, this setup offers 15 to 25 times lower transceiver latency than current 10GBASE-T".
So if switch E fails traffic from B will go via A, C, D. The lower latency of twinax should help mitigate the additionally required route while we change/repair switch E.
The SAN uses the Brocade 1020 CNA cards. Tested and working with the Quanta LB4M switch. OmniOS will run IPMP failover on two NICs to give me redundancy. Each NIC connected to different switches (E and D). If i can get two 4 port SFP+ switches i will do 20Gbe to switch E and 20Gbe to D using IPMP (think it should be possible).
The PFsense boxes use CARP if a firewall goes down. Each pfsense machine is connected to two different switches with nic bonding in a failover configuration.
On each C6100 node we will run xenserver. We will populate the free PCI-E slot with either a 2 port or 4 port intel nic. This gives 5 or 7 ports on each C6100 node. I will use xenserver nic bonding in a failover setup for 5 port C6100 nodes. For 7 port C6100 nodes i think it should be possible to run 2x 1Gbit ports in a LACP setup to a switch. I.e. using 4 ports for storage traffic to the SAN. 2x1Gbit LACP to switch A, 2x1Gbit LACP to switch B. I do not yet know if it is possible to do both LACP and use the LACP ports in a failover configuration with xenserver. Since LACP does NOT work between multiple switches i am unsure if this will work.
Conclusion
This design should fulfill my requirements,
- Redundant: Yes we can lose any 1 switch without losing network connectivity.
- Cheap: Yes, the quanta switches, twinax cables, brocade and intel NICs are cheap on ebay.
- Fast; Yes, the 10Gbe backbone gives us 10Gbe to each switch. Each C6100 will have 1Gbit/s for VM traffic and 1 or 2 Gbit/s for storage/management traffic.
- Expandable: Kinda, at some point we cannot expand the switch chain and still have good performance if a switch dies. For our requirements this 5 switch design will be enough the next couple or years (/ decade?).
Any cons/pros or other comments on this design
?