Hi there all,
I've just finished setting up a nice AIO proxmox server and between it and my workstation (direct connection), it saturates 10Gb or gives a faster then local SSD speed depending on work load (NFS maxes out at 4GB/s but thats a different thread)......and now I want more.
Well not really more as in only speed but also connecting more to my server. At the moment I'm kind of space constrained to a 12ru rack that already has a:
- 4ru server,
- 2ru UPS,
- 1ru SRX320 router with VDSL module (sound tamed and powering 2 AP's 1 VIOP phone)
I'm in the middle of building a new home that has some nice cool space away from the living spaces for a server rack and cabling.
What I'm aiming for in the new setup is for my AIO to be just a ZFS filer connected to:
- 3 (or 4) server vm/container cluster (Proxmox, VMware, home rolled), maybe FDR IB. Each VM server will require:
- 1 x 56Gb IB or 40GBe iscsi to storage
- 1 x 40GBe, data serving,
- 1 x 1Gbe, IPMI,
- 1 x 1Gbe, management,
- 1 x 1Gbe, heartbeat.
- The ZFS filer will require:
- 2 x 56GB-IB or 40GBe
- 1 x 1Gbe, IPMI
- 1 x 1Gbe, management.
- 2 Ubituiti APs using POE (they came with injectors, nice to not have to use them. Power plug management)
- 4 POE surveillance cameras
- 1 POE VIOP phone
- 1 video editing station. Looking at on server editing, the faster the better, 40/10 Gbe?
- 2 workstations. Looking at serving home dirs from the server via NFS and the occasional shunting of large files (2-4 GB). Can live with 10Gbe.
- A fist full (10-ish) of entertainment and smart house bridge devices etc. 1Gb is ample, if not over kill, for those.
- 2 connections going to seperate switches in the office for lab equipment. (lab as in computer lab).
So as far as I can see for now, physically, I'm looking at a solution that provides around:
- 4 x 40Gb or faster ports, either infiniband or ethernet, would like to go FDR 56Gb, (QSFP+)
- 4 x 40Gbe, (QSFP+?)
- 3-4 x 10Gbe, (3x SFP+, 1 x rj-45)
- 15-ish x 1Gbe (rj-45)
- POE would be great as the SRX320 runs a bit hot in my environment with that many POE devices hanging of it and only does 802.3 AF
Absolute overkill for this type of network I know. But I like to play with things, tune and tweak and who doesn't enjoy the resulting performance.
Bare in mind I haven't really sorted out the data serving and cluster solution ( may even go hyper converged and use the filer as a backup and cold storage server).
I would like to achieve this a max of 2 switches (3ru total used for switches, patch panel and cable management) and as little power draw that can be achieved considering the speeds. If, possible, for less then 1000 Australian Brass Razzoo's
So far I've looked at Mellanox and Brocade gear, in particular
- Brocade ICX6610, they are cheap on eBay, but I may have to drop that as the cheap ones need a license activate the 10Gb ports. I choked only coffee when I saw how much they are.
- Mellanox SX6012. Again I might have to revisit this as I was going to use breakout cables and the VPI capabilities and the cheap ones on eBay are just dumb infiniband switches, different to the SX6012 units that I read the manual for and based my planing on.
So there you have my brain dump. I'll up date this as go on with my research, but I've got some wiring decision to finalise soon so I may add some questions in regards to that here as well, like:
Is there such a thing as a QSFP to 10Gbe rj-45 breakout cable?
Is there such a thing as a QSFP room to room solution like there is for rj-45 i.e. the wall face plate?
I've just finished setting up a nice AIO proxmox server and between it and my workstation (direct connection), it saturates 10Gb or gives a faster then local SSD speed depending on work load (NFS maxes out at 4GB/s but thats a different thread)......and now I want more.
Well not really more as in only speed but also connecting more to my server. At the moment I'm kind of space constrained to a 12ru rack that already has a:
- 4ru server,
- 2ru UPS,
- 1ru SRX320 router with VDSL module (sound tamed and powering 2 AP's 1 VIOP phone)
I'm in the middle of building a new home that has some nice cool space away from the living spaces for a server rack and cabling.
What I'm aiming for in the new setup is for my AIO to be just a ZFS filer connected to:
- 3 (or 4) server vm/container cluster (Proxmox, VMware, home rolled), maybe FDR IB. Each VM server will require:
- 1 x 56Gb IB or 40GBe iscsi to storage
- 1 x 40GBe, data serving,
- 1 x 1Gbe, IPMI,
- 1 x 1Gbe, management,
- 1 x 1Gbe, heartbeat.
- The ZFS filer will require:
- 2 x 56GB-IB or 40GBe
- 1 x 1Gbe, IPMI
- 1 x 1Gbe, management.
- 2 Ubituiti APs using POE (they came with injectors, nice to not have to use them. Power plug management)
- 4 POE surveillance cameras
- 1 POE VIOP phone
- 1 video editing station. Looking at on server editing, the faster the better, 40/10 Gbe?
- 2 workstations. Looking at serving home dirs from the server via NFS and the occasional shunting of large files (2-4 GB). Can live with 10Gbe.
- A fist full (10-ish) of entertainment and smart house bridge devices etc. 1Gb is ample, if not over kill, for those.
- 2 connections going to seperate switches in the office for lab equipment. (lab as in computer lab).
So as far as I can see for now, physically, I'm looking at a solution that provides around:
- 4 x 40Gb or faster ports, either infiniband or ethernet, would like to go FDR 56Gb, (QSFP+)
- 4 x 40Gbe, (QSFP+?)
- 3-4 x 10Gbe, (3x SFP+, 1 x rj-45)
- 15-ish x 1Gbe (rj-45)
- POE would be great as the SRX320 runs a bit hot in my environment with that many POE devices hanging of it and only does 802.3 AF
Absolute overkill for this type of network I know. But I like to play with things, tune and tweak and who doesn't enjoy the resulting performance.
Bare in mind I haven't really sorted out the data serving and cluster solution ( may even go hyper converged and use the filer as a backup and cold storage server).
I would like to achieve this a max of 2 switches (3ru total used for switches, patch panel and cable management) and as little power draw that can be achieved considering the speeds. If, possible, for less then 1000 Australian Brass Razzoo's
So far I've looked at Mellanox and Brocade gear, in particular
- Brocade ICX6610, they are cheap on eBay, but I may have to drop that as the cheap ones need a license activate the 10Gb ports. I choked only coffee when I saw how much they are.
- Mellanox SX6012. Again I might have to revisit this as I was going to use breakout cables and the VPI capabilities and the cheap ones on eBay are just dumb infiniband switches, different to the SX6012 units that I read the manual for and based my planing on.
So there you have my brain dump. I'll up date this as go on with my research, but I've got some wiring decision to finalise soon so I may add some questions in regards to that here as well, like:
Is there such a thing as a QSFP to 10Gbe rj-45 breakout cable?
Is there such a thing as a QSFP room to room solution like there is for rj-45 i.e. the wall face plate?