Faster then 10Gb home, home office and lab recommendations (IB, VPI etc?)

MikeWebb

Member
Jan 28, 2018
88
23
8
50
Hi there all,

I've just finished setting up a nice AIO proxmox server and between it and my workstation (direct connection), it saturates 10Gb or gives a faster then local SSD speed depending on work load (NFS maxes out at 4GB/s but thats a different thread)......and now I want more.

Well not really more as in only speed but also connecting more to my server. At the moment I'm kind of space constrained to a 12ru rack that already has a:
- 4ru server,
- 2ru UPS,
- 1ru SRX320 router with VDSL module (sound tamed and powering 2 AP's 1 VIOP phone)

I'm in the middle of building a new home that has some nice cool space away from the living spaces for a server rack and cabling.

What I'm aiming for in the new setup is for my AIO to be just a ZFS filer connected to:
- 3 (or 4) server vm/container cluster (Proxmox, VMware, home rolled), maybe FDR IB. Each VM server will require:
- 1 x 56Gb IB or 40GBe iscsi to storage
- 1 x 40GBe, data serving,
- 1 x 1Gbe, IPMI,
- 1 x 1Gbe, management,
- 1 x 1Gbe, heartbeat.
- The ZFS filer will require:
- 2 x 56GB-IB or 40GBe
- 1 x 1Gbe, IPMI
- 1 x 1Gbe, management.
- 2 Ubituiti APs using POE (they came with injectors, nice to not have to use them. Power plug management)
- 4 POE surveillance cameras
- 1 POE VIOP phone
- 1 video editing station. Looking at on server editing, the faster the better, 40/10 Gbe?
- 2 workstations. Looking at serving home dirs from the server via NFS and the occasional shunting of large files (2-4 GB). Can live with 10Gbe.
- A fist full (10-ish) of entertainment and smart house bridge devices etc. 1Gb is ample, if not over kill, for those.
- 2 connections going to seperate switches in the office for lab equipment. (lab as in computer lab).

So as far as I can see for now, physically, I'm looking at a solution that provides around:
- 4 x 40Gb or faster ports, either infiniband or ethernet, would like to go FDR 56Gb, (QSFP+)
- 4 x 40Gbe, (QSFP+?)
- 3-4 x 10Gbe, (3x SFP+, 1 x rj-45)
- 15-ish x 1Gbe (rj-45)
- POE would be great as the SRX320 runs a bit hot in my environment with that many POE devices hanging of it and only does 802.3 AF

Absolute overkill for this type of network I know. But I like to play with things, tune and tweak and who doesn't enjoy the resulting performance.

Bare in mind I haven't really sorted out the data serving and cluster solution ( may even go hyper converged and use the filer as a backup and cold storage server).

I would like to achieve this a max of 2 switches (3ru total used for switches, patch panel and cable management) and as little power draw that can be achieved considering the speeds. If, possible, for less then 1000 Australian Brass Razzoo's

So far I've looked at Mellanox and Brocade gear, in particular
- Brocade ICX6610, they are cheap on eBay, but I may have to drop that as the cheap ones need a license activate the 10Gb ports. I choked only coffee when I saw how much they are.
- Mellanox SX6012. Again I might have to revisit this as I was going to use breakout cables and the VPI capabilities and the cheap ones on eBay are just dumb infiniband switches, different to the SX6012 units that I read the manual for and based my planing on.

So there you have my brain dump. I'll up date this as go on with my research, but I've got some wiring decision to finalise soon so I may add some questions in regards to that here as well, like:

Is there such a thing as a QSFP to 10Gbe rj-45 breakout cable?
Is there such a thing as a QSFP room to room solution like there is for rj-45 i.e. the wall face plate?
 

Marsh

Moderator
May 12, 2013
2,343
1,179
113
Brocade ICX6610, they are cheap on eBay, but I may have to drop that as the cheap ones need a license activate the 10Gb ports.
For the past week, I purchased 6 x ICX6450 , 1 x ICX6610, all were fully licensed. YMMV
In case , if you brought a Brocade switch that have no license. Take a look at the threads in Networking section, you will find your wish. ;)
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,066
1,902
113
30
fohdeesha.com
  • Like
Reactions: MikeWebb

MikeWebb

Member
Jan 28, 2018
88
23
8
50
Well I'll be. That is very good to know indeed. I might just do away with the inifiniband idea and just use 40Gbe and 10Gbe as the NICs are cheaper as well.

I see you bit of a Brocade boffin, you might be able clear something up. I thought the icx6610...P supported:
4 x 40Gbe QSFP uplink on the rear,
8 x 1Gbe SFP (pay as you go 10Gbe SFP+)
24 x POE 1Gbe rj-45
I'm just going of notes here.

and will the switch with only one power supply do POE (about 7-8 devices mix of AF and AT)?
How noisy is this thing? I had to replace the fans in my srx320 with Noctua ones (bliss).
Any issues with transceiver compatibility?

I think you may be getting a PM from me soon.
 
  • Like
Reactions: fohdeesha

MikeWebb

Member
Jan 28, 2018
88
23
8
50
OMG this site rocks. I hadn't considered the possibilities when flashing. Just did a search on the sx6012 and very much the same possibilities.

Both sound like relatively cheap but involved projects to do, with in my dollar and space budget. I'm getting both. I may even have an eBay win like Marsh, kind of hoping not, I like the sound of fohdeesha's idea. Part of the reason behind this project.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,066
1,902
113
30
fohdeesha.com
There are indeed 4x 40gbps QSFP+ ports on the rear. However two of them are permanently configured as 4x 10gbE breakout ports, so you can not use them for a single 40gbE link - they only operate as 4x independent 10gbE ports, just like the independent SFP+ ports on the front. You just need a QSFP breakout to SFP+ cable to use them ($25 on ebay). That gives you another 8x 10gbE ports just using the two breakout ports on the rear

The remaining two QSFP+ ports are indeed full 40gbE normal ports. The graphic in the post I linked illustrates this pretty well ( think, anyway, I'm not an artist LOL)

And yes, the front 8 SFP+ ports are also 10gbE, you just have to load a license first. the PoE model does indeed have PoE on all ports (available in 24 port or 48 port versions)

PoE is still available on all ports with only one PSU active, but it halves the PoE budget - which isn't exactly an issue, as the PoE budget is 1500w - so with only one PSU you still have 750w of PoE power available

Noise is hard to quantify (yes, I know there's literally a measurement for this, but bare with me). I personally don't find it loud, and a few other members who have purchased one said it's much quieter than they were expecting, but others have said it's loud. It's quieter than a Dell R710 or an LB6M, if you've heard any of those. If it's in a rack with servers, you're not going to hear it

transceiver wise they will take literally anything, they are optic sluts. I've even used infiniband optics in them without an issue. All this is covered in great detail (with examples) in the first few pages of the icx6610 thread, I recommend reading through it
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,066
1,902
113
30
fohdeesha.com
Looking at the SRX320 datasheet, it seems it's exactly as loud as the icx6610 - around 37db. So it'll be as loud as your stock srx320 was. Read through the 6610 thread to see why, but replacing the fans in the 6610 is not really a road worth going down