Top of Rack switch 10gbe

[Destreyf]

I am about to co-locate some servers at a local data center in order to reduce my power consumption at my house as well as have better networking options (35 meg up at home).

I am planning on doing a half rack for now, i am going to be putting in 6 x Dell R630, some super micro servers and a Mellanox SX6036 40gig switch running IPoIB to handle all storage traffic, each server is equipped with an Intel X540 SFP+ nic's and Mellanox ConnectX-3 FDR/QDR QSFP card.

I have gotten most of my setup ready to put in the data center with the exception of the top of rack switch, i would like to be able to connect all of my servers to the TOR switch via SFP+, i don't have a preference on the media type (DAC/Optics).

I will be getting a /27 CIDR and would like to handle this on the switch but i am unsure of the best approach for a top of rack switch, what i'm looking for would have 24 (or preferably 48 for down the road) SFP+ ports and be able to handle the routing.

Any advice would be helpful and a direction on equipment would be awesome.

I am very comfortable with command line configuration, i have extensive history working with Cisco in the past and have configured many switches and vlans, i'm just at a loss for best practices and equipment choices for TOR switches.

 

[Lunar]

I recently got some Arista 7124SX 24 port SFP+ switches. They're pretty nice, and I found them for cheap on eBay. 2 of them powered on seems to do around 200-300W on 120V. If you only want Cisco, Nexus 3064 is nice.

 

[Monoman]

any layer 3 10g switch would work fine for this applications, but I'd really suggest putting a pfsense to handle the CIDR block/firewall for services/NAT/routing/VPN for remote management. This is how I have it setup for myself currently with a /26

 

[kapone]

Why do you need a router, if you're getting a /27 ? That gives you 32 addresses, and it sounds like you have less servers than that?

 

[Destreyf]

I recently got some Arista 7124SX 24 port SFP+ switches. They're pretty nice, and I found them for cheap on eBay. 2 of them powered on seems to do around 200-300W on 120V. If you only want Cisco, Nexus 3064 is nice.

I'm not partial to any brand, i was more clarifying experience dealing with the command line, i would go with whatever is cheapest/best for the job. I did not know about the Nexus 3064, that port config is pretty good looking and they're pretty cheap on ebay too.

any layer 3 10g switch would work fine for this applications, but I'd really suggest putting a pfsense to handle the CIDR block/firewall for services/NAT/routing/VPN for remote management. This is how I have it setup for myself currently with a /26

This is a great idea, I was putting off the best way to handle that, as i was trying to work out a top of rack switch for 10gig, this would make it easier to manage, and allow me to use a simpler 10gig switch.

Why do you need a router, if you're getting a /27 ? That gives you 32 addresses, and it sounds like you have less servers than that?

I plan on doing virtual machines and providing some managed services for some clients and some other use cases, the physical servers themselves would all be on private IP addresses and the public IP's would be used to connect to services that needed a dedicated IP address.

I plan on using a reverse proxy to handle any web hosting traffic to keep my public IP consumption to a minimum.

-----------

Initially i'm starting out with only a few servers, i do quite a bit of IT work and sys-admin work that could (and likely will) be hosted for clients, but i also host quite a few personal projects on these machines, i'm mainly moving these servers to a co-location because of bandwidth(upload)/power limitations at my current residence.

Awesome feedback from all of you, this has helped me answer several questions at once.

 

[gregsachs]

Would this work?
Brocade EMC MP-8000B 32-Port 24x 10GbE FCoE 8x 8Gb Fiber Channel Ethernet Switch | eBay
Brocade 24x10gbe +8x8gb FC, will also do FCoE, $162 obo right now....

I don't know anything about it...

 

[Destreyf]

any layer 3 10g switch would work fine for this applications, but I'd really suggest putting a pfsense to handle the CIDR block/firewall for services/NAT/routing/VPN for remote management. This is how I have it setup for myself currently with a /26

Hey @Monoman i was wondering if you could detail how you'd set that up?

I am actually going to be using OPNSense but it should be similar enough to pfsense, i am just looking for a general outline of how you would configure the /27 to be on a specific vlan as a general overview.

I am also wanting to do several other LAN subnets for VM's and then a network for kubernetes as well, this is early on though!