FCoE

[macrules34]

I have 9 servers that only have one pcie slot and only one Ethernet port. I was thinking about implementing FCoE since I have FC arrays. What hardware do I need to setup this implementation? I have two Cisco Nexus 5020 switches with FC ports for storage. Please let me know if I’m missing something? I’m looking to separate the server management ports from out side main communication.

 

[dsrhdev]

hello,
is nvme-of-tcp not an option?

 

[macrules34]

Here is what I have for hardware:
- 3x EMC AX4-5F (Fiber channel, could go iscsi) storage arrays
- 9x Servers with a single PCI-e expansion slot each, running latest Proxmox VE.
- I do have a server that I built that is sitting idle at the moment that has a x16 2.5” ssd enclosure (no SSDs in it at the moment).

What hardware is required to do NVMe-over tcp?

 

[dsrhdev]

To run NVMe over Fabrics (with a TCP backend), you generally just need a TCP/IP-capable NIC and an NVMe Target Server. As far as I know, the EMC AX4-5F might not support this feature, or FCoE. Its iSCSI support is a good sign, though I couldn't find clear confirmation of that online.
Using your single PCIe port for a relatively expensive FC card might limit the future functionality of your servers. By the way, does that port support PCIe bifurcation?
Could you also share which specific NIC is installed in the server (or embedded on the motherboard)?

 

[MBastian]

1. Tuning NVMe-oF mounts to not starve and disconnect on heavy writes to a HDD pool is not an easy task. The whole protocol is geared toward low latency and would need extensive tuning. From my personal experience with just one client over a 25Gbit/s link I can't even guarantee that it would be even possible with 9 write heavy clients. Also you would need to have at least one dedicated storage server which would limit all clients to the servers network link speed.

2. Imho FCoE might be far the better choice with your setup. As far as I can tell you need FC modules (either N5K-M1008 or N5K-M1404, depending of how many FC arrays you want to connect) and the storage protocol license on the Cisco Nexus 5020 switches. Mind that you also need 10G SFP+ network ports in all clients.

 

[macrules34]

@dsrhdev the EMC AX4-5F only supports 4gb fiber channel. No the PCIe slot dose not support bifurcation. The Nic port is built on the motherboard. Could I use the storage array the add the storage to the NVMe target server and then present that storage to the client servers?

@MBastian I do have a server that I could dedicate to a NVMe over tcp target, the server that I would use has multiple open PCIe slots so I can install a newer NIC. I would have to dedicate the only PCIe slot in the 9 servers to that FCoE card.

 

[kapone]

You have fairly old hardware, and frankly not up to spec for running the kind of workloads (I think) you're trying to run.

4gb FCoE is ancient. And not even that fast. An ancient Mellanox Connectx-3 in ETH mode will murder it at 40/56gbps. And these can be had for like $15/ea.

Your "servers" don't have high speed nics, or enough expansion slots. What's the point of FCoE in this context??

 

[dsrhdev]

4gb FCoE is ancient. And not even that fast. An ancient Mellanox Connectx-3 in ETH mode will murder it at 40/56gbps. And these can be had for like $15/ea.

yes, i agree, since my question was about the pci-e slot, and cx-3 can boot via iscsi even (but im not sure the EMC AX4-5F supports it, i also could find the 4gb FCoE only)

 

[macrules34]

Well, I want to have storage and more network ports, so I thought FCoE would be an option since I only have one expansion slot.

 

[MBastian]

The Nexus 5020 provides the FCoE so the EMC arrays should be good. But 4gb ist truly ancient. As @kapone already wrote, imho not worth the effort. Better buy old Melanox gear.

 

[macrules34]

I’m limited by my storage speed of 4gb. You can have either FC or iSCSi on the array not both.

 

[kapone]

Well, I want to have storage and more network ports, so I thought FCoE would be an option since I only have one expansion slot.

Not gonna work. Honestly? This is all e-waste. Throw it in the trash bin.

 

[macrules34]

Not gonna work. Honestly? This is all e-waste. Throw it in the trash bin.

I can’t afford the latest and greatest equipment, so I have to make do with what I have. Any help (besides throwing in the trash) is greatly appreciated.

 

[kapone]

I can’t afford the latest and greatest equipment, so I have to make do with what I have. Any help (besides throwing in the trash) is greatly appreciated.

It's your call. This might give you some pointers...

https://dl.dell.com/manuals/all-products/esuprt_ser_stor_net/esuprt_ha_cluster_soln/esuprt_ha_cluster_soln_emc_fc_storage/dell-emc-ax4-5-win-ha-ctr_deployment%20guide_en-us.pdf

This equipment is so old that it's likely not gonna support drives greater than 2TB and that too at 3gbps. Forget NVME-oF, you're outclassed in that area.

You're gonna need supported FC cards in your 9x servers to connect the storage to them. And then make do with your onboard NICs for outbound network connectivity.

 

[dsrhdev]

Well, I want to have storage and more network ports, so I thought FCoE would be an option since I only have one expansion slot.

it might be, but you will need to use some PLX riser to bifurcate the only slot (it will cost like cx3-/4 + fc card)

 

[macrules34]

Okay so maybe I’ll do as @kapone said and use the PCIe slot for FC cards and use the onboard Nic for traffic.

Also the servers have A single usb-c port so maybe I can get a usb-c 10gb Nic compatible with Proxmox.

 

[kapone]

Okay so maybe I’ll do as @kapone said and use the PCIe slot for FC cards and use the onboard Nic for traffic.

Also the servers have A single usb-c port so maybe I can get a usb-c 10gb Nic compatible with Proxmox.

If these are "consumer" type systems (i.e. mini/micro/itx etc), there's a good probability that FC cards won't work on them. SMBUS and all that...

 

[macrules34]

Is there anyway to check that?

 

[kapone]

Is there anyway to check that?

Yes. Buy a card, put it in the slot...

Boot up fine? - It works.
No boot? - It doesn't work (and then go down the rabbit hole of SMBUS and taping pins and all that...)

 

[MBastian]

It would help if you could tell us what server hardware you have.