How to prepare Mellanox SX6036 for iSER ?

[Ahrimanes]

RDMA and iSER is all new to me. I want to try it in my home lab, but I cant really find any good guides for it. I understand that I need to configure DCB on my switch that is a Mellanox SX6036 running PPC_M460EX 3.6.8012.

Is there anyone here who can explain how to configure DCB on this switch?

 

[kapone]

Both RDMA and iSER will work with zero support from a switch. Unless you're running at the edge of the performance envelope for the switch, you may not even need to do anything.

First get your environment up and running (without worrying about the switch), and see if everything's working as expected.

 

[Ahrimanes]

@kapone Thank you, I did as you said and yes it worked perfectly out of the box. The performance blew my mind! Compared to regular iSCSI it more then doubled my benchmark results! Sadly truenas decided that iSER should be a enterprise feature... I found a workaround that I hope will continue to work in the future.

 

[Ahrimanes]

I thought it would be a good idea to write a follow up If some one else finds this thread:

The entire rack had been shut off for a couple of days. After powering everything back on, strange problems appeared immediately. The most serious being random lost connections to datastores on all hosts (esx.problem.vmfs.heartbeat.timedout), along with slow read/write performance. After extensive troubleshooting I started to suspect that maybe I did start to configure DCB and forgot to save the config prior to shutdown. So I added these lines to the configuration.

# Global switch config
dcb priority-flow-control enable

dcb priority-flow-control priority 3 enable

# On interfaces
interface ethernet x/x
dcb priority-flow-control mode on

Now the r/w performance is good again and so far (knock on wood) there are no datastore connection losses.

@kapone

 

[BackupProphet]

Since youre already on sx6036, I would recommend using Infiniband instead of RoCE

 

[kapone]

I thought it would be a good idea to write a follow up If some one else finds this thread:

The entire rack had been shut off for a couple of days. After powering everything back on, strange problems appeared immediately. The most serious being random lost connections to datastores on all hosts (esx.problem.vmfs.heartbeat.timedout), along with slow read/write performance. After extensive troubleshooting I started to suspect that maybe I did start to configure DCB and forgot to save the config prior to shutdown. So I added these lines to the configuration.

# Global switch config
dcb priority-flow-control enable

dcb priority-flow-control priority 3 enable

# On interfaces
interface ethernet x/x
dcb priority-flow-control mode on

Now the r/w performance is good again and so far (knock on wood) there are no datastore connection losses.

@kapone

You shut everything down....how dare you!

 

[Ahrimanes]

You shut everything down....how dare you!

I used to work at a hospital where we had a yearly site redundancy test. Its really nerve wracking to shut down an entire production site , but also a good way to find out mistakes like this one

 

[kapone]

I used to work at a hospital where we had a yearly site redundancy test. Its really nerve wracking to shut down an entire production site , but also a good way to find out mistakes like this one

Tell me about it...

I'm making some improvements to my infra, and switching everything to zero touch provisioning, PXE boot, ansible playbooks, sequenced booting etc. In theory...I should be able to cut power to the entire thing, turn it back on, and watch everything essentially replay config files and playbooks and come online.

The impetus for doing this is also to be able to shut everything down in sequence (The only disks I have are in my SAN nodes, everything else is diskless) if power fails, and my UPSes kick in. They can survive for ~3-4 minutes at full load.

This is not an issue in my colo, but my home is the DR site, so...

 

[dsrhdev]

Sadly truenas decided that iSER should be a enterprise feature...

hello,
just add the modules to your /etc/modules:

root@truenas:/mnt/sata/backup/truenas# cat /etc/modules
mlx4_ib
mlx5_ib
ib_ipoib
ib_isert
isert_scst
svcrdma

if you want nfs over rdma also add dropp-in like this:

cat /etc/systemd/system/nfs-server.service.d/override-nfs.conf 
[Service]
ExecStartPost=/bin/bash -c 'echo rdma 20049 > /proc/fs/nfsd/portlist'