40GB Infiniband support for XPEnology... the dream is real!

[PnoT]

Trantor compiled the modules the other day and I was able to get around to verifying connectivity and an initial round of speed tests. If there is anything else that you think I should test or missed please let me know.

XPEnology NAS • View topic - 40GB InfiniBand Support at work on my Xpenology box!

 

[Marsh]

Please let us know your hardware configuration.
I have a SuperMicro 3U chassis running older version Xpenology to take advantage of the Mellanox 10gb network card,
now wondering to upgrade or not.

 

[PnoT]

Please let us know your hardware configuration.
I have a SuperMicro 3U chassis running older version Xpenology to take advantage of the Mellanox 10gb network card,
now wondering to upgrade or not.

I'm running a Supermicro X8SIL-F with an Intel(R) Xeon(R) CPU X3450 @ 2.67GHz with 16GB of RAM. It's an older machine but has IPMI which has come in super handy when doing tests on XPEnology.

The XPEnoboot 5.1-5022.3 added official support for the 10GB Mellanox cards and also DSM 5.1.x

If you're going to upgrade make sure to save a copy of your /etc/synoinfo.conf just in case.

 

[Marsh]

Sorry,
I have 3 severers running Xpenology DSM 5.1 SSD only with 10gb Mellanox card serving NFS and iSCSI for various ESXi ,Proxmox and Hyper-V hosts.

What I meant to say: is upgrading from 10gb to 40gb Mellanox card worth it?

I'll take some time next week to try out.

 

[ehorn]

Thanks for the update PnoT.

Never played with XPE... Might have to now..

Post up your ram drive results when you have some time.

 

[PnoT]

Sorry,
I have 3 severers running Xpenology DSM 5.1 SSD only with 10gb Mellanox card serving NFS and iSCSI for various ESXi ,Proxmox and Hyper-V hosts.

What I meant to say: is upgrading from 10gb to 40gb Mellanox card worth it?

I'll take some time next week to try out.

Oh, I got ya. What are the speeds you're getting on those 10gb cards currently? If you have SSDs in them they should be pretty damn fast.

Thanks for the update PnoT.

Never played with XPE... Might have to now..

Post up your ram drive results when you have some time.

No problem and will do!

 

[Marsh]

I am too cheap to buy 10gb switchs, instead, I build a bunch of low end Intel G32xx ITX box with 2-4gb memory, a $25 Mellanox 10gb EN dual port card and 4 cheap SSD running as RAID0, Use the Xpenology as NAS / SAN with 2 ESXi box to form a three ring cluster.
In case that you are curious, VMs are backup to HP Microserver running Xpenology with RAID5.

Last year with IOMeter testing, I got it up to 8-9Gbit/sec, pretty cheap for the setup, and without any effort on tuning.

Now, I am seeing that you are getting 4 x 10gb/sec link, curious how it is all workout. Certainly, give me some work to do next week.

 

[PnoT]

I'm trying to get a ram disk on my xpenology to work but having some difficulties. When using ramfs to create the drive and mount it I can't create any file son the target above zero KB and it states there's no room on the device.

 

[Marsh]

I just setup 2 Xpenology servers, 1 Windows 2012R2 clients to test the new ip over ib drivers for Xpenology.

First Xpenology server:
MSI Z97-GD65 board with Celeron G3258 cpu, 32 gb memory, single Seagate 7200.7 sata-1 160gb harddisk, 1 Mellanox ConnectX-2 IB card.
Set MTU to 2044
ifconfig ib0 10.10.10.3 netmask 255.255.255.0 broadcast 10.10.10.255 mtu 2044

Second Xpenlology server:
Supermicro X9DAX , 1 E6-2660 cpu , 32 gb mem, single Seagate 7200.7 sata-1 160gb
harddisk, 1 Mellanox ConnectX-2 IB card.
( E6-2600 is clock slower, the benchmark test result actually is slower than G3258 cpu )


Windows client to run CrystalDiskMark
ASUS Z9PA-D8 , dual E5-2650 , 64 gb mem , 1 single OS boot disk , 1 Mellanox ConnectX-2 IB card.
set Windows mtu to match Xpenology ib0 mtu
netsh interface ipv4 show subinterface

Test data size is 100MB and 2GB , to ensure that it is testing against Xpenology ram cache.
The reason for use a single sata-1 hard drive, is to prove Xpenology is caching the test data file in ram, because I do not know how to make a ramfs device for Xpenology.

Result is in next posts.

Result is not encouraging I may stop, not continue testing with 8 x Intel SSD in raid-0
I was getting similar result with the Mallenox 10gb EN card.
To achieve higher speed, last time, I had to use Windows OS to create ram disk, to run benchmark, but it is not real.
I been using Mellanox 10gb EN card for a while , and really happy with it due to plug and play nature.

 

[Marsh]

100MB test data

-----------------------------------------------------------------------
CrystalDiskMark 3.0.3 x64 (C) 2007-2013 hiyohiyo
Crystal Dew World : Crystal Dew World
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

Sequential Read : 724.822 MB/s
Sequential Write : 907.932 MB/s
Random Read 512KB : 557.660 MB/s
Random Write 512KB : 881.376 MB/s
Random Read 4KB (QD=1) : 28.503 MB/s [ 6958.8 IOPS]
Random Write 4KB (QD=1) : 50.446 MB/s [ 12315.9 IOPS]
Random Read 4KB (QD=32) : 183.209 MB/s [ 44728.8 IOPS]
Random Write 4KB (QD=32) : 202.399 MB/s [ 49413.7 IOPS]

Test : 100 MB [Z: 0.2% (0.3/142.2 GB)] (x5)
Date : 2015/05/23 18:24:13
OS : Windows Server 2012 R2 Datacenter (Full installation) [6.3 Build 9600] (x64)

 

[Marsh]

2GB test data

-----------------------------------------------------------------------
CrystalDiskMark 3.0.3 x64 (C) 2007-2013 hiyohiyo
Crystal Dew World : Crystal Dew World
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

Sequential Read : 697.539 MB/s
Sequential Write : 923.856 MB/s
Random Read 512KB : 559.868 MB/s
Random Write 512KB : 900.028 MB/s
Random Read 4KB (QD=1) : 27.992 MB/s [ 6834.0 IOPS]
Random Write 4KB (QD=1) : 47.458 MB/s [ 11586.5 IOPS]
Random Read 4KB (QD=32) : 182.509 MB/s [ 44557.9 IOPS]
Random Write 4KB (QD=32) : 211.584 MB/s [ 51656.3 IOPS]

Test : 2000 MB [Z: 0.2% (0.3/142.2 GB)] (x5)
Date : 2015/05/23 18:49:48
OS : Windows Server 2012 R2 Datacenter (Full installation) [6.3 Build 9600] (x64)

 

[Hank C]

so does it use ram cache for the xpenology? it looks like it uses it though

 

[Hank C]

also did you test it on cifs or iscsi?

 

[Marsh]

cifs only.
Last year, I did a lot of testing with 40gb Mellanox , for my personal home lab use, there isn't load in this house to push beyond 10gb. I stopped and focused my build on Mellanox 10gb EN due to plug and play acts as regular Ethernet card.

 

[Marsh]

Just for fun, overclocked G3258 from 3.2 ghz to 4.2 ghz clock.
100 MB test data, slight increase

-----------------------------------------------------------------------
CrystalDiskMark 3.0.3 x64 (C) 2007-2013 hiyohiyo
Crystal Dew World : Crystal Dew World
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

Sequential Read : 761.719 MB/s
Sequential Write : 933.297 MB/s
Random Read 512KB : 578.754 MB/s
Random Write 512KB : 932.649 MB/s
Random Read 4KB (QD=1) : 29.076 MB/s [ 7098.7 IOPS]
Random Write 4KB (QD=1) : 51.019 MB/s [ 12455.7 IOPS]
Random Read 4KB (QD=32) : 182.660 MB/s [ 44594.7 IOPS]
Random Write 4KB (QD=32) : 204.441 MB/s [ 49912.2 IOPS]

Test : 100 MB [Z: 0.2% (0.3/142.2 GB)] (x5)
Date : 2015/05/23 19:34:07
OS : Windows Server 2012 R2 Datacenter (Full installation) [6.3 Build 9600] (x64)

 

[Marsh]

so does it use ram cache for the xpenology? it looks like it uses it though

Using ram cache to test throughput of the Mellanox 40gb ipoib card.
I only have one sata-1, 8 years old disk in the Xpenology server, there is no way to server up 4,4594 IOPS from 1 hard drive.

 

[Marsh]

Just for fun, overclocked G3258 from 3.2 ghz to 4.2 ghz clock.
2GB test data, slight increase

-----------------------------------------------------------------------
CrystalDiskMark 3.0.3 x64 (C) 2007-2013 hiyohiyo
Crystal Dew World : Crystal Dew World
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

Sequential Read : 756.094 MB/s
Sequential Write : 937.903 MB/s
Random Read 512KB : 580.412 MB/s
Random Write 512KB : 927.573 MB/s
Random Read 4KB (QD=1) : 28.588 MB/s [ 6979.6 IOPS]
Random Write 4KB (QD=1) : 48.505 MB/s [ 11842.0 IOPS]
Random Read 4KB (QD=32) : 182.677 MB/s [ 44598.8 IOPS]
Random Write 4KB (QD=32) : 35.914 MB/s [ 8768.0 IOPS]

Test : 2000 MB [Z: 0.2% (0.3/142.2 GB)] (x5)
Date : 2015/05/23 19:49:14
OS : Windows Server 2012 R2 Datacenter (Full installation) [6.3 Build 9600] (x64)

 

[PnoT]

The transfer is definitely happening via RAM as I can move an 8GB file from 2012 R2 @ 1GB/sec to the xpenology box and see the RAM jump on that box from ~6 to ~13.5GB and it slowly creep back down as the data is being written to disk.

 

[Hank C]

Using ram cache to test throughput of the Mellanox 40gb ipoib card.
I only have one sata-1, 8 years old disk in the Xpenology server, there is no way to server up 4,4594 IOPS from 1 hard drive.

i thought so too! any special config to use the RAM as cache?
also do you have url for xpenology initial install how-to?

 

[Hank C]

Just for fun, overclocked G3258 from 3.2 ghz to 4.2 ghz clock.
2GB test data, slight increase

-----------------------------------------------------------------------
CrystalDiskMark 3.0.3 x64 (C) 2007-2013 hiyohiyo
Crystal Dew World : Crystal Dew World
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

Sequential Read : 756.094 MB/s
Sequential Write : 937.903 MB/s
Random Read 512KB : 580.412 MB/s
Random Write 512KB : 927.573 MB/s
Random Read 4KB (QD=1) : 28.588 MB/s [ 6979.6 IOPS]
Random Write 4KB (QD=1) : 48.505 MB/s [ 11842.0 IOPS]
Random Read 4KB (QD=32) : 182.677 MB/s [ 44598.8 IOPS]
Random Write 4KB (QD=32) : 35.914 MB/s [ 8768.0 IOPS]

Test : 2000 MB [Z: 0.2% (0.3/142.2 GB)] (x5)
Date : 2015/05/23 19:49:14
OS : Windows Server 2012 R2 Datacenter (Full installation) [6.3 Build 9600] (x64)

why is your random write qd=32 dropped so much?