40GB Infiniband support for XPEnology... the dream is real!

[Marsh]

why is your random write qd=32 dropped so much?

The backend storage is only have 1 single old hard drive, sometimes, if it is cache miss, test would hit the single slow drive.

 

[PnoT]

i thought so too! any special config to use the RAM as cache?
also do you have url for xpenology initial install how-to?

This will get you started for sure... look into the tutorials. XPEnology.nl - Opensource NAS

I'm using a program called ImDisk for Windows which is pretty easy to create a RAM drive and then fill it up with whatever you want to use for transferring to the other machine(s) for testing.

 

[PnoT]

The backend storage is only have 1 single old hard drive, sometimes, if it is cache miss, test would hit the single slow drive.

Yea, I'm doing 1GB/sec to start off and about 1/2-3/4 of the xfer my RAM utilization on the xpenology fills up and it all hits the drives at about 450Mb/sec. I think the way that linux allocates RAM for swap is affecting the overall speed as it consumes the RAM we're trying to use for cache but that's just a guess on my part.

 

[Hank C]

if xpenology works, i'll ditch freenas since i don't need l2arc or zil anymore

 

[PnoT]

I have a 1815+ at home and love it but there are just aspects about it that get on my nerves, performance wise, so moving to xpenology was great. I was hesitant at first but after reading and implementing it a few times it's pretty simple really.

 

[Marsh]

Prior to Xpenology ( year 2013 ) and SSD, I used freenas and few other storage OS. I would spent days tweaking just make it work.
Today, it takes 5 to 10 min to install Xpenology using any semi-modern hardware and 2 gb memory. It just work, no more spending hours chasing down strange slow performance issue. I just want a stable NAS / SAN.

You could also run Xpenology as VM on hyper-v and vSphere as well.

 

[Hank C]

save my day and money! thx!

 

[tjk]

Does Xpenology have the HA/failover capabilities that the commercial product offers? How about the distributed volumes across units?

 

[PnoT]

Does Xpenology have the HA/failover capabilities that the commercial product offers? How about the distributed volumes across units?

Synology does have HA but I've never personally used it and I'm not sure if xpenology works with it or not. Best bet would be to hit their forums up and see what people have posted about it.

 

[Marsh]

Stick 4 x Intel 1500 pro SSD ( raid-0 ) into Xpenology test rig , couple tests before taking the test rig apart
Still using 32gb memory as ram cache

Xpenology G3258 at 4.2 ghz , 32gb memory , 1 Mellanox 40gb ipoib network card
4 x Intel 1500 pro SSD in RAID-0
-----------------------------------------------------------------------
CrystalDiskMark 3.0.3 x64 (C) 2007-2013 hiyohiyo
Crystal Dew World : Crystal Dew World
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

Sequential Read : 1043.013 MB/s
Sequential Write : 969.408 MB/s
Random Read 512KB : 963.585 MB/s
Random Write 512KB : 878.187 MB/s
Random Read 4KB (QD=1) : 29.690 MB/s [ 7248.5 IOPS]
Random Write 4KB (QD=1) : 50.404 MB/s [ 12305.6 IOPS]
Random Read 4KB (QD=32) : 276.408 MB/s [ 67482.5 IOPS]
Random Write 4KB (QD=32) : 271.244 MB/s [ 66221.8 IOPS]

Test : 2000 MB [Z: 0.0% (0.3/642.1 GB)] (x5)
Date : 2015/05/24 7:31:39
OS : Windows Server 2012 R2 Datacenter (Full installation) [6.3 Build 9600] (x64)

-----------------------------------------------------------------------
CrystalDiskMark 3.0.3 x64 (C) 2007-2013 hiyohiyo
Crystal Dew World : Crystal Dew World
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

Sequential Read : 1008.367 MB/s
Sequential Write : 836.268 MB/s
Random Read 512KB : 972.618 MB/s
Random Write 512KB : 893.037 MB/s
Random Read 4KB (QD=1) : 29.705 MB/s [ 7252.2 IOPS]
Random Write 4KB (QD=1) : 47.069 MB/s [ 11491.4 IOPS]
Random Read 4KB (QD=32) : 275.313 MB/s [ 67215.1 IOPS]
Random Write 4KB (QD=32) : 246.042 MB/s [ 60068.7 IOPS]

Test : 4000 MB [Z: 0.0% (0.3/642.1 GB)] (x5)
Date : 2015/05/24 7:38:45
OS : Windows Server 2012 R2 Datacenter (Full installation) [6.3 Build 9600] (x64)

 

[Marsh]

Taking memory down to 4gb ram, to match the test data file size 4gb to reduce ram cache effect.

Xpenology G3258 at 4.2 ghz , 4gb memory , 1 Mellanox 40gb ipoib network card
4 x Intel 1500 pro SSD in RAID-0

-----------------------------------------------------------------------
CrystalDiskMark 3.0.3 x64 (C) 2007-2013 hiyohiyo
Crystal Dew World : Crystal Dew World
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

Sequential Read : 622.393 MB/s
Sequential Write : 813.480 MB/s
Random Read 512KB : 511.033 MB/s
Random Write 512KB : 637.990 MB/s
Random Read 4KB (QD=1) : 23.234 MB/s [ 5672.4 IOPS]
Random Write 4KB (QD=1) : 46.632 MB/s [ 11384.8 IOPS]
Random Read 4KB (QD=32) : 102.958 MB/s [ 25136.2 IOPS]
Random Write 4KB (QD=32) : 176.536 MB/s [ 43099.5 IOPS]

Test : 4000 MB [Z: 0.0% (0.3/642.1 GB)] (x5)
Date : 2015/05/24 7:55:39
OS : Windows Server 2012 R2 Datacenter (Full installation) [6.3 Build 9600] (x64)

 

[Hank C]

I tested mine with 5x 3tb SHR and 10gbe network. It is nowhere near yours. Also, I have 12x hdd but it only recognized 5x for no reason.My benchmark is about the same as 12x 3tb RAIDZ in freenas.

 

[Marsh]

Just figure out how to install iperf to my Xpenology hyper-v VM.
I'll go out for couple hours for lunch.
Will be back to setup the entire test rig again, after I dismantled the setup.
I'll report back with iperf measurement.

 

[Marsh]

have 12x hdd but it only recognized 5x for no reason.

you will need to ssh into your Xpenology server, make changes to /etc/synoinfo.conf and /etc.defaults/synoinfo.conf
My 16 drives with LSI 9207 config

esataportcfg="0x00000"
usbportcfg="0xf00000"
internalportcfg="0xfffff"

additional info: https://forums.servethehome.com/ind...s-xpenology-synology-clone-with-lsi-sas.2973/

 

[PnoT]

This is the guide to reference and figure out what values you should use for your own setup.

XPEnology NAS • View topic - Xpenology and SAS Controller - HBA

The /etc/synoinfo.conf actually does nothing that I can see because when you change it nothing ever happens but the /etc.defaults/synoinfo.conf is the one to tinker with. You can modify the file, save it, go into storage manager and after about 30 seconds it will enumerate your new values so you don't have to reboot between tweaking and getting the drives right.

If you need some help let me know

 

[Marsh]

Spent last few hours trying to get the iperf speed higher.

Xpenology using G3258 @4.2 ghz , 16gb mem , CPU usage 85% running "iperf -s -m"

Windows Client dual E5-2650 , 64gb mem ,
iperf -c 10.10.10.3 -P 8 [result = 19 Gbits/second]
increase iperf -c 10.10.10.3 -P 9 or -P 10 makes no difference , capped at 19 Gbits/second
Running bi-directional tests "iperf -c 10.10.10.3 -P 8 -t 30 -d" , 14.7 Gbits/s up and 14.5 Gbits/s down.

Moved Xpenology to I7-3770K host , 16gb mem, CPU usage 25% running "iperf -s -m"
Same result , capped at 19 Gbits/second.

Issue:
Xpenology Mellanox IB driver ignore MTU setting, it is stuck in MTU=1500, tried ifconfig ib0 mtu 2044 or anynumber higher than 2044 would produced error "invalid argument".
Changing MTU to maximum would help but Xpenology driver is stuck at MTU=1500 .

Mellanox card shown connection rate 32 Gbps/s which is correct, I guess I'll be happy to get 25 Gbits/second.

 

[PnoT]

Spent last few hours trying to get the iperf speed higher.

Xpenology using G3258 @4.2 ghz , 16gb mem , CPU usage 85% running "iperf -s -m"

Windows Client dual E5-2650 , 64gb mem ,
iperf -c 10.10.10.3 -P 8 [result = 19 Gbits/second]
increase iperf -c 10.10.10.3 -P 9 or -P 10 makes no difference , capped at 19 Gbits/second
Running bi-directional tests "iperf -c 10.10.10.3 -P 8 -t 30 -d" , 14.7 Gbits/s up and 14.5 Gbits/s down.

Moved Xpenology to I7-3770K host , 16gb mem, CPU usage 25% running "iperf -s -m"
Same result , capped at 19 Gbits/second.

Issue:
Xpenology Mellanox IB driver ignore MTU setting, it is stuck in MTU=1500, tried ifconfig ib0 mtu 2044 or anynumber higher than 2044 would produced error "invalid argument".
Changing MTU to maximum would help but Xpenology driver is stuck at MTU=1500 .

Mellanox card shown connection rate 32 Gbps/s which is correct, I guess I'll be happy to get 25 Gbits/second.

C:\iperf-2.0.5-3-win32>iperf -c 192.168.2.8 -f G -P 8
------------------------------------------------------------
Client connecting to 192.168.2.8, TCP port 5001
TCP window size: 0.00 GByte (default)
------------------------------------------------------------
[ 10] local 192.168.2.5 port 51127 connected with 192.168.2.8 port 5001
[ 8] local 192.168.2.5 port 51125 connected with 192.168.2.8 port 5001
[ 7] local 192.168.2.5 port 51124 connected with 192.168.2.8 port 5001
[ 9] local 192.168.2.5 port 51126 connected with 192.168.2.8 port 5001
[ 6] local 192.168.2.5 port 51123 connected with 192.168.2.8 port 5001
[ 5] local 192.168.2.5 port 51122 connected with 192.168.2.8 port 5001
[ 3] local 192.168.2.5 port 51120 connected with 192.168.2.8 port 5001
[ 4] local 192.168.2.5 port 51121 connected with 192.168.2.8 port 5001
[ ID] Interval Transfer Bandwidth
[ 10] 0.0-10.0 sec 489272 GBytes 48927 GBytes/sec
[ 8] 0.0-10.0 sec 491084 GBytes 49110 GBytes/sec
[ 7] 0.0-10.0 sec 490648 GBytes 49066 GBytes/sec
[ 9] 0.0-10.0 sec 490940 GBytes 49094 GBytes/sec
[ 6] 0.0-10.0 sec 493564 GBytes 49358 GBytes/sec
[ 5] 0.0-10.0 sec 491324 GBytes 49135 GBytes/sec
[ 3] 0.0-10.0 sec 492168 GBytes 49220 GBytes/sec
[ 4] 0.0-10.0 sec 490256 GBytes 49029 GBytes/sec
[SUM] 0.0-10.0 sec 3929256 GBytes 392924 GBytes/sec

I'm a total noob with iperf and just used your commands but that looks like I'm saturating the 40GB IB network?

I think you're right as well Marsh about the MTU setting being odd because I can change it to 2000 and it actually sticks. The 2044 issued command goes through without error but if you query the adapter again it reverts to 1500.

Btw, great job on all the testing!!

 

[Hank C]

how did you log in to synology? did you use root or admin?
I used admin to log in but could not modify the file

 

[Marsh]

ssh to host, username is root , password is what you set for admin user during installation.

 

[Hank C]

ssh to host, username is root , password is what you set for admin user during installation.

thx after the change. still doesn't show up. I am using LSI 2308 and intel c602 ahci