Hi
I set up new fileserver on Gentoo linux with Supermicro case, Xeon, 12GB ECC RAM and 21TB 9265 RAID6 (read/write ~800MB/s).
Supermicro SC846 24x SATA Storage Server 9650SE-24ML SAS846A 2x PWS-1K21P-1R | eBay
I figured out infiniband being best method to connect it to my workstation running Windows 8.1 x64
ASUS P8B WS Professional, i3 3240, 16GB ECC RAM and 6TB 9265 RAID5 (read/write ~600MB/s)
I've bought 2x Mellanox MHGH28-XTC and Genuine Mellanox Infiniband CX4 sff8470 to sff8470 Cable 3m/10ft SAS MCC4L30-003
Genuine Mellanox Infiniband CX4 sff8470 to sff8470 Cable 3m/10ft SAS MCC4L30-003 | eBay
I compiled on Gentoo modules giving infiniband support (mlx4_ib, ib_umad, ib_ipoib, ib_uverbs). Controller is visible by system. ib0 and ib1 interfaces may be configured.
On windows 8.1 I had more problems. As I eventually got to know from Mellanox support last working driver for this Connectx 1 controller is MLNX_VPI_WinOF-3_2_0_wlh_x64 (had to run this with compatibility mode Windows 7). Controller was installed succesfully. Two iboip adapters appeared.
I setup OpenSM.
I firstly used standard settings
Windows port type Infiniband (16 Gbps/s)
Gentoo fileserver port type Infiniband (16 Gbps/s) (ib0 mode datagram)
ibstat shown Link UP and ACTIVE
ping is working OK
But iperf and file copying tests are strange.
Copy over samba from fileserver to Windows is ~1MB/s (terribly slow)
Copy over samba from Windows to fileserver is ~200MB/s (still to slow)
Windows iperf -s fileserver iperf -c (from fileserver to Windows) ~3Mbits/s (terribly slow)
Windows iperf -s fileserver iperf -c -R ~3Mbits/s (from Windows to fileserver) ~6Gbits/s (still to slow)
Then I changed ib0 mode to connected
echo connected > /sys/class/net/ib0/mode
and this changed nothing
Then I aligned MTU to 4092
and this changed nothing
Then I thought to check Windows - Windows (and finally Linux - Linux).
When I run on both machines Windows 8.1 x64 with adapters set up Infiniband port type I gained:
Copy over samba (both directions same results) ~600MB/s (acceptable)
iperf (both directions same results) 8Gbps/s (acceptable)
Then I changed port type to Ethernet (10Gbps/s). Results are pretty much as above.
Then I finally run fileserver Gentoo and workstation Ubuntu 16.04.
Out of the box settings
ib0 mode datagram, port type Infiniband
Copy over samba (both directions same results) ~1MB/s (terribly slow)
iperf (both directions same results) 3Mbits/s (terribly slow)
then I changed ib0 mode on both ends to connected. And results appeared to be cool:
Copy over samba (both directions same results) ~600MB/s (good) (Rather RAID6 limitation)
iperf (both directions same results) 12Gbits/s (good)
Now I stuck. Seems poor performance on Windows - Linux configuratin is caused by software issue.
I had idea to change interface mode to connected on windows but dont know how.
Also I thought to set up connection on port type ethernet. But changing this on windows results in cable disconnected.
I found how to change this on linux:
echo eth > /sys/bus/pci/devices/0000\:01\:00.0/mlx4_port1
But this results in Physical state: Disabled on ibstat and cable is still disconnected. Maybe I am lacking ethernet port type module?
Why out of stock Windows > Fileserver is still slow (200MB/s - 1,5Gbits/s) but acceptable while opposite direction so terribly slow?
I hope this is not OSes incompatible problem but Linux - Linux and Windows - Windows configurations not facing issue is a question.
I appreciate any help.
br
Paweł
I set up new fileserver on Gentoo linux with Supermicro case, Xeon, 12GB ECC RAM and 21TB 9265 RAID6 (read/write ~800MB/s).
Supermicro SC846 24x SATA Storage Server 9650SE-24ML SAS846A 2x PWS-1K21P-1R | eBay
I figured out infiniband being best method to connect it to my workstation running Windows 8.1 x64
ASUS P8B WS Professional, i3 3240, 16GB ECC RAM and 6TB 9265 RAID5 (read/write ~600MB/s)
I've bought 2x Mellanox MHGH28-XTC and Genuine Mellanox Infiniband CX4 sff8470 to sff8470 Cable 3m/10ft SAS MCC4L30-003
Genuine Mellanox Infiniband CX4 sff8470 to sff8470 Cable 3m/10ft SAS MCC4L30-003 | eBay
I compiled on Gentoo modules giving infiniband support (mlx4_ib, ib_umad, ib_ipoib, ib_uverbs). Controller is visible by system. ib0 and ib1 interfaces may be configured.
On windows 8.1 I had more problems. As I eventually got to know from Mellanox support last working driver for this Connectx 1 controller is MLNX_VPI_WinOF-3_2_0_wlh_x64 (had to run this with compatibility mode Windows 7). Controller was installed succesfully. Two iboip adapters appeared.
I setup OpenSM.
I firstly used standard settings
Windows port type Infiniband (16 Gbps/s)
Gentoo fileserver port type Infiniband (16 Gbps/s) (ib0 mode datagram)
ibstat shown Link UP and ACTIVE
ping is working OK
But iperf and file copying tests are strange.
Copy over samba from fileserver to Windows is ~1MB/s (terribly slow)
Copy over samba from Windows to fileserver is ~200MB/s (still to slow)
Windows iperf -s fileserver iperf -c (from fileserver to Windows) ~3Mbits/s (terribly slow)
Windows iperf -s fileserver iperf -c -R ~3Mbits/s (from Windows to fileserver) ~6Gbits/s (still to slow)
Then I changed ib0 mode to connected
echo connected > /sys/class/net/ib0/mode
and this changed nothing
Then I aligned MTU to 4092
and this changed nothing
Then I thought to check Windows - Windows (and finally Linux - Linux).
When I run on both machines Windows 8.1 x64 with adapters set up Infiniband port type I gained:
Copy over samba (both directions same results) ~600MB/s (acceptable)
iperf (both directions same results) 8Gbps/s (acceptable)
Then I changed port type to Ethernet (10Gbps/s). Results are pretty much as above.
Then I finally run fileserver Gentoo and workstation Ubuntu 16.04.
Out of the box settings
ib0 mode datagram, port type Infiniband
Copy over samba (both directions same results) ~1MB/s (terribly slow)
iperf (both directions same results) 3Mbits/s (terribly slow)
then I changed ib0 mode on both ends to connected. And results appeared to be cool:
Copy over samba (both directions same results) ~600MB/s (good) (Rather RAID6 limitation)
iperf (both directions same results) 12Gbits/s (good)
Now I stuck. Seems poor performance on Windows - Linux configuratin is caused by software issue.
I had idea to change interface mode to connected on windows but dont know how.
Also I thought to set up connection on port type ethernet. But changing this on windows results in cable disconnected.
I found how to change this on linux:
echo eth > /sys/bus/pci/devices/0000\:01\:00.0/mlx4_port1
But this results in Physical state: Disabled on ibstat and cable is still disconnected. Maybe I am lacking ethernet port type module?
Why out of stock Windows > Fileserver is still slow (200MB/s - 1,5Gbits/s) but acceptable while opposite direction so terribly slow?
I hope this is not OSes incompatible problem but Linux - Linux and Windows - Windows configurations not facing issue is a question.
I appreciate any help.
br
Paweł