40GB Infiniband support for XPEnology... the dream is real!

Marsh · May 25, 2015

PnoT said:
looks like I'm saturating the 40GB IB network?

PnoT said:
TCP window size: 0.00 GByte (default)

It is too early for me, drinking my first cup of coffee.
32 Gbits/s is equal to 4 GBytes/s , the Mellanox card would passed maximum 4GBytes/s data between two hosts.

PnoT said:
[SUM] 0.0-10.0 sec 3929256 GBytes 392924 GBytes/sec

It is too good to be true

PnoT · May 25, 2015

Hank C said:
thx after the change. still doesn't show up. I am using LSI 2308 and intel c602 ahci

What's not showing up?

Hank C · May 25, 2015

I have 4 disk connect to Intel C602 configured to AHCI and 8 disk connect to LSI 2308 IT mode.
It only recognize 5 of them. Also I have changed the value in synoinfo.conf and hence you see disk 22 with usb.

Here is the output from dmesg
[ 4.568985] sd 1:0:0:0: [sdb] Attached SCSI disk
[ 5.087602] sd 6:0:0:0: [sdg] Attached SCSI disk
[ 7.907754] sd 7:0:0:0: [sdh] Attached SCSI disk
[ 8.107983] sd 7:0:1:0: [sdi] Attached SCSI disk
[ 8.397332] sd 7:0:2:0: [sdj] Attached SCSI disk
[ 8.657683] sd 7:0:3:0: [sdk] Attached SCSI disk
[ 8.934245] sd 7:0:4:0: [sdl] Attached SCSI disk
[ 9.177030] sd 7:0:5:0: [sdm] Attached SCSI disk
[ 9.399217] sd 7:0:6:0: [sdn] Attached SCSI disk
[ 9.645346] sd 7:0:7:0: [sdo] Attached SCSI disk

[ 4.035990] ata1: SATA max UDMA/133 abar m2048@0xd1600000 port 0xd1600100 irq 88
[ 4.035995] ata2: SATA max UDMA/133 abar m2048@0xd1600000 port 0xd1600180 irq 88
[ 4.035998] ata3: SATA max UDMA/133 abar m2048@0xd1600000 port 0xd1600200 irq 88
[ 4.036001] ata4: SATA max UDMA/133 abar m2048@0xd1600000 port 0xd1600280 irq 88
[ 4.036004] ata5: SATA max UDMA/133 abar m2048@0xd1600000 port 0xd1600300 irq 88
[ 4.036007] ata6: SATA max UDMA/133 abar m2048@0xd1600000 port 0xd1600380 irq 88
[ 4.047231] ata7: SATA max UDMA/133 abar m8192@0xd1500000 port 0xd1500100 irq 89

PnoT · May 25, 2015

paste the output of the following

Code:

dmesg | grep scsi

I get an entire list of my drives and here's a sample

Code:

[    9.393695] scsi 6:0:0:0: Direct-Access     ATA      ST4000VN000-1H4168       SC43 PQ: 0 ANSI: 6
[    9.393706] scsi 6:0:0:0: SATA: handle(0x000a), sas_addr(0x5003048000eec64c), phy(12), device_name(0x0000000000000000)
[    9.393711] scsi 6:0:0:0: SATA: enclosure_logical_id(0x500304800000007f), slot(0)
[    9.393985] scsi 6:0:0:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[    9.393994] scsi 6:0:0:0: qdepth(32), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1)
[    9.395145] sd 6:0:0:0: Attached scsi generic sg0 type 0
[    9.643746] scsi 6:0:1:0: Direct-Access     ATA      ST4000VN000-1H4168       SC43 PQ: 0 ANSI: 6
[    9.643758] scsi 6:0:1:0: SATA: handle(0x000b), sas_addr(0x5003048000eec64d), phy(13), device_name(0x0000000000000000)
[    9.643764] scsi 6:0:1:0: SATA: enclosure_logical_id(0x500304800000007f), slot(1)
[    9.643910] scsi 6:0:1:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[    9.643917] scsi 6:0:1:0: qdepth(32), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1)
[    9.645153] sd 6:0:1:0: Attached scsi generic sg1 type 0

Also what value are you using in synoinfo? How many total onboard SATA ports do you have? How many disks are you missing?

Hank C · May 25, 2015

I have a total of 18 sata onboard. 12 sata in use and those are the spinner disk. 7 of them are missing.

Code:

[    4.042046] scsi0 : ahci
[    4.042764] scsi1 : ahci
[    4.043133] scsi2 : ahci
[    4.043620] scsi3 : ahci
[    4.043885] scsi4 : ahci
[    4.044636] scsi5 : ahci
[    4.056918] scsi6 : ahci
[    4.355808] scsi 1:0:0:0: Direct-Access     OCZ      VECTOR150                1.2  PQ: 0 ANSI: 5
[    5.105011] scsi 6:0:0:0: Direct-Access     SAMSUNG  MZHPV256HDGL-00000       BXW2 PQ: 0 ANSI: 5
[    6.852384] iscsi: registered transport (tcp)
[    6.977732] sd 1:0:0:0: Attached scsi generic sg0 type 0
[    6.977824] sd 6:0:0:0: Attached scsi generic sg1 type 0
[    7.035694] scsi7 : Fusion MPT SAS Host
[    7.473229] scsi 7:0:0:0: Direct-Access     TOSHIBA  DT01ACA300               ABB0 PQ: 0 ANSI: 6
[    7.473236] scsi 7:0:0:0: SATA: handle(0x0009), sas_addr(0x4433221100000000), phy(0), device_name(0x0000000000000000)
[    7.473240] scsi 7:0:0:0: SATA: enclosure_logical_id(0x5001e6766c789000), slot(0)
[    7.473322] scsi 7:0:0:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[    7.473326] scsi 7:0:0:0: qdepth(32), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1)
[    7.473812] sd 7:0:0:0: Attached scsi generic sg2 type 0
[    7.723090] scsi 7:0:1:0: Direct-Access     TOSHIBA  DT01ACA300               ABB0 PQ: 0 ANSI: 6
[    7.723098] scsi 7:0:1:0: SATA: handle(0x000a), sas_addr(0x4433221101000000), phy(1), device_name(0x0000000000000000)
[    7.723101] scsi 7:0:1:0: SATA: enclosure_logical_id(0x5001e6766c789000), slot(1)
[    7.723183] scsi 7:0:1:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[    7.723187] scsi 7:0:1:0: qdepth(32), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1)
[    7.723610] sd 7:0:1:0: Attached scsi generic sg3 type 0
[    7.973326] scsi 7:0:2:0: Direct-Access     TOSHIBA  DT01ACA300               ABB0 PQ: 0 ANSI: 6
[    7.973333] scsi 7:0:2:0: SATA: handle(0x000b), sas_addr(0x4433221102000000), phy(2), device_name(0x0000000000000000)
[    7.973337] scsi 7:0:2:0: SATA: enclosure_logical_id(0x5001e6766c789000), slot(2)
[    7.973419] scsi 7:0:2:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[    7.973422] scsi 7:0:2:0: qdepth(32), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1)
[    7.973870] sd 7:0:2:0: Attached scsi generic sg4 type 0
[    8.222814] scsi 7:0:3:0: Direct-Access     TOSHIBA  DT01ACA300               ABB0 PQ: 0 ANSI: 6
[    8.222820] scsi 7:0:3:0: SATA: handle(0x000c), sas_addr(0x4433221103000000), phy(3), device_name(0x0000000000000000)
[    8.222824] scsi 7:0:3:0: SATA: enclosure_logical_id(0x5001e6766c789000), slot(3)
[    8.222905] scsi 7:0:3:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[    8.222909] scsi 7:0:3:0: qdepth(32), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1)
[    8.223425] sd 7:0:3:0: Attached scsi generic sg5 type 0
[    8.472679] scsi 7:0:4:0: Direct-Access     TOSHIBA  DT01ACA300               ABB0 PQ: 0 ANSI: 6
[    8.472687] scsi 7:0:4:0: SATA: handle(0x000d), sas_addr(0x4433221104000000), phy(4), device_name(0x0000000000000000)
[    8.472690] scsi 7:0:4:0: SATA: enclosure_logical_id(0x5001e6766c789000), slot(4)
[    8.472772] scsi 7:0:4:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[    8.472775] scsi 7:0:4:0: qdepth(32), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1)
[    8.473219] sd 7:0:4:0: Attached scsi generic sg6 type 0
[    8.723090] scsi 7:0:5:0: Direct-Access     TOSHIBA  DT01ACA300               ABB0 PQ: 0 ANSI: 6
[    8.723097] scsi 7:0:5:0: SATA: handle(0x000e), sas_addr(0x4433221105000000), phy(5), device_name(0x0000000000000000)
[    8.723101] scsi 7:0:5:0: SATA: enclosure_logical_id(0x5001e6766c789000), slot(5)
[    8.723182] scsi 7:0:5:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[    8.723186] scsi 7:0:5:0: qdepth(32), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1)
[    8.723740] sd 7:0:5:0: Attached scsi generic sg7 type 0
[    8.972991] scsi 7:0:6:0: Direct-Access     TOSHIBA  DT01ACA300               ABB0 PQ: 0 ANSI: 6
[    8.972998] scsi 7:0:6:0: SATA: handle(0x000f), sas_addr(0x4433221106000000), phy(6), device_name(0x0000000000000000)
[    8.973001] scsi 7:0:6:0: SATA: enclosure_logical_id(0x5001e6766c789000), slot(6)
[    8.973083] scsi 7:0:6:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[    8.973086] scsi 7:0:6:0: qdepth(32), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1)
[    8.973496] sd 7:0:6:0: Attached scsi generic sg8 type 0
[    9.222321] scsi 7:0:7:0: Direct-Access     TOSHIBA  DT01ACA300               ABB0 PQ: 0 ANSI: 6
[    9.222328] scsi 7:0:7:0: SATA: handle(0x0010), sas_addr(0x4433221107000000), phy(7), device_name(0x0000000000000000)
[    9.222332] scsi 7:0:7:0: SATA: enclosure_logical_id(0x5001e6766c789000), slot(7)
[    9.222415] scsi 7:0:7:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[    9.222418] scsi 7:0:7:0: qdepth(32), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1)
[    9.222822] sd 7:0:7:0: Attached scsi generic sg9 type 0
[   27.676431] scsi8 : usb-storage 1-1.2:1.0
[   28.679194] scsi 8:0:0:0: Direct-Access     ADATA    USB Flash Drive          0.00 PQ: 0 ANSI: 4
[   28.679391] sd 8:0:0:0: Attached scsi generic sg10 type 0

PnoT · May 25, 2015

The value you should have in /etc/synoinfo.cfg is

Code:

0x3fffffff

If you have 30 this is how the breakdown should look to get the value above.

Code:

0011     1111     1111     1111     1111     1111     1111     1111
3         f        f        f        f         f       f        f

Hank C · May 25, 2015

How did you find I have 30?

Hank C · May 25, 2015

I might have to give up..changed to 30 and still the same @@

PnoT · May 25, 2015

Hank C said:
I have a total of 18 sata onboard. 12 sata in use and those are the spinner disk. 7 of them are missing.

Sorry I misread the numbers.

It looks like the dmesg output you posted is showing 11 drives total with 1 of those being a USB drive but the screenshot you have is only showing 7. Did you add the usbport and esata entries into the synoinfo.cfg as well?

Code:

esataportcfg="0x00000"
usbportcfg="0x000000"

Hank C · May 25, 2015

i changed the usb to 9 and have not changed the esata yet.

Marsh · May 25, 2015

See my post #34 ,
Go back to basic.
make the 3 changes to two files , reboot.
The setting will have slot for 18 drives.

Hank C · May 25, 2015

i see there is a setting "mem_max_mb=32768"
have you changed this before?

changed the esata setting to match yours. Now I am missing 4 now. @@

Marsh · May 25, 2015

Hank C said:
changed the esata setting to match yours

Are you using my setting?

Try bump up to have more slots, there may be "holes" in the middle.
It depends on how the system detects the disks.

Hank C · May 25, 2015

here is what I used:
esataportcfg="0x0000000"
usbportcfg="0xf00000"
internalportcfg="0x3fffffff"

Marsh · May 25, 2015

Please use the config in post #34.
Takes a minute or two and a reboot.

Hank C · May 25, 2015

Marsh said:
Please use the config in post #34.
Takes a minute or two and a reboot.

using your config.
This is soooo weird...

epicurean · Aug 11, 2015

HI Pnot,
How do you get infiniband working with 5592 update 2 of xpenology?
How do you verify that xpenology even recognises the mellanox card?

PnoT · Aug 11, 2015

epicurean said:
HI Pnot,
How do you get infiniband working with 5592 update 2 of xpenology?
How do you verify that xpenology even recognises the mellanox card?

I haven't tried to enable support in the latest builds as every time the build is updated there are new quirks with how to get it to work properly. I've seen went to a dual 10GB card and am through with IB on XPEnology until they incorporate the drivers in the boot images.

If the drivers are loaded properly you should see interfaces starting at ib0/ib1 and so on.

epicurean · Aug 14, 2015

PnoT,
What was the last version of xpenology that you got 40GB to work, and how did you do it?

PnoT · Aug 14, 2015

5.1 something as 5.2 required a recompile in which things got a lot harder to install.

Look at the bottom of page 2 for some tips on how to get it to work that was provided by another forum member over there.

XPEnology NAS • View topic - 40GB InfiniBand Support at work on my Xpenology box!

40GB Infiniband support for XPEnology... the dream is real!

Moderator

Active Member

Active Member

Active Member

Active Member

Attachments

Active Member

Active Member

Active Member

Active Member

Active Member

Moderator

Active Member

Attachments

Moderator

Active Member

Moderator

Active Member

Attachments

Active Member

Active Member

Active Member

Active Member