Infiniband support?

[MACscr]

I have dual 10Gb Top Spin Infiniband cards and a TRX 100 switch with 4 x 10Gb CX4 ports. I just found that out that freebsd's IB support is pretty poor and freenas definitely doesnt support it. Nas4Free might be an option, but doesnt seem like that many people are using IB with it or with freebsd in general. So, my question is, what about solaris variant distros? Before I had IB, i tried out Omnius and found its community to be a bit small, slow, and very little info available. It also ran horribly slow on a usb flash drive because it wasnt optimized for it.

So, with those two things in mind, any recommendations on a distro that would work well with both? Ive honestly almost thought about going with ZFS on Linux, but its still a little ways from production even thought hey claim its ready.

 

[gea]

I have dual 10Gb Top Spin Infiniband cards and a TRX 100 switch with 4 x 10Gb CX4 ports. I just found that out that freebsd's IB support is pretty poor and freenas definitely doesnt support it. Nas4Free might be an option, but doesnt seem like that many people are using IB with it or with freebsd in general. So, my question is, what about solaris variant distros? Before I had IB, i tried out Omnius and found its community to be a bit small, slow, and very little info available. It also ran horribly slow on a usb flash drive because it wasnt optimized for it.

So, with those two things in mind, any recommendations on a distro that would work well with both? Ive honestly almost thought about going with ZFS on Linux, but its still a little ways from production even thought hey claim its ready.

I do not use IB but 10Gbe but you may find some infos at Archive InfiniBand » ZFS Build

If you want to use USB sticks with any Solaris based systems:
- use a minimal server-config like OmniOS or OI/Solaris server
- use quite large and really fast ones (I use 16GB USB3 ones - even when there is not USB3 support, they are mostly faster)
- ZFS mirror them for reliability and performance
- disable atime

With such sticks, management is fast.
And NAS/SAN performance is not dependent to rpool performance.


Beside that most infos about Illumos based systems (Nexenta4, OmniOS, OpenIndiana and SmartOS) are common usable,
(base is the same) even Solaris 11 is not that different.

 

[cactus]

I use IPoIB with OmniOS stable. Setup has to be done through CLI and I used Oracle's how-tos, so I would assume Solaris supports it also.

 

[the spyder]

We primarily use Mellanox QDR Inifiband cards with Solaris 10/11 and OI151. Truenas has ported the IB drive in to freenas, but the licensing killed it for us. Freenas performance is just terrible. So far we see 3.2GB's in Windows 7 fairly consistently.

 

[Patrick]

We primarily use Mellanox QDR Inifiband cards with Solaris 10/11 and OI151. Truenas has ported the IB drive in to freenas, but the licensing killed it for us. Freenas performance is just terrible. So far we see 3.2GB's in Windows 7 fairly consistently.

OT but how did you like TrueNAS? Seems like that would get expensive fairly quickly.

 

[MACscr]

Freenas actually performed quite well with is current version when it came to zfs. The zfsbuild website shows that as well. I guess I can try omnios again, I just find its cli to be a bit frustrating and its hardware support to be even more lacking. I keep reading that OI is going to die off because the main dev left it, but they sure seem way more active than omnios. Especially their irc chat rooms, which I tend to live in. I guess I could give it another shot.

 

[gea]

Freenas actually performed quite well with is current version when it came to zfs. The zfsbuild website shows that as well. I guess I can try omnios again, I just find its cli to be a bit frustrating and its hardware support to be even more lacking. I keep reading that OI is going to die off because the main dev left it, but they sure seem way more active than omnios. Especially their irc chat rooms, which I tend to live in. I guess I could give it another shot.

OI is a pure community project without an enterprise behind. It is not dead but if a main contributor leaves, development is not too fast.
This is not relevant at home. If you want to use Solaris alike systems in production, i would go Oracle Solaris or OmniOS.

Both have paid support, stable releases and bugfixes. With OmniOS the payed support is optional but if you count on it, you should use this option.

Info: next stable for OmniOS is on the way:
ReleaseNotes

 

[MACscr]

So i switched to Omnios and the performance is much better and it recognized my IB card right out the gate. I unfortunately just found out something though thats pretty important. My switch appears to actually have 10Gb Ethernet ports. Doh! Just got into the switch for the first time and found out it has a Broadcom BCM56514 - 48 Port 4Ten-Gigabit Ethernet Line Card (http://www.broadcom.com/collateral/pb/56514-PB01-R.pdf). I am trying to figure how I missed that and what I am going to do next. =(

 

[cabroadb]

So i switched to Omnios and the performance is much better and it recognized my IB card right out the gate. I unfortunately just found out something though thats pretty important. My switch appears to actually have 10Gb Ethernet ports. Doh! Just got into the switch for the first time and found out it has a Broadcom BCM56514 - 48 Port 4Ten-Gigabit Ethernet Line Card (http://www.broadcom.com/collateral/pb/56514-PB01-R.pdf). I am trying to figure how I missed that and what I am going to do next. =(

I just finally saw your thread here. I believe you've posted on numerous other forums per this switch and infiniband. Someone has asked a lot about it if you haven't. If I'd seen it earlier I would have warned you about what you were doing. Infiniband and Ethernet are different protocols and will interoperate. The only way you will be able to interface with your switch is with an Ethernet card.

If you do want to incorporate an Infiniband fabric into your existing network and interface between the Ethernet and Infiniband, you'll have to set up different subnets for the Ethernet and Infiniband IP ranges and route traffic between them, possibly with Linux IP tables or BSD's PF. It's doable and not that difficult to figure out once you have IP going on both sides.

If you are just trying to set up a NAS or single server using a faster network connection to distribute to the slower connection machines, you'll just want to get a 10GbE card with CX4. In this case, your Infiniband card is useless for you unless you're willing to invest in a switch and/or cards for all systems which will be connecting to that interface.