EoIB on Solaris 11.2 setup problems

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

RimBlock

Active Member
Sep 18, 2011
837
28
28
Singapore
Been having a bit of an issue getting EoIB up and running on a Solaris box.

I have been looking at various guides an have had it running until I had to rebuild the SAN.

The problem seems to be between getting the IP network devices created.

The main guide I have been following is from Oracle and can be found here.

The SAN has a QDR Connectx-2 card. Port 1 is damaged so I am using port 2. I have also tried with a DDR card on port 1 with the same issue. The problem persists whether the cable is plugged in or not.

The issue is that the eoip device once created manually will not come up.

Code:
root@San:~#  dladm show-phys
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net3              Infiniband           up         32000  unknown   ibp1
net2              Infiniband           down       0      unknown   ibp0
net1              Ethernet             up         1000   full      e1000g1
net0              Ethernet             up         1000   full      e1000g0
Code:
root@San:~# dladm show-ib net3
LINK      HCAGUID        PORTGUID       PORT STATE   GWNAME       GWPORT   PKEYS
net3      2C903005169F8  2C903005169FA  2    up      --           --       FFFF
Code:
root@San:~# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000
net0: flags=100001004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4,PHYSRUNNING> mtu 1500 index 8
        inet 192.168.2.118 netmask ffffff00 broadcast 192.168.2.255
        etherx:xx:xx:xx:xx:d8
net1: flags=100001004843<UP,BROADCAST,RUNNING,MULTICAST,DHCP,IPv4,PHYSRUNNING> mtu 1500 index 6
        inet 192.168.2.117 netmask ffffff00 broadcast 192.168.2.255
        ether x:xx:xx:xx:xx:d9
lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
        inet6 ::1/128
net0: flags=120002004841<UP,RUNNING,MULTICAST,DHCP,IPv6,PHYSRUNNING> mtu 1500 index 8
        inet6 fe80::225:50ff:fe59:bad8/10
        ether x:xx:xx:xx:xx:d8
net1: flags=120002004841<UP,RUNNING,MULTICAST,DHCP,IPv6,PHYSRUNNING> mtu 1500 index 6
        inet6 fe80::225:50ff:fe59:bad9/10
        ether x:xx:xx:xx:xx:d9
Command being used to create the eoib device
Code:
dladm create-eoib -l net3 -g voltaire-4036 -c 0a-eth-1 eib1
"voltaire-4036" is based on the return from my switch and "0a-eth-1 " is just taken from guides.
Code:
root@San:~# ibswitches
Switch  : 0x0008f10500109582 ports 36 "Mellanox 4036 # voltaire-4036" enhanced port 0 lid 1 lmc 0
From ifconfig -a
Code:
eib1: flags=1000803<UP,BROADCAST,MULTICAST,IPv4> mtu 1500 index 12
        inet 10.0.0.1 netmask ffffff00 broadcast 10.0.0.255
        ether 0:0:0:0:0:0
but this results in
Code:
root@San:~# ipadm show-addr eib1
ADDROBJ           TYPE     STATE        ADDR
eib1/v4           static   inaccessible 10.0.0.1/24
I have reinstalled Solaris a couple of times now and this is getting really frustrating especially as the original time I followed a guide I did not save the link to and it just worked.....

Suggestions on what may be going wrong and how to fix it ?.
 

RimBlock

Active Member
Sep 18, 2011
837
28
28
Singapore
Solaris San to Windows server.

I used to use SRP between Solaris and ESXi but the SRP drivers did not exist I the Windows IB package.

Any other alternatives worth investigating ?.

Strange thing is that when I set up the eoib originally I remember being surprised how easy it was. Come this rebuild and nothing but frustration.

I also notice the IB link is only 10GB for some reason. Prior to the rebuild with two ddr cards (and DBAs firmware for RDMA update) it was running at 16GB. The second card is now a qdr without the RDMA firmware but I do not believe this is the core issue with getting eoib up on the Solaris box.
 

markpower28

Active Member
Apr 9, 2013
413
104
43
I am not sure EoIB is the best option here. Are you trying to use SMB for windows or iSCSI for Windows sine SRP is not an option here?

I have tested ZFS (iSCSI target) and Windows initiator. The performance is just regular IBoIP. It's no where near Windows based SMB 3.0 (RDMA).

I am not sure there is any RDMA type of alternative at the moment for ZFS to Windows client. I ended up with two filers. ZFS for ESXi using SRP (RDMA) and Windows SMB 3.0 for Hyper-V (RDMA).
 
  • Like
Reactions: T_Minus

RimBlock

Active Member
Sep 18, 2011
837
28
28
Singapore
iSCSI is the route I am trying to get going and had got going previously. The performance was not great but was better than GbE.

iSER would have been nice but I would not be surprised if that was also not supported in Windows.

EoIB is the only other option I am aware of but it has been a year or two since I last dug into it.
 

markpower28

Active Member
Apr 9, 2013
413
104
43
IPoIB may be your only option here.

iSER will be a no go for ConnectX 2 since the the newer driver stopped support on those card, it works with ESXi. The driver still in beta 1.8.2

I have not mess with EoIB. How does Windows client aware of it? Is ethernet switch required for that? I am not sure if IB card and IB switch know ethernet by default. You may need some bridge/gateway device for that.
 

RimBlock

Active Member
Sep 18, 2011
837
28
28
Singapore
I have a Voltaire 4036 which has the subnet manager running on it. The card is picked up and has the drivers Installed automatically for Eoib on Windows server 2012. The IB link gets established automatically (16Gb for IB DDR). The connection is on its own subnet although I guess you bound bridge the eoib and standard tcpip connections via the server. There are devices that can also do the bridge get but they tend to be rare and experience site on the used market.

I keep the eoib on the 10.x.x.x network as the connection is only for San traffic. My Gbe networking is on 192.168.x.x and I keep them segregated.

The problem 8 am currently having is getting the Solaris IB device cloned to a eoib devise and then accepting an assigned ip address and coming up.

I could make the windows box a VM on ESXi but I was doing that a year or two ago with srp and while it was fine, I don't really need lots of VMs for anything at this time.
 

RimBlock

Active Member
Sep 18, 2011
837
28
28
Singapore
Yes I have the voltaire small running although I have been just using the default partition.

To be clear, it was all working fine last week. I then needed to reinstall.Saris over the weekend and i am now having issues getting it working group on the Solaris end.

One other question though, my voltaire et work management connection is connected to my land and seems to have a lot of activity (flashing activity light) even when the rest of the network is quiet. Have you seen that with yours ?. I am curios as to why.
 

RimBlock

Active Member
Sep 18, 2011
837
28
28
Singapore
Nope, currently giving it another so now.

Have had a not so fun couple of days getting some LSI 9211-8i cards firmware updated which resulted in a reset of the motherboard BIOS and a reinstall of Solaris. Initial impressions though are that nothing has changed. It seems to be getting the IB ports recognised as network devises for Ethernet traffic is the core problem as they are not getting setup automatically on install.
 

RimBlock

Active Member
Sep 18, 2011
837
28
28
Singapore
Sorted.

Needed to create a IB partition rather than IB device.

Code:
dladm create-part -l net3 -P 0xFFFF pffff.net3
ipadm create-ip pffff.net3
ipadm create-addr -a 10.0.0.1 pffff.net3/ipv4
Setup the iSCSI target as normal, mapped the lun and the Windows box can see and attach without an issue.
 

gea

Well-Known Member
Dec 31, 2010
3,141
1,182
113
DE
btw
I do not use myself but current napp-it includes IB settings in menu system.
(Menu was one with the help of user Frank)