Omni-os FC LUN esxi hosts

[modder man]

I have built an small omni-os (with napp-it) SAN using a SuperMicro 113-6 1026T-URF.

Server Specs
Dual L5520
27Gb ram
6 fujitsu 15k 146GB disk
1 100GB intel DC3700
1 128GB micron SSD
Qlogic QLE2464 Quad port fibre channel card

I am trying to configure FC LUN's for my ESXi hosts. Am I sharing a iSCSI LUN, or is this different? I have the target and host groups setup and the WWN of the host FC cards setup and I cannot get the hosts to see the LUN's for the life of me. Any help would be greatly appreciated.

 

[whitey]

I don't know if this helps you at all or not and its been well over a year since I did this same thing but here are my 'high level' notes on the process. You may have to go research/RTFM a bit more to pull it all together. I think I have done this on both Solaris GA and OmniOS so the binary/package installs may differ a bit but the theory remains the same. It's very similar to iSCSI via the COMSTAR stack w/ minor syntax changes (you'll notice the stmfadm and sbdadm cmds still in use). This is a simple config, no complex fabric masking on initiator target portal group or any of that jazz. There are some good guides on the net, don't have time to dig them up now but if you are struggling I can dig deeper.

Configuring Fibre Channel Devices with COMSTAR
pkg install storage-server
mdb -k
::devbindings -q qlc (note pciex instance and driver qlc)
$q
update_drv -d -i ‘pciex1077,2532′ qlc
update_drv -a -i ‘pciex1077,2532′ qlt
reboot/init 6
mdb -k
::devbindings -q qlc (note driver is now qlt)
$q
svcadm enable stmf
stmfadm list-target -v
fcinfo hba-port (optional)
zfs create -b 64k -V 10G blockstgpool/fclun01
sbdadm create-lu /dev/zvol/rdsk/blockstgpool/fclun01
sbdadm list-lu
stmfadm list-lu -v
stmfadm add-view 600144F0B152CC0000004B080F230004
stmfadm list-view -l 600144F0B152CC0000004B080F230004

 

[modder man]

I think part of my problem is that I dont understand enough to know the right questions to ask. I am little unsure of exactly what my problems are. I followed a guide to setup comstar luns and at this point I should be able to see my luns and I cant.

 

[modder man]

I got a LUN created and it shows up for one host, the other host does not see the LUN. Set to view all.

 

[Deci]

mdb -k
::devbindings -q qlc (note pciex instance and driver qlc)
$q
update_drv -d -i ‘pciex1077,2532′ qlc
update_drv -a -i ‘pciex1077,2532′ qlt

If you haven't done this part you won't be able to use it as a target as by default the client driver is installed for qlogic cards, those commands go through finding the device with the qlc driver attached and then change it to qlt which is the target driver.

The pciex etc portion of the commands in between the ' ' will probably be different (you find what yours are with the devbindings command) so don't just use those commands as they are without checking first.

 

[modder man]

I have completed that section, the SAN FC card is definitely set to Target mode. I think the problems I have surround the brocade 200E switch. Unfortunately there doesn’t seem to be a whole lot of documentation out there for it and the GUI seems a bit clumsy.

 

[Deci]

Can you change the cables the clients are using between each other and the working one still works? (dud fibre lead)
Can you bypass the switch and directly connect the clients to the host and it works? (Check if the switch is causing the issue)
Can you change the port on the switch and it still work? (might have config in it seperating ports off or restricting access/acl, might be setup to operate differently to what you are trying to do with it, from memory there is more than one operating mode/style. If all else fails make it operate in the dumbest mode possible so it acts like an unmanaged device and allows all ports access to all other ports and check if the clients can see the target.

 

[TuxDude]

I don't know anything about Omni-OS, but I can probably help if the issue is with the Brocade switch. Can you describe or do a quick diagram to show how everything is physically connected together? And also get the output of the 'zoneshow' command from the switch. How are you managing the switch, the CLI or the web-gui?

Edit: While you're in there, may as well also see if the switch's firmware is up to date. The 'firmwareshow' command will tell you that.

 

[modder man]

Well I dont know that the switch is really right or anything at this point butIi did get it to work. I discovered that you have to run the setup manager to set which ports are HBA and which are storage. I had a bunch of invalid connections. Now all three esxi hosts see my storage luns.

Output from switch

-----------------------------------------------------------------
SANSW01:admin> zoneshow
Defined configuration:
cfg: cfg zone_4_0; zone_4_1; zone_4_2; zone_4_3; zone_5_0; zone_5_1;
zone_5_2; zone_5_3; zone_6_0; zone_6_1; zone_6_2; zone_6_3;
zone_7_0; zone_7_1; zone_7_2; zone_7_3; zone_8_0; zone_8_1;
zone_8_2; zone_8_3; zone_9_0; zone_9_1; zone_9_2; zone_9_3;
zone_10_0; zone_10_1; zone_10_2; zone_10_3; zone_11_0;
zone_11_1; zone_11_2; zone_11_3; zone_12_0; zone_12_1;
zone_12_2; zone_12_3; zone_13_0; zone_13_1; zone_13_2;
zone_13_3; zone_14_0; zone_14_1; zone_14_2; zone_14_3;
zone_15_0; zone_15_1; zone_15_2; zone_15_3
zone: zone_10_0
1,10; 1,0
zone: zone_10_1
1,10; 1,1
zone: zone_10_2
1,10; 1,2
zone: zone_10_3
1,10; 1,3
zone: zone_11_0
1,11; 1,0
zone: zone_11_1
1,11; 1,1
zone: zone_11_2
1,11; 1,2
zone: zone_11_3
1,11; 1,3
zone: zone_12_0
1,12; 1,0
zone: zone_12_1
1,12; 1,1
zone: zone_12_2
1,12; 1,2
zone: zone_12_3
1,12; 1,3
zone: zone_13_0
1,13; 1,0
zone: zone_13_1
1,13; 1,1
zone: zone_13_2
1,13; 1,2
zone: zone_13_3
1,13; 1,3
zone: zone_14_0
1,14; 1,0
zone: zone_14_1
1,14; 1,1
zone: zone_14_2
1,14; 1,2
zone: zone_14_3
1,14; 1,3
zone: zone_15_0
1,15; 1,0
zone: zone_15_1
1,15; 1,1
zone: zone_15_2
1,15; 1,2
zone: zone_15_3
1,15; 1,3
zone: zone_4_0
1,4; 1,0
zone: zone_4_1
1,4; 1,1
zone: zone_4_2
1,4; 1,2
zone: zone_4_3
1,4; 1,3
zone: zone_5_0
1,5; 1,0
zone: zone_5_1
1,5; 1,1
zone: zone_5_2
1,5; 1,2
zone: zone_5_3
1,5; 1,3
zone: zone_6_0
1,6; 1,0
zone: zone_6_1
1,6; 1,1
zone: zone_6_2
1,6; 1,2
zone: zone_6_3
1,6; 1,3
zone: zone_7_0
1,7; 1,0
zone: zone_7_1
1,7; 1,1
zone: zone_7_2
1,7; 1,2
zone: zone_7_3
1,7; 1,3
zone: zone_8_0
1,8; 1,0
zone: zone_8_1
1,8; 1,1
zone: zone_8_2
1,8; 1,2
zone: zone_8_3
1,8; 1,3
zone: zone_9_0
1,9; 1,0
zone: zone_9_1
1,9; 1,1
zone: zone_9_2
1,9; 1,2
zone: zone_9_3
1,9; 1,3

Effective configuration:
cfg: cfg
zone: zone_10_0
1,10
1,0
zone: zone_10_1
1,10
1,1
zone: zone_10_2
1,10
1,2
zone: zone_10_3
1,10
1,3
zone: zone_11_0
1,11
1,0
zone: zone_11_1
1,11
1,1
zone: zone_11_2
1,11
1,2
zone: zone_11_3
1,11
1,3
zone: zone_12_0
1,12
1,0
zone: zone_12_1
1,12
1,1
zone: zone_12_2
1,12
1,2
zone: zone_12_3
1,12
1,3
zone: zone_13_0
1,13
1,0
zone: zone_13_1
1,13
1,1
zone: zone_13_2
1,13
1,2
zone: zone_13_3
1,13
1,3
zone: zone_14_0
1,14
1,0
zone: zone_14_1
1,14
1,1
zone: zone_14_2
1,14
1,2
zone: zone_14_3
1,14
1,3
zone: zone_15_0
1,15
1,0
zone: zone_15_1
1,15
1,1
zone: zone_15_2
1,15
1,2
zone: zone_15_3
1,15
1,3
zone: zone_4_0
1,4
1,0
zone: zone_4_1
1,4
1,1
zone: zone_4_2
1,4
1,2
zone: zone_4_3
1,4
1,3
zone: zone_5_0
1,5
1,0
zone: zone_5_1
1,5
1,1
zone: zone_5_2
1,5
1,2
zone: zone_5_3
1,5
1,3
zone: zone_6_0
1,6
1,0
zone: zone_6_1
1,6
1,1
zone: zone_6_2
1,6
1,2
zone: zone_6_3
1,6
1,3
zone: zone_7_0
1,7
1,0
zone: zone_7_1
1,7
1,1
zone: zone_7_2
1,7
1,2
zone: zone_7_3
1,7
1,3
zone: zone_8_0
1,8
1,0
zone: zone_8_1
1,8
1,1
zone: zone_8_2
1,8
1,2
zone: zone_8_3
1,8
1,3
zone: zone_9_0
1,9
1,0
zone: zone_9_1
1,9
1,1
zone: zone_9_2
1,9
1,2
zone: zone_9_3
1,9
1,3

 

[modder man]

and the firmware

SANSW01:admin> firmwareshow
Appl Primary/Secondary Versions
------------------------------------------
FOS v6.2.0c
v6.2.0c

 

[TuxDude]

I'm guessing setup manager must be something in OmniOS, because the FC switch doesn't care if a port is connected to a target or an initiator - they're all just F-ports as far as the switch is concerned. If the zoning configuration on the switch is empty/disabled, then the default is to allow everything to talk to everything - it will probably work and in a very small/simple environment like this is probably fine to leave like that. There is typically no problem with FC initiators (your ESXi hosts) being able to see and talk to each other and you don't have multiple clients/targets that you would want to keep separate for security issues.

Best practice in a FC fabric is usually to configure single-initiator zoning, which basically means what it sounds like: only one FC HBA in any zone. This will keep broadcast notifications across the fabric down to only what needs to see them. In your current configuration, any time any of your ESXi hosts reboots every single other host is also seeing a FC RSCN to notify it that a new device has logged into the fabric. The recommended configuration would be to configure one zone for each ESXi host (I'm assuming that you're only connecting a single port from each host to the switch), with each of those zones containing the WWN of that hosts HBA plus also the WWNs of all of the ports on your storage target. If you have 4 ESXi hosts that means you will end up with 4 zones, and that the WWNs from your storage target are in all 4 of those zones. In a normal production-level FC environment you would also have 2 matching sets of switches that do not connect to each other at all (dual-fabric), your hosts with dual-port HBA's (or dual single-port HBAs) would be connected with a single port to each fabric, both fabrics would be configured with zones for each initiator WWN, and your target device would also have its ports split across both fabrics. Then MPIO at the SCSI layer takes care of load-balancing across all of the available ports/fabrics

As to your firmware, it is close to the latest available for that switch, but you might want to upgrade. The 200e cannot run any firmware newer than 6.2.2x (newer 8gb / 16gb switches are running the 7.x code), but 6.2.2f9 is the most up-to-date release for your hardware, and it contains an updated bash to fix the shellshock vulnerability. As you are already on 6.2.x you can upgrade directly from the existing version to 6.2.2f9 and unless there is another critical vulnerability found and patched that will likely be the last firmware ever released for that switch.

 

[modder man]

Setup Manager is a utility in the GUI of the brocade 200E. Once I ran that and told the switch manually which was storage and which was not it worked fine. There very well could be more to this that I just dont understand, but it seemed to fix the problem

 

[TuxDude]

The Brocade 200e's that I have here (though mine are HP-branded) are on firmware 6.2.2f, and I don't have a 'setup manager' anywhere in the web/java-based GUI. In any case, most likely that tool used the info you gave it to automatically setup the single-initiator zoning I described above. If you post the output of the 'zoneshow' command I will probably be able to tell you if that is what it did, or if that tool wiped out the zoning and left everything wide open.

 

[modder man]

the zoneshow output is above.

 

[TuxDude]

Ok, so that tool configured zoning for you, using port-based zoning (I prefer WWN-based, but both work), and configuring only a single target + single initiator per zone. I'm going to guess you told it that your storage target (OP says it has a quad-port HBA) was plugged in to ports 0, 1, 2, 3 on the switch. Zones exist there to allow every port on the switch to talk to ports 0-3. As a quick example, zone_4_0 contains members 1,4 and 1,0 - those members are in the format <domain-id>,<port> - you only have a single switch which is domain-id 1, so that zone allows port 4 to talk to port 0. zone_4_1 contains 1,4 and 1,1 and allows port 4 to talk to port 1, and so own down the entire list of them.

If you never plan to add additional storage targets to this config, you can just leave it as-is and keep connecting additional clients to ports 4+ on the switch and everything will work just fine. If you plan to add more storage targets, or grow to multiple FC switches etc. then I recommend clearing out that config and using name-based zoning instead which will make maintaining the configuration over the next X years far simpler.