I'm using this ESXI network setup in my test server running:
- OmniOS as the NFS share
- ESXi local vSwitch to connect OmniOS NFS share to ESXI for datastore
- 9000 MTU for Jumbo Frame support on storagenet vswitch and 2nd OmniOS nic.
- 1500 MTU for regular traffic to smb shares to other outside machines on 1st OmniOS nic.
Thought I would share it as I heard about the concept in a few places but not any step by step guide.
I will assume you already have OmniOS setup with networking and will add a second Network Card using vmxnet3 to the OmniOS VM. Also assume VMware tools are already installed and vmware tools \bin perl script was not modified.
Regular data traffic to outside clients/servers will be on:
virtual machine port group: "VM Network" - with OmniOS ip address set to 192.168.1.15
ESXi VMKernel Port - Management Network - IP address 192.168.1.220
Storage traffic will be on a new vSwitch on IP address in range:
192.168.20.1 - 192.168.20.254
with virtual machine port group: "storagenet"
with OmniOS ip address set to 192.168.20.15
ESXi VMKernel Port - "VMKernel-storagenet" - IP address 192.168.20.220
Steps:
Under "Configuration -> Networking" in ESXi 5.5
Setting up separate vSwitch for storage traffic:
1. Click Add Networking.
2. Choose "Virtual Machine" type
3. Choose "Create a vShpere standard switch" and uncheck any "vmnic1" etc. we want "No physical Adapters".
4. Choose network label "storagenet".
5. Choose Finish.
Add VMkernel Port to new vSwitch
(allowing NFS mount via this virtual switch to OmniOS)
6. Make a note of the vSwitch# for new Virtual Machine Port Group.
7. Click Add Networking.
8. Choose "VMKernel"
9. Select "Use vSwitch#" (that you noted)
In the preview it should list the new storagenet.
10. Choose network label "VMKernel-storagenet".
11. Next.
12. Use the following IP settings:
IP: 192.168.20.220
(this will be the IP you must authorize in NFS share)
Subnet mask: 255.255.255.0
13. Next, Finish.
Edit MTU settings in vSwitch and VMkernel to accommodate Jumbo Frames
14. Choose "Properties" next to storagenet vswitch#
15. Click on vSwitch on left view of properties and choose "Edit"
16. Change MTU to 9000 and click OK.
17. Click on VMKernel-storagenet on left view and choose "Edit"
18. Change MTU to 9000 and click OK.
(now your virtual switches will pass MTU 9000 packets correctly)
Add new vmxnet3 network card to OmniOS VM.
19. Edit Settings on OmniOS VM
20. "Add" -> "Ethernet Adapter"
21. Choose type: vmxnet3, Network Label: storagenet
22. Click Next, Finish.
Setup IP address in OmniOS VM
(this assumes you already have a setup vmxnet3s network adapter)
23. Create interface if needed
# ipadm create-if vmxnet3s1
24. Setup interface ip address
# ipadm create-addr -T static -a 192.168.20.15/24 vmxnet3s1/v4
Show adapters
# dladm show-link
LINK CLASS MTU STATE BRIDGE OVER
vmxnet3s0 phys 1500 up -- -- (management interface 1 GB physical)
vmxnet3s1 phys 1500 unknown -- -- (Storage interface 10G virtual)
Check the speed and state
#dladm show-phys
25. Configure Jumbo Frames in OmniOS (refer to other thread)
See:
https://forums.servethehome.com/ind...le-and-napp-it-vmxnet3-and-jumbo-frames.2853/
Note: Different MTU settings for file sharing/regular traffic and virtual storage network.
If you would like the original adapter vmxnet3s0 to stay as MTU 1500,
and the new vmxnet3s1 to be MTU 9000 perform the JumboFrame setups in 25, and afterwards:
****************************
*** Allow one VMXNet3 adapter to be MTU 1500,
*** otherwise will default to max 9000,
*** this setting will persist across reboots.
****************************
Change regular file sharing MTU so packets don't have to be re-fragmented.
# ipadm set-ifprop -p mtu=1500 -m ipv4 vmxnet3s0
26. Ensure ZFS sharenfs security is set to allow new ip address of storage vmkernel:
You can use : to allow multiple hosts:
# zfs set sharenfs=rw=192.168.1.220:192.168.20.220,root=192.168.1.220:192.168.20.220 pool/dstore
or just one host (allowing new esxi vmkernel storage ip address to connect):
# zfs set sharenfs=rw=192.168.20.220,root=192.168.20.220 pool/dstore
------------------------------------
Unmount any NFS shares from old IP address in ESXi
27. Remove from inventory any virtual machines running off old NFS storage
28. Remove NFS datastores shared off old NFS storage ip address (and switch).
Add NFS shares to ESXi from OmniOS using new storagenet vSwitch
29. Click on "Configuration" - "Storage" - "Add Storage"
30. Choose Network File System, "next"
31. Enter: Server: 192.168.20.15 (new vmxnet interface we added for OmniOS)
Folder: /pool/dstore
Datastore Name: VSAN-pool-dstore
32. Profit from fast NFS storage!
- OmniOS as the NFS share
- ESXi local vSwitch to connect OmniOS NFS share to ESXI for datastore
- 9000 MTU for Jumbo Frame support on storagenet vswitch and 2nd OmniOS nic.
- 1500 MTU for regular traffic to smb shares to other outside machines on 1st OmniOS nic.
Thought I would share it as I heard about the concept in a few places but not any step by step guide.
I will assume you already have OmniOS setup with networking and will add a second Network Card using vmxnet3 to the OmniOS VM. Also assume VMware tools are already installed and vmware tools \bin perl script was not modified.
Regular data traffic to outside clients/servers will be on:
virtual machine port group: "VM Network" - with OmniOS ip address set to 192.168.1.15
ESXi VMKernel Port - Management Network - IP address 192.168.1.220
Storage traffic will be on a new vSwitch on IP address in range:
192.168.20.1 - 192.168.20.254
with virtual machine port group: "storagenet"
with OmniOS ip address set to 192.168.20.15
ESXi VMKernel Port - "VMKernel-storagenet" - IP address 192.168.20.220
Steps:
Under "Configuration -> Networking" in ESXi 5.5
Setting up separate vSwitch for storage traffic:
1. Click Add Networking.
2. Choose "Virtual Machine" type
3. Choose "Create a vShpere standard switch" and uncheck any "vmnic1" etc. we want "No physical Adapters".
4. Choose network label "storagenet".
5. Choose Finish.
Add VMkernel Port to new vSwitch
(allowing NFS mount via this virtual switch to OmniOS)
6. Make a note of the vSwitch# for new Virtual Machine Port Group.
7. Click Add Networking.
8. Choose "VMKernel"
9. Select "Use vSwitch#" (that you noted)
In the preview it should list the new storagenet.
10. Choose network label "VMKernel-storagenet".
11. Next.
12. Use the following IP settings:
IP: 192.168.20.220
(this will be the IP you must authorize in NFS share)
Subnet mask: 255.255.255.0
13. Next, Finish.
Edit MTU settings in vSwitch and VMkernel to accommodate Jumbo Frames
14. Choose "Properties" next to storagenet vswitch#
15. Click on vSwitch on left view of properties and choose "Edit"
16. Change MTU to 9000 and click OK.
17. Click on VMKernel-storagenet on left view and choose "Edit"
18. Change MTU to 9000 and click OK.
(now your virtual switches will pass MTU 9000 packets correctly)
Add new vmxnet3 network card to OmniOS VM.
19. Edit Settings on OmniOS VM
20. "Add" -> "Ethernet Adapter"
21. Choose type: vmxnet3, Network Label: storagenet
22. Click Next, Finish.
Setup IP address in OmniOS VM
(this assumes you already have a setup vmxnet3s network adapter)
23. Create interface if needed
# ipadm create-if vmxnet3s1
24. Setup interface ip address
# ipadm create-addr -T static -a 192.168.20.15/24 vmxnet3s1/v4
Show adapters
# dladm show-link
LINK CLASS MTU STATE BRIDGE OVER
vmxnet3s0 phys 1500 up -- -- (management interface 1 GB physical)
vmxnet3s1 phys 1500 unknown -- -- (Storage interface 10G virtual)
Check the speed and state
#dladm show-phys
25. Configure Jumbo Frames in OmniOS (refer to other thread)
See:
https://forums.servethehome.com/ind...le-and-napp-it-vmxnet3-and-jumbo-frames.2853/
Note: Different MTU settings for file sharing/regular traffic and virtual storage network.
If you would like the original adapter vmxnet3s0 to stay as MTU 1500,
and the new vmxnet3s1 to be MTU 9000 perform the JumboFrame setups in 25, and afterwards:
****************************
*** Allow one VMXNet3 adapter to be MTU 1500,
*** otherwise will default to max 9000,
*** this setting will persist across reboots.
****************************
Change regular file sharing MTU so packets don't have to be re-fragmented.
# ipadm set-ifprop -p mtu=1500 -m ipv4 vmxnet3s0
26. Ensure ZFS sharenfs security is set to allow new ip address of storage vmkernel:
You can use : to allow multiple hosts:
# zfs set sharenfs=rw=192.168.1.220:192.168.20.220,root=192.168.1.220:192.168.20.220 pool/dstore
or just one host (allowing new esxi vmkernel storage ip address to connect):
# zfs set sharenfs=rw=192.168.20.220,root=192.168.20.220 pool/dstore
------------------------------------
Unmount any NFS shares from old IP address in ESXi
27. Remove from inventory any virtual machines running off old NFS storage
28. Remove NFS datastores shared off old NFS storage ip address (and switch).
Add NFS shares to ESXi from OmniOS using new storagenet vSwitch
29. Click on "Configuration" - "Storage" - "Add Storage"
30. Choose Network File System, "next"
31. Enter: Server: 192.168.20.15 (new vmxnet interface we added for OmniOS)
Folder: /pool/dstore
Datastore Name: VSAN-pool-dstore
32. Profit from fast NFS storage!