Network segment for storage only

[animefans]

I have a very small network at home, with ICX7150 as my primary/"core" switch

I would like to setup a separate segment just for storage, and use jumbo frame

this separate segment will be running thru microtik CRS305-1G-4S+IN

The GBE port will connect to my primary network, while the SFP port will be used for storage traffic

What's the best way to setup my networking to accomplish this goal?

Router is opnsense. It's T730 with 4 port NIC. One of the port will be for the Microtik's 1GBE

My network IP is 192.168.1.x, with 3 other vlans : 192.168.20.x, 192.168.50.x, 192.168.60.x

My current setup is
Microtik IP : 192.168.88.1 (default)
OpnSense port for Microtik : 192.168.88.2
OpnSense Gateway setup to point to Microtik IP
OpnSense Static Route : Network Address -> 192.168.88.0/24, Gateway Microtik

When I plugin Microtik to my laptop, I have no problem accessing it (192.168.88.1)
When I plugin Microtik to my lab, I can't ping it
machines in my vlan and opnsense can ping 192.168.88.2 just fine

machine in vlan can't ping 192.168.88.1 (no answer/100% lost)
opnsense ping 192.168.88.1 return

ping: sendto: Invalid argument

That error message seems to indicate missing route

netstat -r on opnsense return the following
some entries redacted for obvious reason

Internet:
Destination        Gateway            Flags     Netif Expire
default            <comcast gateway> UGS        igb0
dns9.quad9.net     <comcast gateway> UGHS       igb0
<wireguard server>         link#13            UH          wg0
<wireguard client>      wg0                US          wg0
<comcast network?>    link#1             U          igb0
<comcast gateway> link#1             UHS         lo0
localhost          link#7             UH          lo0
192.168.1.0/24     link#2             U          igb1
OPNsense           link#2             UHS         lo0
192.168.20.0/24    link#10            U      igb1_vla
OPNsense           link#10            UHS         lo0
192.168.50.0/24    link#11            U      igb1_vla
OPNsense           link#11            UHS         lo0
192.168.60.0/24    link#12            U      igb1_vla
OPNsense           link#12            UHS         lo0
192.168.88.0/24    192.168.88.1       UGS        igb2
192.168.88.2       link#3             UHS         lo0
resolver2.opendns. <comcast gateway> UGHS       igb0
resolver1.level3.n <comcast gateway> UGHS       igb0

Is this a configuration issue, or static route is not the way to go?

For those host that'll be using the storage network, it will always have a "leg"/port into my core network, so storage segment has no need to access internet
In fact, I will be giving them totally separate IP (10.10.30.x)
I simply want all the machines in storage segment to talk to each other thru SFP/jumbo frame

 

[coxhaus]

The way I would do this in the Cisco small business world is to use a Cisco small business layer 3 switch. Create VLANs normally and turn on interVLAN routing on the L3 switch. Then create an ACL to limit access on the storage segment VLAN. It is pretty simple. I have no idea using Microtik but the concepts should work the same.

Skip jumbo frame, more trouble than it is worth.

 

[Blinky 42]

If you have dedicated storage network interfaces on your hosts participating in the storage network, just use the 10.10.30.x/24 block for the storage network and directly plug them into the Mikrotik. Since all the hosts will have other network interface(s) that are used for non-storage traffic just make sure the mounts reference hosts on the 10.10.30.0/24 block and then configure it for 9k MTU on the interfaces and the 4 SFP+ ports on the switch. The Mikrotik doesn't need to be connected to your core network at all once it is setup since it is just doing storage traffic.

I would probably place all the SFP+ ports on the switch in a new vlan and make that the only (untagged / native) vlan on those ports so you can plug in the 1G copper into your main network and not have the storage traffic leak into the main network. Set the mikrotik's management IP to something useful in your normal network range and call it a day.

I did something similar at home with the 10/40G traffic on fiber & DAC's on a dedicated network block and just make all the SMB/NFS mounts refer to the servers using those IPs.