Hi folks. I wanted to take the time to do a write up for the infiniband switch I got. I'm aware there is official documentation out there for the product, however, I wanted to walk through the processes as I receive my hardware and set everything up to maybe hopefully help others who choose to implement infiniband at work or at home and maybe to help those who purchase the same switch and are confused on how to set it up or configure it.
I would prefer to span some of the different sections into their own posts so I'll reserve the next 3 posts for that and will update each accordingly and delete un-used posts.
First things first: the switch.
The switch I'll be using for this is specifically the MIS5030Q-1SRC, otherwise known as the InfiniScale IV IS5030.
The rear of the unit houses the power supplies(max of 2)and the fan unit. My switch only came with one power supply. Both objects are easily hot swappable and removable.
The fan unit:
The PSU:
After receiving the switch the first thing I did was power on and connect my RJ45 to RS232 cable. I found these 1 dollar cisco cables on ebay and bought 2 in case one was busted, not bad for 2 bucks and the first one I tried worked perfectly.
New Cisco Console Cable RJ45 to DB9 Cable Switch Router SHIP Today 744664241835 | eBay
When you first initially get a switch that has either been defaulted to factory settings or is brand new the Mellanox documentation states you must use the console port for the setup before you can use the "Mgmt" port. The Mgmt port is how you connect to the web interface of the switch and manage it.
Since I did this on a Windows machine I found HyperTerminal immensely useful and efficient for connecting to the switch's serial interface. I don't remember at what exact point in Microsoft's OS lineup they began removing HyperTerminal from the add/remove features but luckily I found a standalone download somewhere on the net that works just fine in Server 2012 R2 DataCenter.
Dropbox - HyperTerminal.zip
Connecting to the interface is very simple. If you've never used any kind of COM1 interface before just use these settings on the initial popup you get when HyperTerminal opens:
After you click apply and okay, after a few seconds the switch will output information into the console asking if you want to run the wizard for the initial configuration. Type 'y' here. **If you get a login prompt and suspect the switch was reset by the seller prior to shipment the default login is username admin, password admin.
Additionally, it is possible after all that some sellers do not reset switches. They simply may sell "working/tested pulls". If you need to reset the switch there is a very small paper-clip-like hole on the front-left housing of the switch. This reset switch performs two types of resets:
- A quick, 1 second press of this reset switch brings all ports down and brings them back up
- Holding the reset switch for 15 seconds the entire switch restarts and the login password is deleted. You will then be able to enter without a password and set a new one*.
*Based on Mellanox's documentation it is unclear is this procedure completely resets all of the settings of the managed switch software itself. Further testing is needed.
The following information is taken directly from the IS5030 installation guide and serves to explain all of the possible prompts and outcomes you get when configuring the switch via the wizard.
Wizard Prompts:
Other setup configs:
a) zeroconf config -
b) static ip config -
Once your switch is properly configured you should be able to access the web management interface from any browser using the IP it was assigned.
The IS5030 and IS5035 switches are both internally managed in that I believe this means the switch can act as a subnet manager on its own instead of relying on the user to individually initialize OpenSM nodes on clients to hand out IPs to infiniband interfaces. I'll confirm this once I have more of my equipment.
In most of, if not all cases, the 5030/5035 switches utilize FabricIT for their internally managed software interface. No additional software needs to be installed on a client side, however, I believe it is possible that FabricIT is a premium software offering from Mellanox and may require licensing when utilized separately or with switches that don't natively include FabricIT. In the midst of my reading this switches documentation it is stated that the switch includes a license to this internally managed switch firmware/OS.
It is true that it does include a license;it is both embedded into the firmware and it is physically located underneath the informative pull out tab located under the USB port(see the above pics)in case you needed to reinstall the key or need to reflash FabricIT for whatever reason.
This is the default login page you get when accessing the web interface:
After logging in you get a quick summary page with quite a decent amount of options both up top and to the side.
The summary views for things like the temperature and fans are actually quite informative and nice to look at.
Surprisingly there are a lot of different settings you can change. Frankly I didn't expect to get this level of configuration.
Port Configuration:
Now this next bit is interesting. If you haven't already noticed, upon initial configuration of the switch the subnet manager is not actually set to run by default and this is made clear by the message in the top right hand corner stating "subnet manager is not running". Under Fabric MGMT is where your options for the subnet manager reside.
I'm not exactly clear at this point on if the settings are optimal but to me it reads as such: enable the subnet manager on this node(dv-is5030), set the SM priority for this node to be the highest(meaning when other nodes are present this particular unit will handle subnetting and then trickle down through the priority list on hardware failures).
The subnet settings get even more advanced under "Advanced"(go figure) and "Expert".
This pretty much wraps up where I'm at right now. Shout out in a reply if you want to see a specific image from one of the configuration pages in the web interface and I'd be happy to help out. Or if you want me to check for anything specific(such as a feature).
The next steps in my project will be outlined in the following reserved posts. I'm waiting on my QSFP cables and ConnectX-2 VPI cards to come in and then I'll be installing them and configuring them in a Server 2012 R2 environment.
I would prefer to span some of the different sections into their own posts so I'll reserve the next 3 posts for that and will update each accordingly and delete un-used posts.
First things first: the switch.
The switch I'll be using for this is specifically the MIS5030Q-1SRC, otherwise known as the InfiniScale IV IS5030.
The rear of the unit houses the power supplies(max of 2)and the fan unit. My switch only came with one power supply. Both objects are easily hot swappable and removable.
The fan unit:
The PSU:
After receiving the switch the first thing I did was power on and connect my RJ45 to RS232 cable. I found these 1 dollar cisco cables on ebay and bought 2 in case one was busted, not bad for 2 bucks and the first one I tried worked perfectly.
New Cisco Console Cable RJ45 to DB9 Cable Switch Router SHIP Today 744664241835 | eBay
When you first initially get a switch that has either been defaulted to factory settings or is brand new the Mellanox documentation states you must use the console port for the setup before you can use the "Mgmt" port. The Mgmt port is how you connect to the web interface of the switch and manage it.
Since I did this on a Windows machine I found HyperTerminal immensely useful and efficient for connecting to the switch's serial interface. I don't remember at what exact point in Microsoft's OS lineup they began removing HyperTerminal from the add/remove features but luckily I found a standalone download somewhere on the net that works just fine in Server 2012 R2 DataCenter.
Dropbox - HyperTerminal.zip
Connecting to the interface is very simple. If you've never used any kind of COM1 interface before just use these settings on the initial popup you get when HyperTerminal opens:
After you click apply and okay, after a few seconds the switch will output information into the console asking if you want to run the wizard for the initial configuration. Type 'y' here. **If you get a login prompt and suspect the switch was reset by the seller prior to shipment the default login is username admin, password admin.
Additionally, it is possible after all that some sellers do not reset switches. They simply may sell "working/tested pulls". If you need to reset the switch there is a very small paper-clip-like hole on the front-left housing of the switch. This reset switch performs two types of resets:
- A quick, 1 second press of this reset switch brings all ports down and brings them back up
- Holding the reset switch for 15 seconds the entire switch restarts and the login password is deleted. You will then be able to enter without a password and set a new one*.
*Based on Mellanox's documentation it is unclear is this procedure completely resets all of the settings of the managed switch software itself. Further testing is needed.
The following information is taken directly from the IS5030 installation guide and serves to explain all of the possible prompts and outcomes you get when configuring the switch via the wizard.
Wizard Prompts:
Other setup configs:
a) zeroconf config -
b) static ip config -
Once your switch is properly configured you should be able to access the web management interface from any browser using the IP it was assigned.
The IS5030 and IS5035 switches are both internally managed in that I believe this means the switch can act as a subnet manager on its own instead of relying on the user to individually initialize OpenSM nodes on clients to hand out IPs to infiniband interfaces. I'll confirm this once I have more of my equipment.
In most of, if not all cases, the 5030/5035 switches utilize FabricIT for their internally managed software interface. No additional software needs to be installed on a client side, however, I believe it is possible that FabricIT is a premium software offering from Mellanox and may require licensing when utilized separately or with switches that don't natively include FabricIT. In the midst of my reading this switches documentation it is stated that the switch includes a license to this internally managed switch firmware/OS.
It is true that it does include a license;it is both embedded into the firmware and it is physically located underneath the informative pull out tab located under the USB port(see the above pics)in case you needed to reinstall the key or need to reflash FabricIT for whatever reason.
This is the default login page you get when accessing the web interface:
After logging in you get a quick summary page with quite a decent amount of options both up top and to the side.
The summary views for things like the temperature and fans are actually quite informative and nice to look at.
Surprisingly there are a lot of different settings you can change. Frankly I didn't expect to get this level of configuration.
Port Configuration:
Now this next bit is interesting. If you haven't already noticed, upon initial configuration of the switch the subnet manager is not actually set to run by default and this is made clear by the message in the top right hand corner stating "subnet manager is not running". Under Fabric MGMT is where your options for the subnet manager reside.
I'm not exactly clear at this point on if the settings are optimal but to me it reads as such: enable the subnet manager on this node(dv-is5030), set the SM priority for this node to be the highest(meaning when other nodes are present this particular unit will handle subnetting and then trickle down through the priority list on hardware failures).
The subnet settings get even more advanced under "Advanced"(go figure) and "Expert".
This pretty much wraps up where I'm at right now. Shout out in a reply if you want to see a specific image from one of the configuration pages in the web interface and I'd be happy to help out. Or if you want me to check for anything specific(such as a feature).
The next steps in my project will be outlined in the following reserved posts. I'm waiting on my QSFP cables and ConnectX-2 VPI cards to come in and then I'll be installing them and configuring them in a Server 2012 R2 environment.
Last edited: