You actually only get to use 64k jumbo frames with IB when you're using the protocol natively, otherwise you're limited to 4096(having a hard time remember if it's 4096 or 9000)when in IP mode, which is a shame. There's absolutely no way I've found in Windows to increase the jumbo frame MTU beyond the 4096/9000 value at the switch or NIC level.
Infiniband setup. Advice needed.
- Thread starter grandmasterneil
- Start date
-
- Tags
- infiniband mellanox
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Robert, you have it right. It may have been a misnomer to say intra-office as opposed to inter-office. The offices are separate but are within reach of fiber.
Each workstation will be processing the same source 4k footage. Each 4k raw frame averages 5MB @ 30 frames per second. In order to properly stream on an edit bay, I need to maximize bandwidth. It probably will not be realistic to be able to stream more than one station simultaneously, let alone to multiple stations simultaneously but during a render, each node can operate on a frame and then send it back to storage- a la render farm. I'm trying to remove the network from being a bottleneck.
Each workstation will be processing the same source 4k footage. Each 4k raw frame averages 5MB @ 30 frames per second. In order to properly stream on an edit bay, I need to maximize bandwidth. It probably will not be realistic to be able to stream more than one station simultaneously, let alone to multiple stations simultaneously but during a render, each node can operate on a frame and then send it back to storage- a la render farm. I'm trying to remove the network from being a bottleneck.
I'm not 100% sure on how IB actually works when you're not using it in IP mode. My understanding is when you're using infiniband natively it becomes application dependent, no? Interestingly, the ScaleIO system requirements list an IBoIP network as one of their recommended requirements.
Anyway, Samba is a unix-side thing, not Windows, it provides interoperability between a Unix client and Windows CIFS/SMB shares, but doesn't necessarily provide support for all the same "bonus" protocols and technologies inherent in tech like SMB.
I think to still connect to Windows shares you need some kind of addressing protocol like IPv4. I could be totally wrong that though and would love for someone else to chime in.
Anyway, Samba is a unix-side thing, not Windows, it provides interoperability between a Unix client and Windows CIFS/SMB shares, but doesn't necessarily provide support for all the same "bonus" protocols and technologies inherent in tech like SMB.
I think to still connect to Windows shares you need some kind of addressing protocol like IPv4. I could be totally wrong that though and would love for someone else to chime in.
I believe you are correct. As far as I know just about every nfs implementation supports ib/rdma/NFS.
Did a quick google and reading stuff like smb2.2 (cifs) screams over infiniband so both sides are doable. Also smb 3, and smb direct...
Did a quick google and reading stuff like smb2.2 (cifs) screams over infiniband so both sides are doable. Also smb 3, and smb direct...
Found a good article on mellanox
HowTo Configure NFS over RDMA (RoCE)
HowTo Configure NFS over RDMA (RoCE) | Mellanox Interconnect Community
under centos 7.0
with nice references and performance numbers.
and on the MSFT side
Microsoft Based Solutions
Accelerating Remote Storage Access - Breakthrough Performance, Scaling and Resiliency
http://www.mellanox.com/page/microsoft_based_solutions
HowTo Configure NFS over RDMA (RoCE)
HowTo Configure NFS over RDMA (RoCE) | Mellanox Interconnect Community
under centos 7.0
with nice references and performance numbers.
and on the MSFT side
Microsoft Based Solutions
Accelerating Remote Storage Access - Breakthrough Performance, Scaling and Resiliency
http://www.mellanox.com/page/microsoft_based_solutions
Might also be interesting to see if anyone has implemented SMB Direct/SMB over RDMA: Improve Performance of a File Server with SMB Direct
That's the route I was thinking once I had the network up.
That's the route I was thinking once I had the network up.
SMB on both sides is less complicated. I haven't dug through current windows servers. I have been looking primarily at CentOS for my compute as intel is primarily supporting the xeon phi and associated mp libraries on the redhat side. Work'ish on windows but real suppport is on the ix side. I am only planning windows vms for games and software that wont run on *ix (and even then I will be trying for wine in most cases).
I don't believe there is a single video editing product other than maybe obs that provides full feature editing on the linux platforms and I haven't ever researched rendering engines.
I don't believe there is a single video editing product other than maybe obs that provides full feature editing on the linux platforms and I haven't ever researched rendering engines.
DaVinci Resolve Studio is largely known as a color grading tool but it also functions as a NLE and is available for Linux.
-edit- I mistakenly said the free version was available for Linux.
A little overboard for my backgammon videos and user/client walkthroughs 
. Will have a look. For my occasional video, obs has had an extremely low learning curve. Screen videos, voiceovers, transitions, chapter cards, talking head... No B-Roll, 4k 60fps, multi-camera, true colour required here. If I remember to shave I consider my videos a success.
. Will have a look. For my occasional video, obs has had an extremely low learning curve. Screen videos, voiceovers, transitions, chapter cards, talking head... No B-Roll, 4k 60fps, multi-camera, true colour required here. If I remember to shave I consider my videos a success.Just giving you options.
Back to IB, has anyone rolled their own cables? I know the transceivers can be cost prohibitive but I'm considering these: Mellanox SSF-8436 QSFP+ * MC2207310 * InfiniBand * Optical Transceiver * Emcore
Just need to know what kind of fiber I need to run and if it's possible to run to a wall plate and into a patch panel- just to keep things clean. If not, I don't really mind just running really long cables.
Back to IB, has anyone rolled their own cables? I know the transceivers can be cost prohibitive but I'm considering these: Mellanox SSF-8436 QSFP+ * MC2207310 * InfiniBand * Optical Transceiver * Emcore
Just need to know what kind of fiber I need to run and if it's possible to run to a wall plate and into a patch panel- just to keep things clean. If not, I don't really mind just running really long cables.
I was just wondering about rendering on the xeon phi's and found the embree "ray tracing engine" was ported to the phi and autodesk maya uses embree. Looks like a bit of fun when I'm not studying openmp, data mining and machine learning.
If you're going to attempt to "roll your own" IB cables, look into cisco bidi 40G transceivers for up to 100m, seen them new go for under $100. OM3/4 rated LC-LC cables are waaaaaaaaaaaaay cheaper than that insane MPO 12 strand stuff.
Honestly though, looking for deals on >5m QDR/FDR active optical cables is cheaper (and faster for FDR). Use passive copper cables for anything <5m.
Honestly though, looking for deals on >5m QDR/FDR active optical cables is cheaper (and faster for FDR). Use passive copper cables for anything <5m.
I had linked to this earlier but the link highlighting sucks.
Custom & OEM 40G QSFP+ Tranceiver Optics | FS.COM
Buy 40GBASE-SR4 QSFP+ Transceiver Module for MMF | Fiberstore
Lots of options there for ends.
Custom & OEM 40G QSFP+ Tranceiver Optics | FS.COM
Buy 40GBASE-SR4 QSFP+ Transceiver Module for MMF | Fiberstore
Lots of options there for ends.
Thanks for the suggestions on cable ends. What type of cable do I need to use? I think QSFP+ uses 4 bi-directional signals but I'm uncertain of the wavelength or even if that's correct.
Thanks in advance for the info!
Thanks in advance for the info!
Don't buy those, as it says: "Good Condition. Please note that the cables have been cut." I'm not sure how cut cables can be considered good condition, but unless you know how to splice fibers back together you won't be getting very far.Just giving you options.
Back to IB, has anyone rolled their own cables? I know the transceivers can be cost prohibitive but I'm considering these: Mellanox SSF-8436 QSFP+ * MC2207310 * InfiniBand * Optical Transceiver * Emcore
Just need to know what kind of fiber I need to run and if it's possible to run to a wall plate and into a patch panel- just to keep things clean. If not, I don't really mind just running really long cables.
But using the QSFP transceivers that @Chuckleb linked you could do runs about as long as you want, through patch-panels, etc. For a short-range transceiver you will need MultiMode fiber (OM3 or OM4 should be good to 100m or so), or if you need to go long range then you need a long-range transceiver and single-mode fiber - single-mode will get you quite a few km if you need.
The optics on those transceivers use MPO connectors, I'm guessing 8 strands. You can get MPO patch panels, or you can get all variety of things to break out the strands inside the MPO cable (and put them back together in the same order on the other end) to turn eg. an 8-strand MPO into 4 duplex-LC connectors, or whatever else you might need. You will want to do some research on the subject - there is some signal loss on every connector the optical signal goes through and you have to stay within specs. There are also some issues with single-mode connectors and patch-panels, but I forget the details about that.
If you look at the links I have mentioned they list the type of cable and connector you need to use. There is a table in there with cables, wavelengths, and other information. This is going to cost quite a bit... If you can use premade, it will be cheaper. For example, you need two ends, that's $200 already without fiber, etc.