Thank you the reply. I like reading through such details. It gets my mind going about interesting things to look into. It sounds like you want to get into the driver/firmware area and even play around with a software defined element in the loop that gives you sway over how things get processed and moved around at the protocol level ... i.e : implement your own protocol. I'd love to know if there are cost effective ways to do this and am surprised this isn't a bigger (thing). This made me think of SDN (software defined networking) and then I decided to search for : Software Defined NIC ...X2 are not supported except by Forum and limited but useful downloads, the real trick is avoid 3rd party X2 and X3 they must be Mell genuine. Otherwise its hit or miss .. which binary is overloading the timeslots ? oh its completely fair scheduler
Besides, the firmware bugs are obvious real issue that no one here has much experience with.. and then Dell / HP / EMC / IBM all return nightmare issues that are all one off thus useless , as binary drivers differ with OS / hardware combo. Thus advice is useless unless u clone that exact topology .. almost impossible
Best price on new genuine Mell ConnectX 3 is $80 (yet my issue is its not RoCE (2) which is UDP IB connectionless or stateless (yet there are pseudo paths poss) the main thing is its TCP buffer free meaning its unreliable real time (not as bad as it sounds) the other part of this is the Ether encapsulation ( oCE ) i like the RDMA part of it, but dont need the overhead of Ethenet encap.. Im not sending it remotely, and certainly when i am , the 4 packets are going to a mobile client that cant deal with RoCE , besides RoCE cant send 4 packets , Ether cant send 4 packets , more like 300 just to get the ball rolling..
RoCE is designed for distributed datacenters . databases like Hadoop & Elastic Search all the statistics are on hundreds of remote servers that need secure Ethernet access (TCP) or https// banks for example.. Why would any private researcher need this unless dealing with the genome pool or banking.. both of these planned, a gene transplant lab in Costa Rica and deposit on a First bank in Durham City
My interest is in cutting apart the CE and exposing the UDP IB part for internal DB packet storage, over IB .. I was looking at FPGA boards to so this, like the X4 060 at $2800 ea but found quad port cards with KU115 PGA fabric .. once a ser-vice takes off its great to have a forward plan to expand past one datacenter.
Whereupon, it appears something just fell out the back of the mellanox truck :
Hot off the press (4 hours old)
Mellanox Announces BlueField Software-Defined SmartNIC Adaptors | StorageReview.com - Storage Reviews
I'm sure slapping a beefy soc as well as ddr is going to make this a super pricey card but it seems like its what you're after. Not sure what you can do in terms of topology by which you do the main manipulation on one node and then send more simple/protocol standard packets downstream but I get your drift. Might be worth some time to look into software defined NICs and see what kind of magic you can whip up on your power house nodes while keeping your downstream nodes on something like connectx-2 using standard protocols. I am seemingly looking at a similar situation for myself.