Dual GPU Pass-thru LAN Box

[Boris]

Build’s Name: Summer PC
Operating System/ Storage Platform: ESXi 6.0, 2x Windows 8.1, pfSense
CPU: Intel Xeon E5-1620 v2
Motherboard: AsrockRack EPC602D8A
Chassis: SilverStone Grandia GD10
Drives: 2x WD Xe 600Gb WD6001BKHG, 4x OCZ Agility 2 90Gb, 2x WD Se 2Tb WD2000F9YZ
RAM: 8x 4Gb ECC REG Parity Kingston KVR1333D3D8R9S/4G
Add-in Cards: LSI 9260CV-4I, TP-Link TL-WDN4800, 2x Radeon 6950, SilverStone SST-EC04 USB 3.0 Controller
Power Supply: Seasonic SS-1000XP
Other Bits: Probably will go into custom water cooling loop

Usage Profile: Standalone ESXi host with two GPU passed into two different VM for entertainment purposes.

Other information: I decide to start my own thread and do not litter in other people threads.

A bit more about usage profile: Every summer for last years i invite few my old friends to join me at vacation, i rent a house for few weeks for it. I have no real vacation for myself (my choice), so i am doing my job (wholesales) at vacation. All previous years i take my 17" notebook with me, but last time i decide to bring something more poweful (I'm going on vacation by car, so i am not really limited with items i can take with me). I will take two displays and this Summer PC with me so will be able to play something better than tablet games, and probably play something online together. All houses i rent before have WiFi, but router always placed in bad place, so i have to drop patch-cord thru whole house, or try to connect over WiFi. ESXi does not support any WiFi adapter and i decide to try pfSense VM with WiFi card passed into. Sirs from other thread told me it's possible to setup pfSense proper way to share WiFi internet over vitrual machines. I have NO expirience with pfSense, but i have two months before my vacation, so i will try at least. If i will have no luck with it - i just buy wireless AP for easy way (but one more box, few more wires, few more mess).

A bit more about hardware: As you may see many hardware is old. Yes, this PC will be build with a lot of old hardware i has. Like graphic cards (one 6950 i has before and second i just just bought from auction to use crossfire outside of summer vacation. This computer located in my living room and sometimes i play something at TV with xbox gamepad). I think at autumn i will upgrade my and my wife computers to 390X's, so my old 290X's will take it place in Summer PC, but at first i want to check how it work.

A bit more about cooling: As i wrote in header - i think to go custom water cooling loop. Today i passed some benchmarks to compare it with results i will get after pass GPU to VM. And pair of that 6950 is loud as hell. I imagine... vacation... evening... i start game, my friend start game... people around us begins to bleed from the ears... I am custom watercooling fan, i have some spare parts left from my previous builds. Like 2x universal GPU blocks, 480mm radiator, reservoir, pack of fittings, tubes, coolant. Missed only Narrow-ILM waterblock and pump and some more fittings i do not have, but will need for that build. Also i bought Koolance 280mm shroud and at next week i will find someone to cut holes at case top to install it. I will have lack of cooling for area around CPU, LSI controller chip and videocard memory and VRM, because my GPU blocks is not "full cover".

A bit more about possible future of build: If i get good impressions with such VM i will probably build much better "Summer PC". Once i saw that small lady EPC612D4U-8R i decide what it's pretty good for more compact "Summer PC" build.
But some things i have to worry about:

1. Need two SINGLE SLOT graphic cards, which hard to get (it must be powerful as well, not just single slot which not hard to find). Sure it must be single slot with fullcover waterblock. But most cards at this moment have one or two of DVI ports and it's ruin single slot possiblities.
2. Do i get enough USB ports to pass it for two VM to have enough ports to attach two keyboards, two mouses.
3. I will have only one x8 slot left, how to use it? PCIe SSD? WiFi card? Raid card? USB card?

Also i imagine single computer for me and my wife with two GPU for two VM again. But it's really may add me more troubles than fetish

UPDATE: And yes, i forgot to write. I have some expirience with GPU pass-thru for gaming in close past. This computer contain ESXi 6.0 and one VM with single 6950. Game work smooth and gamepad as well. But it was not so modern game, so i will try to test it with GTA 5 and something else. Sure 6950 is old, but at 1080p with default settings GTA 5 running at ~30fps on bare metal. I also test it with 3DMark11 and Metro Last Light benchmark. To compare later. I will try to bench something else next days.

PS Sorry for wall of text and my English which probably hard to read, it's not my native language and I'm a bad student.

 

[Boris]

Reserved for possible pictures.

 

[T_Minus]

Now this sounds like a fun build!!!

 

[legopc]

Looking forward to pictures!!! I tried this once before with my own rig and a 4870, I had a lot of fun trying to get that working... But have you thought about audio? You could try passing through the onboard controller to one vm, but then you still have 1 vm without audio.

 

[Boris]

I will have no problems with audio. Because it goes via HDMI and display have internal speakers.

I just notice today what my 6950 have different PCB design and GPU location a bit different, so i unable to connect both cards with such link:

 

[T_Minus]

Not 100% but you may be able to get audio over USB too. ANd USB over IP. I plan to test

 

[Boris]

Not 100% but you may be able to get audio over USB too. ANd USB over IP. I plan to test

I was able to pass-thru audio over HDMI. No problem at all.

 

[T_Minus]

I was able to pass-thru audio over HDMI. No problem at all.

That's what I"m doing too, I was just curious if audio passed over the USB over IP worked too, I don't see why it wouldn't since it's driver based for that OS.

 

[T_Minus]

Why would you keep a 6950 when you can get a 280x for <$200 NEW now days...

My 270x I paid $95 for, and I had at once around 3 all paid <120$... games fine, and can be had.

I started with a 290x when released, sold it when mining brought value up higher than I paid, got 270x games look almost the same on my 30" 2560x1600 screen, no AA the rest are setting rather high and I manage 50s FPS avg in BF4, and TitanFall looks gorgeous and 0 lag or frame drops.

I'm curious how passing through a 270x would work now may have to do that for my family TV HTPC pass-through and see if I could game on that.

 

[Boris]

Why would you keep a 6950 when you can get a 280x for <$200 NEW now days...

I think i will upgrade my and wife's computers with 390X's at autumn. So two my current 290X's become free and i use it for Summer PC. In this case i decide to not invest additional money into graphic cards for Summer PC, but yes, 270X was possible option, but in that case no Crossfire with 6950 like now.

Also i must say my 2x6950 crossfire work bad at this moment. In GTA V i got around 60 fps, but when you moving fast everything moves unevenly. Maybe it's because disk I/O is bad now. My BBU at my OLD LSI 8704eEM2 died few days ago and i experiencing throughput issues. It's why i replace 8704EM2 with less old LSI 9260CV-4I, but physicaly it's not yet replaced.

 

[Boris]

Here is how it looks without PSU wires, but with GPU Link installed. It's amazing, but 100% different cards have almost same GPU location.

While mounting cards in notice what a GPU located not at 100% similar position. Maybe 1mm "left" and 1mm "lower". But i loosen the screws and mount link. Only leak-test will judge.


I already order 3x Narrow ILM mounting plates for waterblocks (1 for that build and 2 for possible future build. Looks like EK WB just discontinue this product, so i catch last 3 i found around).

Rest components i also order today, think we get it all at next week and will continue this build.

PS STH support only 1Mb pictures, at first i try to upload 3Mb without any luck But 1Mb also not bad. I always like hardware forums there you can look at hardware fullscreen.

PS2 And i keep "Reserved pictures post" clear, because it's not final pictures. But i do not think what in the end it will looks awesome.

 

[T_Minus]

Nice setup! very clean, and tightly fit in there I'm getting a bit excited about your build to see what I can do in the future

Do you think the E5-1620 will be enough for more than 1 person to game?

 

[Boris]

Thank you, T_Minus. (i do not know how to link someone name in reply )

Nice setup only before PSU cables come in And it's become a bloody mess. But i will try (and i did it before dissasemble it) to attach cables to chassis bottom. It will not solve everything, but help a bit.

Regarding cores... Can't say. 4 Cores. 2 per person. Most games do not use more than 2 cores i believe. Will see. But i think it will be AT LEAST not so bad. And will be really complicated if i try to replace it with 4960X from my PC, because CPU waterblock have hard connection with mosfet waterblock. i will have to remove both and to do it i will have to remove motherboard from chassis. Or just dry a loop and it's MUCH MORE complicated.

 

[T_Minus]

Thank you, T_Minus. (i do not know how to link someone name in reply )

Nice setup only before PSU cables come in And it's become a bloody mess. But i will try (and i did it before dissasemble it) to attach cables to chassis bottom. It will not solve everything, but help a bit.

Regarding cores... Can't say. 4 Cores. 2 per person. Most games do not use more than 2 cores i believe. Will see. But i think it will be AT LEAST not so bad. And will be really complicated if i try to replace it with 4960X from my PC, because CPU waterblock have hard connection with mosfet waterblock. i will have to remove both and to do it i will have to remove motherboard from chassis. Or just dry a loop and it's MUCH MORE complicated.

You type a username and put @ in front and it auto hyperlinks them and notifies the person they were mentioned iirc.

Sounds reasonable, which games doy ou play most? Or plan to use it for the most?

 

[Boris]

Got it, @T_Minus. Thanks.

Regarding games - i can't say i am greatest gamer of all gamers But you may think so, because i am doing "virtualized gaming PC".
In past i pay much attention to online games, WoW, LoTRO, EVE Online. Now i still play EVE with my multiple accounts and sometimes login into WoW and Elders Scrolls Online. But i decide what online games give you nothing for money and time you spend Much better to play some GTA or Witcher or any "Elders Scrolls Offline" after hard day with work, or just free evening. Or any "city/village builder" game.
But i expect what we will get much more fun at evening if we play WoW Arena or any other PvP game while we sit in same room, or maybe GTA Online (didn't try it yet). But if everything goes good who know, maybe next summer challenge will be 4K gaming! :-D

To be honest i think it's more challenge and fun for me to build it and set up properly and see how it's works. Gaming is secondary. For my job i use my VM at my colocation server. So i just connect to it and not need much horsepower.
It's why i call it "Standalone ESXi host with two GPU passed into two different VM for entertainment purposes." Entertainment, not just gaming!

 

[Atomicslave]

I have a kind of similer setup I am working on, Asus P6TSE with an i7 960 and 2x 6950's its mostly good to go trying to get USB passthrough working for a wireless keyboard and mouse but i am thinking that might not work and i have to passthrough a usb controller. Will post some results next week with some fps benchmarks.

 

[T_Minus]

From my research YES you need to pass a USB Controller.

If I recall the good ones are ~$70 on Amazon. I got one a couple months ago for this purpose and haven't put it to use yet. They have some cheaper ones with less ports too.

Hmm.. I wonder if a "USB HUB" that's "seen" can be passed through if it's ON a USB port on the motherboard? That would be like passing a SATA disk I'd imagine ? Should work in theory or I'm totally wrong LOL!~!

 

[Boris]

I was able to pass on-board USB controller to first VM. And have additional PCI-e x1 controoler for second VM.

UPD: Last watercooling parts was sent to me at Friday, so expect everything here at next week. In addition i did find fullcover waterblock for second 6950 for 15$. Not sure yet, maybe order it just to have

 

[T_Minus]

I was able to pass on-board USB controller to first VM. And have additional PCI-e x1 controoler for second VM.

UPD: Last watercooling parts was sent to me at Friday, so expect everything here at next week. In addition i did find fullcover waterblock for second 6950 for 15$. Not sure yet, maybe order it just to have

Well that's awesome

 

[Boris]

I have small chance to obtain one or two E5-2690 for maybe 500$ each. Used and without warranty. Will see how it goes. If so i will become pretty happy and buy dual head MB for my computer. I think Sandy still good enough.