Resizable Bar on Asus Dual 621 Sage, success

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Venturi

Active Member
Apr 22, 2016
145
174
43
So, I've had this build since September 2020. It succeeded the prior build that included 4 Titan V's, including the Titan V 32GB CEO Ed.

The updated build uses 2 nVidia 3090 RTX in SLI/nvLink.

Current build:
2x 3090 rtx Founders Edition & SLI bridge
2x 8180M (56/112 cores)
1.5 TB ram
Asus c621 Sage Dual socket motherboard Bios build 9900 series (beta)
4x Raid Samsung 860 pro (4TB each, 16TB total) data and backups
1x Samsung 860 pro (4TB) data and backups
1x Sabrent 8TB nvme for Apps and Games
1x Sabrent 4TB nvme (OS)
1600W digital power supply
Asus PA32UCX-P monitor, updating to Asus PA32UCG
SFF Thermaltake P1 micro ATX case (modified.. heavily)
MS Windows Data Center 2019/2020 for OS (release candidate)
1x pound of flesh and blood

I have been working on several iterations of beta bios and exploring Resizable Bar

In order to test I have also had top use a beta bios for my 3090 RTXs, and a developer driver for the 3090 RTXs.

Thursday there will be a release driver from nvidia that may or may not include the resizable bar and may or may not also included the FE 3090 bios update as well.

On my early testing, I have seen little to no marginal improvements for 80% of the specified apps/games I tried. But did see improvement in two beta patch games and my medical imaging software that can use the feature.

While it is early, I hope that the feature matures, but I don't see it as a magical elixir that improves everything.

While I use my personal rig for researching algorithms in CDSS and medical imaging, it does also blow off steam as a gaming build. With a heavily optimized OS, apps, and build, it has been able to compete well with he highest scores on the proverbial benchmark sites, but I remain as anonymous as I can. Again, it was never built to play games, its purpose is to get me through my dissertation.

On that note, I was able to resurrect SLI/nvlink functionality in Dx11 backwards, and OpenGL. DX12 and Vulkan games have their own baked in multi-GPU support. I use Vivaldi as my browser and in there I have been able to have use the GPUs for most things and it does make use of BOTH gpus. For those wondering where to set that in the browser is: vivaldi://flags A side note, nvidia disabled the 3090 from doing SLI for pre DX12, but I was able to reverse engineer it again.


I will add to this thread as I monkey with my different research efforts , but may also report on the antics of games and benchmarks. This site is not for frivolous things so I will not make it a point to no more than lightly touch on gaming as an assessment function and nothing else.

The build does run super quiet, (inaudible at idle), cool, with temps well below any measure of concern. CPU temp sat idle, hang around 22C -in a house that is 20.5C, under load they may creep up to 48C. the 3090 FE RTXs at idle (even the fans turn themselves off) stay at around 31C and under load get as high as 69C.

here are some pictures of interest.:

RBBios.jpg

RBBios2.jpg

PCside.jpg

PCback.jpg

desk.jpg
 

Stephan

Well-Known Member
Apr 21, 2017
920
697
93
Germany
Two questions: You're the guy who bent the left Noctua like mad in order to not block the first PCIe slot, right? Painting last picture by Carl Spitzweg?
 

Venturi

Active Member
Apr 22, 2016
145
174
43
Yep, that's me, 'cept I had to bend BOTH Noctuas out 18mm

the painting, yes 1872
 

Stephan

Well-Known Member
Apr 21, 2017
920
697
93
Germany
I am baffled by the antithetic nature of this picture of your office.

Your computer has around the same compute power as the #1 Top500 supercomputer from only around 15 years ago. A marvel of 2020s technology. And of course more than capable of ripping off the masks off of nature's many faces to reveal its deeper secrets through computation. Yet only a progenitor of disruptive "things AI" to come to this world, judging by Lex Fridman's recent Jim Keller interview on the matter. Quite scary to some come-early thinkers, because no-one knows what this will mean for humanity, or what entire industries this might eat for lunch soon.

On the other hand a Biedermeier epoch picture, that parallels our stifling 2020s political world like no other. Cancellation of public free speech, surveillance of universities, censorship of the press, firing of liberal/national professors, aristocrats clinging to power, sounds familiar? Had this progress preventing modus operandi not died a well deserved death in the 1848 German revolutions, that machine on the floor would in my opinion not exist.

Anyhow. I think this motherboard is great because there have been many interesting BIOS hacks for it, especially to run all sorts of CPUs and tweak the RAM. Mind sharing how technically you bent the Noctuas without damaging them? Waiting 5+ hours for ungoogled-chromium to compile is getting a bit boring here.
 

Venturi

Active Member
Apr 22, 2016
145
174
43
Stephan,
the Noctua heatsinks involved more than mere bending unfortunately. Moving 18mm required different methodologies. The pipes are easily creases and the wrong type of stress will also flake of the coating.

The measured approach:
The process involved using two wooden rods about 23mm in width inserted in the piping and cupped by the curvature. The base was secured in a vice using two plastic trim tools as bumpers as to not make marks on the heatsink base. The third component was a wooden ruler that fit perfectly in the ascending recess of the thin side of the radiator area. Lastly, have table top with the vice re-orientated to a wall/cabinet/solid surface (or floor as my vice allowed rotation) and have a a 2x4 available that is as long as distance to wall/floor and vice as needed.

Set up:
-wooden rods in place
-base mounted in the vice using plastic bumpers to secure it
-wooden ruler in radiator grove/recess
-2x4 braced between radiator side (with rod in recess) and a vertical solid surface (or floor).

1
Advance the foot of the vice to compress heatsink/radiator towards wall/floor using advancement screw on vice. Doing it slowly, methodically, and almost sneaking up on the process start the shear/bend:
2
Bend/compress in a parallel path the heat pipes and radiator as much as you can keeping radiator square/parallel with base. You will be pushing the base of the radiator more than the top to avoid bending the radiator over.

Result
18mm of "shear" from original centered radiator over heatsink. No crinkles or compression marks on pipes, no compression marks on radiator.


I did the process for for the first heatsink twice because I originally calculated that 12mm would be enough, however on test fitment, the metal clips for the fan were with 2mm from GPU and not within my comfort zone. So the final design was shy of 2cm of travel.

The first, not so measured, approach:
Some brutal honesty - I had 3 heatsinks. The first one was a "test" or "feasibility study" candidate because it had some visual anomalies from the packaging it came in. Using an oak plank on the ground, the same wooden ruler in the recess.... using a rubber mallet - beat the heatsink into submission the desired shear travel - or until all your frustrations from the day are satisfied. The heatsink radiator looked unmarked but had traveled 24mm and was not as parallel to to the heatsink base as I would want. While usable and passable, it looked like crap was below my aesthetic ruler, and thus decided on the more measured approach. Another consideration is that 24mm makes the Noctua heatsink almost touch the motherboard /vrm heatsinks.


An observation:
Did Noctua get it wrong? WTF what were they thinking? Did they design the heatsink to automatically decommission the first pcie slot? Did they assume that was acceptable on a board that promotes its GPU fitment capabilities to render unusable the first pcie slot??

Or did I get it wrong in my pursuit of efficiency and silence as there are three other choices of heatsinks and opted to pick the pain in the ass less noisy solution?

Regardless, the final result seems good....
now if I could only get the same Noctua fans in another colour other than desert camo...
 
Last edited:

bogdi1988

New Member
Feb 25, 2021
2
0
1
I have been working on several iterations of beta bios and exploring Resizable Bar

In order to test I have also had top use a beta bios for my 3090 RTXs, and a developer driver for the 3090 RTXs.

Thursday there will be a release driver from nvidia that may or may not include the resizable bar and may or may not also included the FE 3090 bios update as well.
Venturi, can you run nvidia-smi -q and do a screenshot of what the BAR1 size is set to? And curious how would one acquire such bios :)
 

Aardvark0

New Member
Jun 27, 2018
15
6
3
@Venturi
Do you not find that multiple numa nodes with a dual socket board has a significant negative effect on gpu compute? I switched to trx40 because of that, at least for vfx, similar setup 2x 3090, now with a 3970x.

With resizable bar enabled, do you know the appropriate setting for Above 4G limit (40bit-48bit) is for nvidia 30-series cards? Intel bios may also call this MMIOH Base and MMIO High Size. There is a little bit of info on this on the radeon open compute page, but can't find any info on how nvidia supports BAR.

With Server 2019 SAC not including desktop experience, is the only way to get Server 2019 with desktop at an equivalent or later version than Win 10 20H2 - is install server 2022 preview? I'm after server 2019 with desktop which has hardware accelerated GPU scheduling.

Thanks
 
  • Like
Reactions: nnunn

Venturi

Active Member
Apr 22, 2016
145
174
43
HI,

Aardvark, when I get back from my trip I will look at the settings for MMIOH and MMIO. I believe the latter is set at 1024/and 56T, but I'll verify. I have not seen a negative effect on GPU compute on a dual socket board, I had previously used 4 titan V's in a single socket and 4 Titan Vs in a dual socket and found the dual socket to be the better performer on / for GPU. Although admittedly I only tested that function on dual 8180m chips on the current sage 621 dual. Benchmarking showed a clear lead at the card level on the dual.
Before this get's over analyzed, this was with 4 x Titan V and 4x full bandwidth pcie. There are arguments in favor of resources, lanes , etc.
A more reasonable and logical test would have been one Titan V in each configuration for Gpu compute, but I never tried that. That would be interesting.

I use server 2019 data Center with desktop experience and Ubuntu. I spent some time limiting MS intrusion. telemetry. and optimizations and now it runs as a workstation. I am glad to answer any questions on that set up. When in Windows I have only used the past iterations of data center. However that is also because I can't have the same control over win 10 home, pro, or enterprise. Otherwise, I'm in Ubuntu


Bogdi1988, when I get back I'll run nvidia-smi -q and let you know. Doing part of my residency at the current moment.

Thank you

J
 

domrockt

New Member
Apr 18, 2020
9
0
1
36
Germany
@Venturi
Hey Hi , may i ask where do you get that lot of Modded/Beta BIOS?
I own an Asus c621e sage with two ES Chips 26cores and have no clue where to Look at for more diverse BIOS Options.

My Specs for good measure

Asus sage c621e 128gb Samsung DDR4 @ 2990mhz
2*26Core Intel Platinum ES
3090 RTX EVGA
R9 290 Gigabyte
Unraid 6.9
Windows10/Workstation wife/myne
A bunch of SAS Drives and nvmes for Cache and an Intel DC3600 as Windows Boot Drive.
 

Venturi

Active Member
Apr 22, 2016
145
174
43
@Venturi
Hey Hi , may i ask where do you get that lot of Modded/Beta BIOS?
I own an Asus c621e sage with two ES Chips 26cores and have no clue where to Look at for more diverse BIOS Options.

My Specs for good measure

Asus sage c621e 128gb Samsung DDR4 @ 2990mhz
2*26Core Intel Platinum ES
3090 RTX EVGA
R9 290 Gigabyte
Unraid 6.9
Windows10/Workstation wife/myne
A bunch of SAS Drives and nvmes for Cache and an Intel DC3600 as Windows Boot Drive.
domrockt.
the bios is an internal bios from asus, its not downloadable anywhere.
it isn't meant for distribution yet and experimental.

Please PM ,me for further assistance on this
 

Venturi

Active Member
Apr 22, 2016
145
174
43
Yes sir,
its the same version firmware that I already had, feel batter about it as I guess it didn't require any revisions, but performance is still a lackluster improvement.