Search results

  1. T

    ASRock Rack TRX40D8-2N2T Bringing AMD Threadripper to Servers

    Ah the orientation of the socket bit me in the ass. I was used to ordering them with the Noctua NH-U12S TR4-SP3 cooler, and mounting two Delta 11k RPM fans to it, but they assume the workstation airflow direction. Now it's pulling heat from ~1cm clearance to the GPU, and pushing heat into 1cm...
  2. T

    ASRock Rack TRX40D8-2N2T Bringing AMD Threadripper to Servers

    I ordered two systems with this board for our rendercluster, we already have 4 systems based around the Asrock TRX40 Creator board, all with 4 GPU's. This board will be a nice upgrade.
  3. T

    Benchmarking read speed without hitting the network

    Oh yeah sorry, the workstation is Windows 10 Pro, the server is Ubuntu Linux running smbd, those are some important details obviously. I was incorrect about the NAS NIC, it's Broadcom not Intel, full specs are here: G221-Z30 (rev. 100) | GPU Servers - GIGABYTE Global we got a BIOS patch to have...
  4. T

    Benchmarking read speed without hitting the network

    Workstation: 1 NVMe 2TB drive, 10gbit NIC (Aquantia AQC107) NAS: 8 SATA 2TB SSD's in RAID 5 2x10gbit NIC (Intel) EPYC 7282 with 32GB ram Network: Unifi 16XG switch has the NAS connected directly on 2 bonded 10gbit ports (so 20gbit in theory), it's connected to a XS508M 10gbit switch, which is...
  5. T

    Gigabyte Shows Marvell ThunderX3 R282 2U Server

    Have these big ARM cpu's shipped to anyone yet? Super curious if anyone's already ported their codebase to ARM and got some sort of benefit. Is it specifically machine learning they're targeting, or do more general purpose sort of things also do well on ARM?
  6. T

    Why aren't ThreadRippers dominating the HPC market?

    Ah sorry so I didn't really make the situation clear. We've been running a cluster of simple workstations. Then we figured out we could add a GPU to each 3900X based system and double the performance by running 2 instances on one machine. Now I got budget to roughly quadruple our throughput...
  7. T

    Why aren't ThreadRippers dominating the HPC market?

    They've got a good old per machine licensing, so not per core. This software is really workstation oriented, even though they have a fully functional network processing system. I'm definitely thinking long and hard, the thing we've already got one of these threadripper systems up and running...
  8. T

    Why aren't ThreadRippers dominating the HPC market?

    I totally forgot, but earlier this morning DDR4 4000 32GB modules came out, which means I can have near 30% faster memory in my ThreadRipper machines. I wonder if that's better than having double the amount of memory channels. Is double the amount of memory channels really just double the...
  9. T

    Why aren't ThreadRippers dominating the HPC market?

    hi @fp64, I'd love to get my hands on the source code, but unfortunately we don't have that luxury. The software is off the shelf, we're just trying to figure out the best hardware to run it on. If we're lucky the software vendor will find out they can do better on server hardware, but as far as...
  10. T

    Why aren't ThreadRippers dominating the HPC market?

    I guess we're maybe just at a sort of weird spot. We're using software that's dealing with some weird concurrency issue, it doesn't scale well past ~8 cores. Possibly lock contention or whatever, but we don't have control over it. The way we deal with it is splitting the datasets, running the...
  11. T

    Why aren't ThreadRippers dominating the HPC market?

    Hey thanks for pitching in @fp64 that's interesting you'd say that. I haven't been able to benchmark an EPYC yet for our software, though I recently did get my hands on a system so I should be able to do that soon. I do know that the Ryzen's outperform both the Xeon's and the i9's on our...
  12. T

    Why aren't ThreadRippers dominating the HPC market?

    Well I'll be damned. Why did they have to go and make that last slot smaller? That board looks absolutely perfect except for that last pcie slot.. So I guess we're not the only ones :p Edit: ohh I just noticed that last slot is open ended so it will definitely fit the 4th card. That's...
  13. T

    Why aren't ThreadRippers dominating the HPC market?

    Yeah but wouldn't a big player just ask ASRock to make them a board? Basically slap an ast2500 on that Trx40 creator board and you're there. If the chips that good, why aren't they making those boards anyway? Is it just that the markets still too small? Companies are just waking up to Epyc Rome...
  14. T

    Why aren't ThreadRippers dominating the HPC market?

    For context: my company is running a small photogrammetry data processing cluster. Photogrammetry (like most data processing) doesn't scale super well horizontally, so high clock speed CPU's dominate. In addition to a fast CPU, photogrammetry also makes use of the GPU at certain steps of the...
  15. T

    Choosing a server/chassis for GPU workload

    We ordered a first test system, didn't end up with a full service seller but just a parts seller that has an assembly service (saving money where we can). They're doing basic tests now to see if it boots and they sent a picture: Looks like it all goes together easily, might even fit a fifth...
  16. T

    Fastest 256GB RAM with the TR 3970X

    Thanks! We're actually running 4 concurrent processes that each need about 64GB. If it turns out they need more, perhaps the hardware will be slightly less effective, but it wouldn't be the end of the world. It might turn out that we'll go the EPYC route for the rest of the nodes if this doesn't...
  17. T

    Fastest 256GB RAM with the TR 3970X

    Hi, at my office we're doing photogrammetry, and it's pretty CPU and RAM intensive. I'm assembling a system based around the AMD 3970X and the ASRock TRX40 Creator. We need (at least) 256GB RAM, so I was looking at the QVL and I noticed that for the corsair modules it says it only supports 2 or...
  18. T

    Choosing a server/chassis for GPU workload

    Ah that makes total sense. I'll have to run the numbers to see if it makes sense at all. I slightly misremembered, the 2U case for the 3970x with water cooling has 3 pci-e slots vertically stacked so only 1 gpu will fit, the 3u version does have support for 2 gpu's, but it is not water cooled...
  19. T

    Choosing a server/chassis for GPU workload

    Cool! GPU's are actually not much of a bottleneck for us, we run multiple instances of a photogrammetry software, and it has very spiky use of the GPU. Unfortunately the spikes usually coincide (because our datasets are usually chunked and then submitted at once) so we need multiple GPU's to...
  20. T

    Choosing a server/chassis for GPU workload

    Thanks for the update, just found this thread because of it. I'm designing a 4GPU system and arrived at the same supermicro case, so good to know it really probably is the most attractive case out there right now. Our software requires a high clock speed CPU in addition to the GPU's, so instead...