I am trying to solve a similar problem. I have dual 7773x processors as well but on the Tyan board S8253.
I recall reading in a review on a similar set of SuperMicro fans that the reviewer made some adjustments in bios settings that improved the fan performance by a lot and lowered the noise as well. Something about auto intelligent fan control in the settings. Not sure if that's helpful as it sounds like you've tried messing with the settings already. I've been looking into using the SNK-P0064AP4 as well but have similarly had my reservations about if it will work or not. The SNK-P0066AP4 is just a Wraith Ripper and is listed at 280W but I've read it doesn't really and is listed as lower, at 250W, elsewhere, so maybe SuperMicro is unreliable for this sort of information. I wouldn't expect that from them, not good.
Noctua teased a new variant of the DH-15 to come out that could cool stuff as high as 400W+ but it looks like it has been delayed based on the product roadmap on the website. That isn't scheduled to come out until 23Q1 though.
Another Dynatron option you can try is the A39. It is rated for up to 280W and 3U+. There is also the T17 which is for 280W and 1U+. And finally, there is also the L29 server cooling solution, which is for 280W and 1U+. Problem is they are all very loud.
I think if you are wanting to control the sound, using a case is going to be a better form factor than using a rackmount chassis. They can be larger and facilitate more airflow arrangements that are quieter. Server aren't meant to be quiet.
For me, after much deliberation, I am going to try using a Meshify 2 XL with 2 of the Arctic Freezer 4U SP3 coolers. It's worth noting that there may still be a difficulty getting these to fit into some 4U cases such as those with a motherboard tray. These are rated for 300W and also are designed with dual socket SP3 configurations in mind.
If that doesn't work I am going to try using AIOs. The Alphacool Eisbaer Pro Aurora 360 was specifically designed for the larger backplates. I am also thinking to maybe try the EK AIO 360 as it seems to have good reviews and I've been told that despite having smaller block coverage it still is the best option AIO wise on something like TRPs or this. One part I am still unsure about for that one is that a TR4/SP3 mounting bracket is needed for it, I was recommended using the Asetek TR4 bracket, but this is not an Asetek based pump, so I am not sure if that will work for it.
All that being said, I believe these Arctic Freezer coolers should work. I have been preferential towards using a server chassis as well but I think it's just not tenable. If sound isn't an issue it can be done, but if it is, it is better to build with a tower case. It seems I should also be able to switch out the fans with NF-F12 Industrial PPC-2000 PWMs, which as far as I can tell give it slightly better performance, but reduce the noise by 25%.
I'll follow up with how it goes once I give it a shot.