Z690 do-it-all box

[colinb]

Building out a do-it-all box for home, would appreciate some advice/thoughts on the below. Mainly, how can I squeeze a QSFP 40G (or at least SFP 10G) NIC in there and not be crippled by bandwidth bottlenecking at PCIe/IO stage?!

Edit: Uses to be VMs (Nextcloud, Prometheus, InfluxDB, Grafana, Win/Mac/Linux playgrounds etc.), NVR for local CCTV (Frigate), Audio server (Room), media server (Jellyfin). There is a separate "deep freeze" system for archiving, so only media "on deck" will be on this system

Edit: Final system shown below, strikethroughs show original setup
Used
ASRock Z690 Pro RS w/ i7-13600 + 4x16Gb 32GB DDR4
m.2-1 4.0 x4 - NVMe SSD (SSD pool)
m.2-2 3.0 x4 - NVMe SSD (Boot + VM images)
m.2-3 4.0 x4 - NVMe SSD (mirror m.2-1)
PCIe 5.0 x16 - 4x NVMe SSD PCIe switch card in ZFS mirrors (VM boot and data disks, torrents & Usenet staging whilst downloading/seeding) - oversubscribe if not saturating bandwidth
PCIe 4.0 x4 - PERC H330 (8x SSD/HDD in ZFS mirrors)
PCIe 3.0 x4 - DC P3600 1.6TB SLOG Mellanox ConnectX-3Pro MCX314A-BCCT Dual QSFP NIC
PCIe 3.0 x1 - Coral Dual TPU on switched adapter card
Unused
PCIe 3.0 x1
m.2-E key
8x SATA
2 x USB 2.0 Headers (Support 4 USB 2.0 ports)
2 x USB 3.2 Gen1 Headers (Support 4 USB 3.2 Gen1 ports)
1 x Front Panel Type C USB 3.2 Gen2x2 Header (20 Gb/s)
2 x USB 3.2 Gen2 Ports (10 Gb/s) (ReDriver) (Rear)
2 x USB 3.2 Gen1 Ports
2 x USB 2.0
1 x RJ-45 LAN Port (2.5GbE)

 

[colinb]

Ok, so refining the question a bit. After some reshuffling, I have one PCIe 3 x4 slot, and two PCIe 3 x1 slots available. Into these, I would like to fit a DC P3600 SLOG, and a 10 or 40Gb/s NIC.

For the uses described above, does it make more sense to compromise the SLOG on a x1 slot, or the NIC? Having never used a SLOG before I have no idea at all what sort of IO it needs to be worthwhile.

 

[colinb]

Ok, worked it through.

Ditching the SLOG altogether, moving to async writes, avoiding NFS shares, and maxing out RAM to use ARC and/or RAMDISK instead.

Risk is that I lose the last 10s or so of data written in the event of a power out - but those are very rare here plus there's a UPS online so that risk is minimal. I can probably live with th dataloss if it occurs - and tbh, power out here only happens in big storms, during whic I doubt I will be writing lots of data!!