NAS SERVER FOR 4K HDR AND HEAVY VFX

[sultan.of.swing]

Which ethernet card you use?

I am using a 100GBE CX-5 in the client PC and a 40GBE CX-4 in the server so the link is connected at 40GBE.

I can't even get windows copy past 1.5GB/s between two local nvme drives that can do double or tripple that in CrystalDiskMark. Do you change a particular setting?

I do not have anything special changed other than setting MTU to 9000 on the required devices.
Client machine is just Windows 11 Pro for Workstation.

Client machine is an Epyc 7402P, PCI-E lanes are very important to not only how many you have but how they are laid out.

Back when I was doing some testing on another setup I found out having the Mellanox card and the NVME's on the chipset lanes would cause transfers to come to a complete halt.

I've found on setup's with limited PCI-E lanes it was best to put the Mellanox Card on the CPU lanes and the NVME's on the Chipset lanes.
My current setup allow's all lanes to be CPU on both the client and the server.

 

[ca3y6]

In my case these are H12SSL and H11SSL server motherboards so with plenty of PCIe lanes and transfers between HHHL SSD cards, so 8 lanes, high speed devices. And I get the same problem across machines so unlikely to be a bad setting. This on windows server 2019, 2022 and 2025. Plus crystaldiskmark shows these SSDs are capable of writing up to 2.8-3.5GB/s sequential depending on the queue depth, so the disks are fine, there must be something broken with windows.

 

[sultan.of.swing]

In my case these are H12SSL and H11SSL server motherboards so with plenty of PCIe lanes and transfers between HHHL SSD cards, so 8 lanes, high speed devices. And I get the same problem across machines so unlikely to be a bad setting. This on windows server 2019, 2022 and 2025. Plus crystaldiskmark shows these SSDs are capable of writing up to 2.8-3.5GB/s sequential depending on the queue depth, so the disks are fine, there must be something broken with windows.

The issue with Epyc system's and SMB is the lack of raw single thread clockspeed which can slow down single threaded task's such as SMB transfers.

From my Epyc system to the server I can max out the 40gbe no problem but from the server back to the epyc system I only get a max of around 16gbps.

 

[ca3y6]

The issue with Epyc system's and SMB is the lack of raw single thread clockspeed which can slow down single threaded task's such as SMB transfers.

From my Epyc system to the server I can max out the 40gbe no problem but from the server back to the epyc system I only get a max of around 16gbps.

but if that was the case it would also affect crystaldiskmark, which it doesn’t. It’s not SMB specific since it also affects drive to drive copies. I know it is also not SMB related since I can also max out a 40gbe connection between two epyc machines over SMB via crystaldiskmark.

 

[sultan.of.swing]

but if that was the case it would also affect crystaldiskmark, which it doesn’t. It’s not SMB specific since it also affects drive to drive copies. I know it is also not SMB related since I can also max out a 40gbe connection between two epyc machines over SMB via crystaldiskmark.

All drive to drive copies etc will be single threaded, try the drive to drive copy with Robocopy.

 

[bugacha]

It's a good card for NAS, cheap and fast. Not the best if you build a software based router, but handles NAS traffic well.

 

[bugacha]

~150€/$ piece

it doesnt cost this

the card is $50

you should change your mentality around warranty when stuff is dirt cheap

 

[sultan.of.swing]

Single port 25gbe CX-4 lx can be had for around $30 USD.

 

[Lycius]

it doesnt cost this

the card is $50

you should change your mentality around warranty when stuff is dirt cheap

Single port 25gbe CX-4 lx can be had for around $30 USD.

it's not only a question of warranty, there are a lot of counterfeit cards on the market, how can you verify that?

 

[sultan.of.swing]

I've bought 6 Mellanox card's off ebay this past year. Pretty easy to tell if they are legit or not.

 

[pimposh]

There are PCIe 4 cards on the market that can do 25gbps with x4 but they usually cost more.

Just wonder which cards are you reffering to ? MLX5/6/E810 ? Is it confirmed they correctly handle 25Gbps/50Gbps on PCI4/x4 layout scenario ?

 

[Lycius]

I've bought 6 Mellanox card's off ebay this past year. Pretty easy to tell if they are legit or not.

How you do ?

And what you do when you receive a fake one ?

 

[bugacha]

Just wonder which cards are you reffering to ? MLX5/6/E810 ? Is it confirmed they correctly handle 25Gbps/50Gbps on PCI4/x4 layout scenario ?

I haven't tested myself but looks like E810 does support it

E810-XXVDA2 running at PCIe x4

I am trying to determine what speeds the E810-XXVDA2 will be available when running the network card in a PCIe Gen 4 x4 Slot. Would it support 2x 10GbE ports or 1x 25GbE port? Thanks, Chris

community.intel.com

It requires physical x8 slot tho, they don't make them in x4 form factor

 

[pimposh]

I ask since in my friends M90q from Lenovo with PCI4x4 riser board MLX5 refuse to exceed 21Gbps and oddly MLX4 works faster. But let’s not hijjack thread from OP. Cheers

 

[Lycius]

on two computer i have two rj45 port : 2.5gb and 1gb

is it possible to make an SMB with these two ? or two identical speed are needed ?

 

[bugacha]

on two computer i have two rj45 port : 2.5gb and 1gb

is it possible to make an SMB with these two ? or two identical speed are needed ?

You can bond then as active/passive, but this wont increase speed.

its possible. 2.5gbps is decent speed for SMB, really depends on what you do

 

[Lycius]

it's for connect a work computer to my actual NAS, i use it to edit and transfert video, actually i use the 2.5gb only, but after reading this topic, i was thinking that maybe i could use both ethernet port ( 2.5gb and 1gb ) in smb to increase a bit the speed ( until i buy two mellanox card)

 

[bugacha]

it's for connect a work computer to my actual NAS, i use it to edit and transfert video, actually i use the 2.5gb only, but after reading this topic, i was thinking that maybe i could use both ethernet port ( 2.5gb and 1gb ) in smb to increase a bit the speed ( until i buy two mellanox card)

No, not without a big hassle. You would need 2 NICs on NAS, IPs in different subnets and maybe, maybe SMB multichannel will work across 2 NICs with different speeds. What speeds you will get in real life is a big unknown

 

[ca3y6]

Some switches support link aggregation, but even that is only of limited use, i.e. a single connection will still only use a single cable. It will help with the bandwidth of multiple connections.