Gigabyte MJ11-EC1 EPYC 3151 Mystery

[xxfreshman]

CEACENT CNS41CX16W adapter with SFF-8654 8i PCIe 4.0 85 Ohm cable and mini 6-pin to 6-pin cable .

There are couple of things you should now:

The mini 6-pin to 6-pin cable needs to be modified to corespond with the pinout showed by the manufacturer on the product specification page.


The adapter has a resistor (I think) C21 next to the mini 6-pin plug , you will have to cut the mini 6-pin connector's extruded bottom edge off if present to avoid knocking the C21 off the board.


The 30W limit mentioned by another member was for this adapter and not for the Ceacent adapter. So I am not sure where the issue is ,maybe it doesn't support Bus Width above 64 bit but then 3 members reported success for Tesla P4 (256bit) and 1 member for Quadro T1000 (128bit).

Phu, its more complicated as i thinked.

Begins that all that stuff is not anymore aviable.

Now i ordered the 3. Hardware hopefully this is working.

Universal SFF-8654 8i to PCIe 4.0 x16 Extension Card Graphics Card Adapter PCIe | eBay

This product SFF-8654 interfaces PIN signal is defined by general signal. If your machine SFF-8654 interfaces PIN signal is defined as special design, this product is not compatible. This product is SFF-8654 8i to PCIe x16 interfaces, and actually it can only go to PCIe x8, that is, PCIe x8 in...

www.ebay.de

and hope fully i dont need to make changes to the Power.
But as description you can use the Normal Power Adapter.

cu

 

[hmartin]

Phu, its more complicated as i thinked.

Begins that all that stuff is not anymore aviable.

Now i ordered the 3. Hardware hopefully this is working.

Universal SFF-8654 8i to PCIe 4.0 x16 Extension Card Graphics Card Adapter PCIe | eBay

This product SFF-8654 interfaces PIN signal is defined by general signal. If your machine SFF-8654 interfaces PIN signal is defined as special design, this product is not compatible. This product is SFF-8654 8i to PCIe x16 interfaces, and actually it can only go to PCIe x8, that is, PCIe x8 in...

www.ebay.de

and hope fully i dont need to make changes to the Power.
But as description you can use the Normal Power Adapter.

cu

I have tested these adapters too. They work fine. You can find them for ~half the price of eBay on AliExpress though.

Just be mindful of what you install in the adapter. A GPU expecting 75W from the slot might melt the SATA power connector which is rated for much less (~30W)

 

[_Dejan_]

Im start one of my board with 4x16GB RAM SK Hynix HMA82GR7AFR4N-VK, modify over BMC SSH board to EC0 to enable SYS FAN control and remove not working sensors, flash latest BMC, Flash F02 BIOS(Modified v2 from German forum) and install TrueNAS Scale to it.
Currently running 12 hours without issues.
Idle power usage is 32-33W MAX peak has been 63W:
MJ11 Motherboard (Original CPU Fan running at 1500rpm)
4x16GB RAM
1x Toshiba XG5 256GB NVMe
2x BeQuiet Silent wings 4 120mm PWM High-Speed(Running at 600RPM)
Old LC Power LC6550 v2.2 PSU
BMC and 1x LAN connected

 

[efeschiyan]

Hi,

I am facing the Same issue under Proxmox. The NIC freezes with the PCIe Lost Link Message after 1-2 days. I use pcie_aspm=Off but also tried the Performance setting, etc.

Did you find a solution? I have ordered parts to use the 8i for an external Card, but the internal ones would be sufficient for me... If they would Work.

Thanks,
Florian

hey there,

pci=nommconf ahci.mobile_lpm_policy=0 pcie_port_pm=off pcie_aspm.policy=performance irqpoll

is what I've been using since that post last year. it's been working stable for a month now.

 

[Th0mas51]

FYI, I'm still in the process of searching for a better CPU cooler for the board.

I managed to remove the backplate of my 2 motherboard, that way I might have more choice in term of CPU cooler compatibility.

For those who are interested, you can find the instructions below.

Use that at your own risk, I'm not responsible if you break something.

Tools:
- a hairdryer
- a piece of razor blade
- a flat screwdriver
- dental floss

Steps:
1) use the hairdryer to warm up the glue a bit (be careful to not overheat the motherboard !)
2) using the razor blade, without damaging the motherboard, try to make some space in the corner
3) use the hairdryer to warm up the glue a bit
4) insert the dental floss in the corner and try to push the glue as far away as possible
5) use the hairdryer to warm up the glue a bit
6) using the razor blade and the screw driver, gently operate the screwdriver to detach the backplate from the glue
7) use the hairdryer to warm up the glue a bit
8) gently continue the process until the backplate is removed

Be careful to not overheat the motherboard with the hairdryer.

If you have removed the black insulation tape like I did, and if you plan to use a metal backplate, you will have to add some sort of insulation material (i.e tape), as metal backplates should not be touching the motherboard.

 

[_Dejan_]

Im run memtest on my configuration with 4x16GB RAM SK Hynix HMA82GR7AFR4N-VK and I get a lot of ECC errors... Any idea what I can do? Im already replace all 4 sticks with another 4x16GB RAM SK Hynix HMA82GR7AFR4N-VK but didn't help...

Im try setup speed for RAM to 933MHz but looks like is not accepted:





Any idea what I can do? I will later try with only 2x16GB RAM...

I had plan run 3 boards: One withTrueNAS with 1x16GB or 2x16GB RAM and two for Proxmox with 4x16GB RAM...

 

[dreunion61]

@_Dejan_ Those are dual rank modules. Try to avoid those and use Samsung modules.

 

[_Dejan_]

@dreunion61 Thanks for your reply. Can you check BMC Screenshot, there is detected as 1 rank modules... In post Im write wrong model... Correct is SK Hynix HMA82GR7AFR4N-VK This should work 4x16GB but BIOS for some reason do not accept 933MHz RAM speed...
With 2x16GB Memtest work fine witout errors or ECC errors...

But I have issues that board reset itself after few hours of running iperf3... I run iperf3 server(2 instances) on MJ11 and 2 iperf3 clients one from my Pc and one from another pc.
What Im test:
1.) TrueNAS Scale and Proxmox both reset after few hours. I try also kernel option:

pcie_aspm=off

2.) I test on PVE 8.3.0 and 8.3.3, and in both there is the same... I try also kernel options:

pci=nommconf ahci.mobile_lpm_policy=0 pcie_port_pm=off pcie_aspm.policy=performance irqpoll

3.) I try on PVE 8.3.3 disable C6 state using zenstates.py script. After little less than 3 hours server reset.
4.) I try windows 11 and run on it iperf3 server for 18 hours without issue.

Problem cant be with Intel drivers because I have dual I210-AT on another server(Gigabyte MC12 MB) and Proxmox and dont have any issue(only difference is that they are connected to chipset instead of cpu)... I dont have any idea what to try... I will need to unpack another board to see if have same issues...

EDIT: Right now I run second motherboard on stock BIOS and stock BMC and with one 16GB RAM stick because with 2 it didn't boot(BMC show only NO SIGNAL)... Will see how will this board work...

 

[_Dejan_]

I’m unpack second motherboard and put 1x16GB stick in it …

BIOS is F09 and BMC 12.49.06. BIOS settings are untouched and nvme with installed Proxmox has been moved from previous board to this one. iperf3 has work without issues for 20 hours…

Now I can change memory speed to 933MHz and board accept it I’m also run Memtest with this speed for few hours:


Maybe me do something wrong... One stick of memory I run in slot B1 I’m test 2666MHz(Auto) and 1866MHz(933MHz in BIOS) and work fine.

Then I’m install additional memory so I’m try 2x16GB one in A1 and another in B1 slot, BIOS has been configured to 1866MHz(933MHz) and board need long time to show Gigabyte logo and in BIOS I see that memory speed is 2666MHz I’m check OC settings and has been 933MHz. I save settings again and it few times(If I’m count correct it try 3 times) try to boot(diagnostic numbers go to 70) then NO SIGNAL and start again. Then I’m in BIOS change memory speed to 1066MHz(2133MHz) save settings and reboot. Board boot normally and show 2133MHz memory speed...

I’m test iperf3 for 14 hours and it work without issues...

 

[_Dejan_]

Some additional tests with second motherboard... Im try 4x16GB memory sticks and for now I didnt find working combination... At 1066MHz memory speed board normally post but thera are a lot of ECC errors. If I set memory speed to 933MHz board do not post normally. It try same as 2x16GB modules with this settings few time turn off/on and then few times start initialization and then post wit 1066MHz speed(In BIOS is still 933MHz)...

Any sugestion what to try?

 

[Andiii]

4x single rank RAM

 

[_Dejan_]

4x single rank RAM

I don't understand what you want to say? Im using single rank RAM's... I use SK hynix HMA82GR7AFR4N-VK which are 1Rx4 ...

 

[Andiii]

Ah okay, i have had missed that Info. In the German forum they found this workaround, install 2 of your RAMs in the blue Slots. Set These settings :

AMD CBS -> UMC Common Options -> DDR4 Common Options -> Data Bus Configuration:
RTTnom 7
RTTwr Dynamic ODT off
RTTpark 2

~ >DRAM Timing Configuration:
Memory Clock Speed 1067MHz
Trcdrd 10h
Trcdwr 10h
Trp 10h
ProcODT 48 Ohm

Save and Install the other 2 RAMs.

 

[_Dejan_]

Im try BIOS reset by BIOS and by jumper on MB but do not make any difference... 933MHz memory speed is accepted and work only when I use single RAM stick. If I use 2 or 4 is not accepted and board before post reboot few times...
Im try similar settings but with:

Memory Clock Speed 933MHz
Trcdrd 13h
Trcdwr 13h
Trp 13h
ProcODT Auto

I will try exact settings from german forum.

 

[Andiii]

I use dual Rank so i use these settings, which were recommended in the same thread:

 

[_Dejan_]

@Andiii your settings looks like work ok Previous board looks like didn't like them for some reason...

 

[_Dejan_]

I don't understand this boards...
Im runn test for around 10-12 hours until it complete... And there has been no errors and no ECC errors... Then Im run Proxmox and iperf3 for around 12 hours without issues...
Then Im reflash first board with stock BIOS. With modded didn't work and there has been ECC errors. And run ti for another 12 hours until test complete with no errors and no ecc errors. Im use another 4 memory sticks so second board stay untouched. Then Im power on second board and again start mem test and in 4 minutes receive more than 30 ECC errors... Then Im reboot it and check if RAM settings are correct and they are. Then Im run mem test again and it run normally for one hour???

 

[s0lid]

Got Mellanox CX4121A NIC to work with this adapter and cable while using EC0 bios:

Cablecc Oculink SFF-8612 4X to PCI-E 16X Adapter for Mainboard Graphics Card | eBay

This oculink pci-e converter enables belowing applications. The converter can be installed into the system internally through the SFF-8611/8612 oculink interface. Cablecc Mini mSATA SSD to SATA 7pin Hard Disk Drive PCBA Extension Adapter SATA.

www.ebay.com

Wonderwires OcuLink PCIe PCI-Express SFF-8611 4i to SFF-8654 Slimline SAS SSD | eBay

OcuLink SFF-8611 male to SFF-8654 male. If you need a specific length of cable, with specific connectors at specific angles then we are here for you. 50cm Mini SAS SFF-8643 HD 90° Angle Elbow to Mini SAS 36 Pin SFF-8087 Convertor.

www.ebay.com

Not so much luck with dual NVME drives hooked to the 8i oculink port. If my idea of an ultra janky hack with the refclk's doesn't work. Then I suppose I've to convert the M.2 slot into U.2 and use 2.5" SATA disk as boot drive instead of a M.2 NVME. I just don't have that much of a space free for proper pci-e to U.2 adapters in the case I'm planning on using.



The SSD cage is aligned like that because I needed some space to plug cable in and out of the psu. Still need to make nice bracket from aluminium for the fans and that pci-e adapter.

Board seems to play nicely with 4x Micron MTA36ASF4G72PZ-2G6E1SG 32GB RDIMM Sticks. Haven't done any memtest just yet.

 

[_Dejan_]

Board seems to play nicely with 4x Micron MTA36ASF4G72PZ-2G6E1SG 32GB RDIMM Sticks. Haven't done any memtest just yet.

This are dual rank modules I would like to see if there will be ECC errors using all 4 slots on this motherboard...

 

[_Dejan_]

Right now I run one board with:
4x16GB SK hynix HMA82GR7AFR4N-VK
LC Power LC6550 v2 2 PSU
1x Optane M10 16GB for Proxmox VE
1x Be Quiet SILENT WINGS 4 120mm PWM High Speed @600RPM for CPU
1x Be Quiet SILENT WINGS 4 120mm PWM High Speed @750RPM for input
1x Be Quiet SILENT WINGS 4 120mm PWM High Speed @750RPM for output

Idle power consumption running Proxmos without any optimization is 31-32W before idle Im run memtests and power usage has been 70-72W(FAN's run at higher speeds):



With 2x Intel D3-S4610 960GB SSD's in ZFS mirror idle power consumptions increase to 35W.

With 3 running VM's power increase to 40-42W

If together with VM's run 2 instances of iperf3(From 2 different hosts) power increase to 45-46W

Im transfer more than 2TBytes of data and board didn't reboot: