Gigabyte MJ11-EC1 EPYC 3151 Mystery

[janek202]

i allready got another mainboard, after management booting the screen stays black, have tryed 3 types of normall ddr4 ram and 32gb samsng sticks M393A4K40BB1-CRC whit no post,... im totally lost,...

am i even booting it right ?

Have you tried with 2 modules in blue slots? Sorry for a silly question, but is the fan spinning?

 

[ZoMbY]

Have you tried with 2 modules in blue slots? Sorry for a silly question, but is the fan spinning?

1 module, 2 module only in blue slots, fans spinning, the BMC controler is starting and it allso shows on the monitor, after that its all black, no post at all the selles send me a replacement and it does the same, now im questioning my skills

 

[hmartin]

@hmartin Are they running at 2666Mhz or do they clock down to 2133Mhz?

They run at 2666MHz when two are present, with more than two modules the speed drops to 2133MHz.

looks like this is a faulty motherboard that i have. BMC does not read the ram that is physically inserted, it states i have 1 stick of 16gb ram in there ) . KVM has no signal,...
does not post,.... bummer for me

My BMC also reports only one DIMM installed until you boot and it completes POST, then they are all displayed in the BMC inventory. Same as janek202 said.

i checked and its not,... in the QVL there are only 2666MT ram, this samsung is 2400MT - M393A4K40BB1 -CRC0Q
does the bmc work regardless ??

Yes, the BMC should be working even if your RAM is not compatible with the EPYC. The BMC is an independent ARM system with its own RAM.

 

[ZoMbY]

so now i ordered some ram fom the QLV list,.... lets see what happens,...

 

[PeterF]

i allready got another mainboard, after management booting the screen stays black, have tryed 3 types of normall ddr4 ram and 32gb samsng sticks M393A4K40BB1-CRC whit no post,... im totally lost,...

am i even booting it right ?

What PSU do you use?
Are both the 12V and Standby power connected?
Have you also bought the adapter cable from the seller?

BR
Peter

 

[user9482389]

A friend of mine was so kind and modelled his own IO shield for me, you can find it on his repository.

Fits perfectly and better than the one found by @joer34 .

Now I can finally finish this build .

 

[ZoMbY]

What PSU do you use?
Are both the 12V and Standby power connected?
Have you also bought the adapter cable from the seller?

BR
Peter

adapter cable came whit the purchase, for psu im using a corsair hx1000, bothy 12v pins are populated,..

 

[Nautor]

Hi all,

I ordered 2 of this boards to play with them, I've been thinking quite a while of test the Epycs, so it was a good opportunity. I currently have a Supermicro x10sdv-6c for my proxmox and virtualized truenas with it's own dedicated LSI IT Mode HBA. Performance wise seems pretty similar to my supermicro and I'm trying to replicate my supermicro conf for spare.

The thing is, I was trying to replicate the work of others here adding a slimsas 8i to PCIE with the boards I ordered from Ram König, but all my efforts ends in failure. I have tried these 2 scenarios:

Pcie cards I tried in both motherboards:
1Gb 2x port ETH intel NIC (Pcie 4x)
1 old nvidia Gpu
1 H310 Dell PERC HBA in It mode

Test A:
Cable SFF-8654 8i to sff-8611(oculink) x2
SFF-8611 to PCIe board. Same used by others before.

Test B:
C-Payne SlimSAS PCIe gen4 Device Adapter 2* 8i to x16
SlimSAS 8i to SlimSAS 8i cable.

I tried with the external power sata and 4 pin power cable and without it, no difference.

Both tests end in failure, it is like the cards don't even received power or they are not booting or loading, the leds on the HBA and the NIC didn't blink as when they are booting the firmware in normal condition. I tested also the 3 cards and they work fine in other motherboards.

I restored the bios to default several times, I tried to change some parameter in the bios but no luck at all. What I'm doing wrong? Do I need to flash the bios from the EC1 to EC0 to make it work, or what I'm missing here? Did you changed anything in the bios conf?

@jnrnbt. @SadoKitten

 

[_Dejan_]

Hi everyone,

I think to buy 2 of this boards to replace my current server:
Supermicro H11DSi
1x Epyc 7601 (HT is disabled)
128GB (8x 16GB DDR4 ECC RDIMM) RAM
2x Samsung PM983 960GB M.2
Mellanox MCX354A-FCBT

This system is total overkill for my VM's(Sophos FW, HomeAssistant OS, TrueNAS, IPTV Server, GPS Track Server, UNIFI controller and one Win 10 VM) and it do not offer me redundancy...
CPU on Proxmox idle most of time(Last month CPU usage is here) spikes are mostly when HAOS is updating:


I think that 2 of this Gigabytes will be powerfull enought and give me HA for some of VM's which need to run all time(Sophos FW and HomeAssistantOS)...

Im quick read thread and Im not sure if Im understand if it I can connect everything what I want to this motherboard...
I would like use one 10gbit(Dual port if possible) card with SFP+ connection(One will be for LAN another for replication/migration but I think I can live also with only one) and one or two Samsung PM983 or PM9A3 U.2 2,5" SSD's.

ECC RAM I already have(I have spare from main server) which is on compatible list... I have plan use 64GB(4x16GB) per server.

Im not sure if is better connect network card connect over M.2 adapter or SFF-8654 adapter? And one U.2 drive connect over slimsas(If Im read correctly it offer 4x SATA or PCIe3 x4) and another U.2 drive connect over M.2 or SFF-8654 (Depend of which one is used for network card)...

For OS Im not sure anymore what I will use Im think to switch to TrueNAS Scale but then Im read that it do not offer HA? Running TrueNAS as VM like now(Use 2 SATA drives) it need to give me raw mapping pcie device(local SATA ports) and as I understand they need to be in seperate IOMMU Group to passtrough them... How are devices split to IOMMU Groups on this board?

 

[user3872465]

As already said, the EC1 is missing some settings from the EC0 BIOS.

Warning: I advise against flashing your BIOS, it can damage your system. Crossflashing in particular is even more dangerous!
I backed up the BIOS from the EC1 and flashed the current version F02 from the EC0 via AFUWIN.
The system booted up without a problem and the missing settings are now there.
The "SL_SAS Control" option to switch the SFF-8654 4i between PCIe and SATA is there, but I did not test that.
I was interested in setting the PCIe_1, which should be the SFF-8654 8i (or as the silkscreen names it U2_2) to 4x4x4x4.

I connected the SFF-8654 8i port with an Amphenol RSL74 to a generic SFF-8654 8i to 2xM.2 X4 board, equipped with two PCIe 3.0 NVMe drives.

The system was still not able to detect the second drive, so it seems bifurcation still does not work.
In any situation just the one slot worked, and I was not able to get any drive to work in the second slot, even when only one drive was installed.

Maybe the port is really just electrically a X4 port, or the settings from the EC0 BIOS just do not work with the EC1.
Hopefully just my generic SFF-8654 8i to 2xM.2 board does not work, but I doubt that.

I Recently got that board too, my first reaktion to get 2 U.2 drives working with bifurcation was the same as you so I cross flashed it. Now I dont have the old Bios anymore as I wasn't to smart as to make a backup of it. Is there an option you could share the original Bios with me so I can revert the flash?

 

[BadMax02]

Im not sure if is better connect network card connect over M.2 adapter or SFF-8654 adapter? And one U.2 drive connect over slimsas(If Im read correctly it offer 4x SATA or PCIe3 x4) and another U.2 drive connect over M.2 or SFF-8654 (Depend of which one is used for network card)...

For OS Im not sure anymore what I will use Im think to switch to TrueNAS Scale but then Im read that it do not offer HA? Running TrueNAS as VM like now(Use 2 SATA drives) it need to give me raw mapping pcie device(local SATA ports) and as I understand they need to be in seperate IOMMU Group to passtrough them... How are devices split to IOMMU Groups on this board?

TrueNAS Enterprise offers High Availability as far as i remember correctly, tho that costs money and its more specific build for the machines that truenas also sells.
The IOMMU Groups (when enabled) seem to have the all the sata drives in one group, meaning if proxmox was installed on a pcie drive it could theoretically run the drives nmatively inside a VM. There are alot of IOMMU groups making it quite Virtualization friendly, so i dont think that should be an issue there.

My setup is running one of these SlimSAS 8i port to dual m.2 Slot board and having a Optane 16gb ssd in there and than a m.2 10gbe rj45 network adaptor in the native m.2 slot of the Board, seems to work quite well and made it possible to use the pcie slot of the jonsbo n2 case im using.
im also using the SlimSAS 4i to 4x Sata cable and one sata connector of the board for 5x 12TB in a Raid Z2 (Raid 6 but ZFS) and plan on using the remaining 3 Sata ports for 3x 4Tb Sata SSDs in a Raid Z1 (Raid 5 but ZFS) but thats probably gonna be for after new years.
2x 32GB of 2666 MT/s Registered ECC Ram.

i migrated my Windows server i had before to truenas scale with a backup restore over the network and i just migrated my physical OPNsense router to a vm inside truenas aswell (it just acts as a OpenVPN Server for now), also setup a docker container running UrBackup Server for windows Client backups but havent deployed it to any of my Windows PCs yet.

I copied all my Data i had on a single 10tb drive on my windows server to the 5 Drive array and am actively using it now for all my stuff

If interested i could post a Picture of my build, and when they board without the ATX adaptor are on oiffer for 49,99€ again than im definitly gonna buy 1 or 2 more of them and try doing stuff with a Proxmox cluster and networked storage from the Truenas system over iscsi or something

 

[_Dejan_]

BadMax02
Thanks for information about IOMMU Groups.

Right now I'm create 3 VM's with TrueNAS Scale and will try build cluster... As Im read one or two year ago Cluster and HA has been possible only on Enterprise and their hardware. But with Scale edition as Im understand Clustering is possible also without license and on custom hardware... First what Im found is that Cluster with TrueCommand is deprecated and will be removed... So another option is using MinIO ... Will see what test will say...
Im looking for solution like this:



So each host will have all data. I need distributed SMB which is availible on some virtual IP and VM's which are distributed over different hosts but can simple run on any host(Some of host fail or is in maintenance)...

Main thing which right now is unknown for me is if I will with only one drive in ZFS(Stripe) (One U.2 SSD and one 8TB HDD per host) cluster will offer me protection against data coruption... In ZFS Mirror I know that there is scrubing which found errors and correct them, but this mean 2xSSD + 2xHDD for each server whcih will increase power consumption...

I would like to have minimal devices in each server to decrease power consumption to minimum. This is also reason why RJ45 10Gbit card is not option for me because it normaly draw double power as SFP+ cards... Also in switch I will need additional transciver which add power consumption and heat... Im thinking of using Intel X710-DA2 or Mellanox CX312A or CX312B card. One SFP+ for local LAN traffic and another for servers cluster data... If SlimSAS 8i work with x8 adapter like this one on link it will be ok:

SlimSAS 8i (SFF-8654) to PCIe x8 Slot Adapter

This PCIe x8 adapter provides exceptional performance in terms of data transfer speed and reliability. It is the perfect solution for connecting a SAS2 drive to a computer system via a PCIe slot.

www.microsatacables.com

My system is not that big as your and need only 450GB fast storage(nvme) for VM's and 3.3TB space for my files(NAS)...

 

[joer34]

I Recently got that board too, my first reaktion to get 2 U.2 drives working with bifurcation was the same as you so I cross flashed it. Now I dont have the old Bios anymore as I wasn't to smart as to make a backup of it. Is there an option you could share the original Bios with me so I can revert the flash?

Does the board have dual bios, or bios recovery? If yes, you could download bios file and reflash or restore... Otherwise, I could dump bios chip and send you dump, but it requires programmer and pretty sure it will not work, because, one cannot simply restore dump A to board B (same model but different data....)

 

[boxyrobot]

Here is the guy who uploaded to actual bmc firmware. The firmware 12.60.39 you have posted isn't the actual one. As i write this post: the actual version was 12.61.01. Now as i write this post i have checked for a new version and here we are. A new version was released at Nov. 15 with version number 12.61.17. Here is the link to the gigabyte download.

How did you come about this link? Tried searching GIGABYTE support page/s with no luck.

 

[prime420]

How did you come about this link? Tried searching GIGABYTE support page/s with no luck.

On the board is a ASPEED AST2500 chip as BMC installed, luckly gigabyte sometimes keep things simple as possible and develope only one firmware for all boards where they used this chip. For every board which used this chip, a seperate device profile is existing in the firmware. So if a new version was released i check the changelog, if the our board are removed from the supported devices. If not, i flash it to the BMC, if the updates was succesfull i will share the link.

Last BMC patch level: 12.61.17, released 15 Nov. 2023

 

[user3872465]

Does the board have dual bios, or bios recovery? If yes, you could download bios file and reflash or restore... Otherwise, I could dump bios chip and send you dump, but it requires programmer and pretty sure it will not work, because, one cannot simply restore dump A to board B (same model but different data....)

Not sure probably not but theres the option to dump the bios from the IPMI web UI. And as I flashed a different boards bios already onto it which i want to reverse it defo is possible. So it should be possible to just restore the Bios via a dump from another board.
But I can also wait on u/prime420 to share their progres son the BMC update (tho that is not directly the BIOS)

 

[Gizmo_Ger]

The 12.61.17 works just fine.

 

[whcvip]

Just ordered a few of these boards, just such good value.

Any luck getting the two PWM headers working? Flashing the V1.2 variant of this board got them working?

Am thinking of building in the Jonsbo N3 case, with room for 8x SATA drives. Then my plan is to install a GPU (P2000) and a 10G NIC in the M.2 slot.

I see several M.2 NIC's on Aliexpress and ebay, but seems like they have a rather small heatsink, and also do cost a fair bit more then a standard PCIe card. But came across this item, wondering if it will work, M.2 adapter to SFP+ with SAS cable and needing SATA power.

 

[gtawelt]

The manual states that the CPU Fan Port supports PWM. SYS1 and SYS2 are not mentioned. I own the complete rack system and can control SYS Fans over the IPMI, but they are connected at the power distribution board. So the 2 fan channels on the IPMI are not connected to the onboard PWM ports.

The SFP+ NIC seems quiet interesting, i think it could also connect to the SlimSAS Port with an SFF8087 to SlimSAS cable. Sadly the NIC ist not available in my country.

 

[whcvip]

Strange they includes two extra PWM headers, and not put them to use. Did also notice the fans in the G431-MM0 server was mounted to the power distribution board, and the fans there is 6-pin fans. Well maybe a fan hub is best solution, want to run 5 fans total (or maybe just 3), and for them to be controlled according to temperature. Or maybe som of the fan controller plugins in Unraid is able to control those fan headers.

Anyways looking forward playing around with these boards, hope my 2400MT RDIMM will work, and ordered some SlimSAS 4i to 4x SATA cables of Aliexpress, and some SlimSAS 8i cables and x16 adapter boards. The build in 1G NIC's is fine to start with, but else have some 2.5g NIC I can use wit the M.2 adapters i also have on the way from Aliexpress.

Hope it is all worth it, in the end it all runs up cost wise. But hoping to get two identical systems up and running, replacing my 4x4-drive NAS servers (Asustor and QNAP) I have now, with just 2x8-drive ones, and get a speed bump plus GPU for transcoding (P2000 or P1000).

Anyone tried building in the Jonsbo N3, got two of those on the way, seems to be best 8xSATA case with room for full size PCIe cards. Or other cases worth taking a look at in same form factor?