Gigabyte MJ11-EC1 EPYC 3151 Mystery

[BadMax02]

First a warning. This can potentially brick your system. I have had one side effect. The product name disappeared from the BIOS and instead there is a text "To be filled in by OEM" It does not seam to have any impact on running the system.

I did all my testing with the main CPU off, only the BMC powered. I suggest you do the same!

Hey, thank you so much for that guide, it cleaned up my build nicely with way less wires and splitters because i was running all my fans from the cpu header with a splitter.
I Used an Arduino with its Reset Pin Shorted to Ground to prevent it from turning on and hooked up the TX adn RX pins 0 and 1 to the corespondenting pins of the BMC and it actually worked fine with putty.

I actually did the BMC edit with the system turned on and using the sysadmin user and it all went fine, for some reason it still says the old device strings and names but the fans show up, so hey im not complaining haha.

One side note is that if you somehow manage to break your BMC doing this, you can always just flash a completely new BMC onto the nand chip using the EFI Shell and the guide provided in the firmware zip file, the last page shows how to flash it in the EFI Shell, you would need to hook up a monitor and keyboard to do this tho.
I know because ive actually bunked up flashing a newer version of the bmc a few weeks ago.

 

[PeterF]

That would be awesome, thankyou!

Hello,

Attached is the original F09 Bios that I dumped from my system and made into a flashable .rbu image. It is not possible to flash back the dump as is via the WEBGui or AFUxxx. I compared to the MJ11-EC0 image from Gigabyte site and found the difference. There are a lot of system specific NVRAM variables in the dump. In a hex editor I exchanged these with FFs. Empty space is all bits set. Adding the rbu 10 bytes at the end with the right checksum made it flashable. You can check this with your own BIOS dump. From hex 00294000 they should be identical. There is a part that UEFITOOL calls padding that is not empty and that differs. I have flashed BIOS for one motherboard to another without problems.

If you instead prefer to modify your dumped BIOS I can give instructions to do so.

You should flash this from the BMC with the main CPU off. After the flashing when you restart, the system will reboot several times before coming to the BIOS screen.

Of course be warned that all flashing can potentially go wrong and you will have to take all responsibility

I have uploaded my BIOS to dropbox. here is the link

Dropbox

www.dropbox.com

The sha256 checksum is f7eeaeefa17fe98ec56007c24aa50e6e392b2fca862b4330a33a42d91e5b42f3

Peter

 

[PeterF]

Hey, thank you so much for that guide, it cleaned up my build nicely with way less wires and splitters because i was running all my fans from the cpu header with a splitter.
I Used an Arduino with its Reset Pin Shorted to Ground to prevent it from turning on and hooked up the TX adn RX pins 0 and 1 to the corespondenting pins of the BMC and it actually worked fine with putty.

I actually did the BMC edit with the system turned on and using the sysadmin user and it all went fine, for some reason it still says the old device strings and names but the fans show up, so hey im not complaining haha.

One side note is that if you somehow manage to break your BMC doing this, you can always just flash a completely new BMC onto the nand chip using the EFI Shell and the guide provided in the firmware zip file, the last page shows how to flash it in the EFI Shell, you would need to hook up a monitor and keyboard to do this tho.
I know because ive actually bunked up flashing a newer version of the bmc a few weeks ago.

Hello,

Nice to hear it worked for you also!

Yes flashing the BMC from the EFI shell (or Linux) works and is a good way. If it does not come up correctly you can flash again.

But I do not think that the SKU info is stored in the BMC image. The Gigabyte BMC firmware is the same for all products with the same Aspeed version. It contains no system specific information. This must be stored in another flash or eprom memory

BR
Peter

 

[monotux]

Finally managed to build a system with this motherboard, installed FreeBSD 14 just to see if my disks were OK and see if it booted. No issues.

Fit very nicely in an old Fractal Design Node 304.

Now comes the hardest part - what do I do with this system? First idea is to replace my current file server and that might be what I end up doing. Not sure if I will even attempt to buy the adapters needed to get 10 Gbps on this machine.

Maybe a backup server at someone elses place?

 

[Gymnae]

Hello,

Attached is the original F09 Bios that I dumped from my system and made into a flashable .rbu image. It is not possible to flash back the dump as is via the WEBGui or AFUxxx. I compared to the MJ11-EC0 image from Gigabyte site and found the difference. There are a lot of system specific NVRAM variables in the dump. In a hex editor I exchanged these with FFs. Empty space is all bits set. Adding the rbu 10 bytes at the end with the right checksum made it flashable. You can check this with your own BIOS dump. From hex 00294000 they should be identical. There is a part that UEFITOOL calls padding that is not empty and that differs. I have flashed BIOS for one motherboard to another without problems.

I was struggling to flash back my dump as well, but flashrom came to the rescue. I was able to boot into Proxmox, run the tool, create a backup of the F02 Bios and flash back the untouched dump I create in the BMC WebGui

You need the newest version, which you have to compile yourself. These are the commands I used:

./flashrom -p internal:laptop=this_is_not_a_laptop -c MX25L12833F/MX25L12835F/MX25L12845E/MX25L12865E/MX25L12873F  -r backup.com
./flashrom -p internal:laptop=this_is_not_a_laptop -c MX25L12833F/MX25L12835F/MX25L12845E/MX25L12865E/MX25L12873F  -w /mnt/usb/image.bin

Question for @PeterF: After your mod, did you see a change also for the PCI-E devices? I'm trying with a bunch of SFF8654 cables (original Gigabyte, Asus, no name) to attach a PCI-E card with an adapter to the SlimSAS i8 port on the motherboard, but it's not getting recognized. Maybe that is related to the SKU. I tried the original and the F02 Bios, but there's no change.

 

[whcvip]

After your mod, did you see a change also for the PCI-E devices? I'm trying with a bunch of SFF8654 cables (original Gigabyte, Asus, no name) to attach a PCI-E card with an adapter to the SlimSAS i8 port on the motherboard, but it's not getting recognized. Maybe that is related to the SKU. I tried the original and the F02 Bios, but there's no change.

Still waiting for my adapters to arrive (ordered from Aliexpress). Have ordered two different models, this one here, and the one in your post, plus unbranded cables from eBay and Aliexpress, plus these ones from 10Gtek, that is rated 100ohm as the one used in the server of this board (the cheap ones ordered was rated 85ohm).

But am a little concern about you not being able to get it working, have read through a German forum, also dedicated to this board, and also users there are reporting they cannot get the adapted 8i to x8 pcie working. Planning to running a GPU in my slot, so really hope there is a solution to this.

is there anyone here that have successfully adopted SlimSAS 8i to Pcie x8?

 

[janek202]

is there anyone here that have successfully adopted SlimSAS 8i to Pcie x8?

Currently i build a test setup to host proxmox and pfsense as a virtual host. Managed to get the 8i working into an x8 slot.
Cable

https://a.aliexpress.com/_Exw3XtP

Pcie riser

https://a.aliexpress.com/_Exh8Aut

10g double network card

https://a.aliexpress.com/_EIkVT7b

Just need to figure out an case to house the stuff.

Chielos had a success. I ordered these adapters too and I’ll try connecting a 10g card in a few days.

Edit: Looks like PCIe expects impedance of 85 Ohms, and SAS 100 Ohms.

https://www.mouser.com/pdfDocs/987652-0234.pdf

I think 85 Ohms cables with are the correct ones in this case.

 

[whcvip]

Chielos had a success. I ordered these adapters too and I’ll try connecting a 10g card in a few days.

Edit: Looks like PCIe expects impedance of 85 Ohms, and SAS 100 Ohms.

https://www.mouser.com/pdfDocs/987652-0234.pdf

I think 85 Ohms cables with are the correct ones in this case.

As far as I understand, only some cards do work in the pcie adapter. Maybe the 8i slot is defective on some boards, could be a reason they sell them so cheap, and the seller RAM-KÖNIG only mention 1x SlimSAS port in the description in the eBay listing and their website. And only a few full servers was listed for sale by piospartslap, not to mention they have (as of this post) 1164x SlimSAS 8i cables for sale on their website, that originated (presumably) from one og these GPU servers.

Well only speculations, but still hope all these adapters I ordered, plus cables don't end up in a drawer, don't currently have any other motherboards I can use them on.

EDIT: The SlimSAS 8i cable that was used in these GPU servers, was 100ohm cables. But as you mention, 85ohm is for pcie and 100ohm for SAS spec. I ordered both just to test and see what works best, if it works at all .

 

[PeterF]

I was struggling to flash back my dump as well, but flashrom came to the rescue. I was able to boot into Proxmox, run the tool, create a backup of the F02 Bios and flash back the untouched dump I create in the BMC WebGui

You need the newest version, which you have to compile yourself. These are the commands I used:

./flashrom -p internal:laptop=this_is_not_a_laptop -c MX25L12833F/MX25L12835F/MX25L12845E/MX25L12865E/MX25L12873F  -r backup.com
./flashrom -p internal:laptop=this_is_not_a_laptop -c MX25L12833F/MX25L12835F/MX25L12845E/MX25L12865E/MX25L12873F  -w /mnt/usb/image.bin

Question for @PeterF: After your mod, did you see a change also for the PCI-E devices? I'm trying with a bunch of SFF8654 cables (original Gigabyte, Asus, no name) to attach a PCI-E card with an adapter to the SlimSAS i8 port on the motherboard, but it's not getting recognized. Maybe that is related to the SKU. I tried the original and the F02 Bios, but there's no change.

Very good to know that flashrom works. I tried it also but probably not the latest version. It did not detect any flash.

I have not tries to connect anything to the slimsas 8i port. I use a sata breakout cable for the slimsas 4i and that works well.

BR
Peter

 

[BadMax02]

Just throwing my few cents inhere regarding the slimsas 8i cables, i got mine from some random aliexpress seller selling a slimsas 8i to 2x m.2 card and having the option to buy the cable aswell, bought both and ive been running a m.2 in that board connected to that 8i port no problem.
Its only 4 lanes but it does work the other m.2 ofc does not, because no bifurcation :/

I might buy another of these boards just for testing and might try bios modding the original updated bios to see if bifurcation can somehow be enabled, also might try and solder that second 8i port in aswell, i do have a hot air gun and semi good soldering skills, so might be interesting to see if that works.

 

[hmartin]

I was struggling to flash back my dump as well, but flashrom came to the rescue. I was able to boot into Proxmox, run the tool, create a backup of the F02 Bios and flash back the untouched dump I create in the BMC WebGui

You need the newest version, which you have to compile yourself. These are the commands I used:

./flashrom -p internal:laptop=this_is_not_a_laptop -c MX25L12833F/MX25L12835F/MX25L12845E/MX25L12865E/MX25L12873F  -r backup.com
./flashrom -p internal:laptop=this_is_not_a_laptop -c MX25L12833F/MX25L12835F/MX25L12845E/MX25L12865E/MX25L12873F  -w /mnt/usb/image.bin

Question for @PeterF: After your mod, did you see a change also for the PCI-E devices? I'm trying with a bunch of SFF8654 cables (original Gigabyte, Asus, no name) to attach a PCI-E card with an adapter to the SlimSAS i8 port on the motherboard, but it's not getting recognized. Maybe that is related to the SKU. I tried the original and the F02 Bios, but there's no change.

You should also be able to dump from the BMC, since it has mux access to the BIOS flash. However, I haven't been able to get it working yet. Note that you need root access on the BMC (but this is easy given the stock firmware has hard-coded credentials).

To dump BIOS, first run /sbin/pre_action_update_bios.sh then the BIOS is created as an MTD device mtd7:

$ /sbin/pre_action_update_bios.sh
Inside Set Gpios Dir Output

Set gpios dir output success

Pin 139 is set to Output
Inside Set Gpio low

GPIO pin is set to low

$ dmesg
[  152.510000] Creating 1 MTD partitions on "MX25L12805D":
[  152.520000] 0x000000000000-0x000001000000 : "Host SPI Flash"
[  152.540000] Configuring the Host SPI Clock to HCLK/16
$ cat /proc/mtd
dev:    size   erasesize  name
mtd0: 04000000 00010000 "fullpart"
mtd1: 001f0000 00010000 "conf"
mtd2: 001f0000 00010000 "conf"
mtd3: 03ba0000 00010000 "ec"
mtd4: 022d6000 00010000 "root"
mtd5: 00070000 00010000 "dre"
mtd6: 00512000 00010000 "www"
mtd7: 01000000 00010000 "Host SPI Flash"

For some reason reading /dev/mtdblock7 or /dev/mtd7 just gives repeating 0x04 bytes:

sysadmin [/var/tmp]# dd if=/dev/mtdblock7 bs=65535 count=1 | hexdump -C
00000000  04 04 04 04 04 04 04 04  04 04 04 04 04 04 04 04  |................|
*
0000fff0
1+0 records in
1+0 records out
65535 bytes (64.0KB) copied, 0.010000 seconds, 6.2MB/s

I haven't figured out why yet.

Just throwing my few cents inhere regarding the slimsas 8i cables, i got mine from some random aliexpress seller selling a slimsas 8i to 2x m.2 card and having the option to buy the cable aswell, bought both and ive been running a m.2 in that board connected to that 8i port no problem.
Its only 4 lanes but it does work the other m.2 ofc does not, because no bifurcation :/

I might buy another of these boards just for testing and might try bios modding the original updated bios to see if bifurcation can somehow be enabled, also might try and solder that second 8i port in aswell, i do have a hot air gun and semi good soldering skills, so might be interesting to see if that works.

I struggled to find the correct connector for the second SAS 8i port, the Foxconn one is impossible to obtain as it's discontinued and no one seems to have stock at a reasonable price, but the Amphenol U10B074200T is also compatible.

DM me if you're interested, I ordered 20 total to be economical with the fixed costs of ordering via a forwarding agent from Taobao.

I expect soldering will be difficult, since the pads are entirely under the plastic connector housing.

 

[Gymnae]

Chielos had a success. I ordered these adapters too and I’ll try connecting a 10g card in a few days.

Edit: Looks like PCIe expects impedance of 85 Ohms, and SAS 100 Ohms.

https://www.mouser.com/pdfDocs/987652-0234.pdf

I think 85 Ohms cables with are the correct ones in this case.

The linked adapter is currently not shipping to Europe due to missing EPR. I’m in contact with the seller on alibaba and aliexpress and they are working on a solution.
These seem the best bet so far.

I struggled to find the correct connector for the second SAS 8i port, the Foxconn one is impossible to obtain as it's discontinued and no one seems to have stock at a reasonable price, but the Amphenol U10B074200T is also compatible.

DM me if you're interested, I ordered 20 total to be economical with the fixed costs of ordering via a forwarding agent from Taobao.

I expect soldering will be difficult, since the pads are entirely under the plastic connector housing.

I was also thinking about soldering a second connector and thought about either hot air gun or oven. Need to test the temperature their solder melts at…

 

[janek202]

As far as I understand, only some cards do work in the pcie adapter. Maybe the 8i slot is defective on some boards, could be a reason they sell them so cheap, and the seller RAM-KÖNIG only mention 1x SlimSAS port in the description in the eBay listing and their website. And only a few full servers was listed for sale by piospartslap, not to mention they have (as of this post) 1164x SlimSAS 8i cables for sale on their website, that originated (presumably) from one og these GPU servers.

Well only speculations, but still hope all these adapters I ordered, plus cables don't end up in a drawer, don't currently have any other motherboards I can use them on.

EDIT: The SlimSAS 8i cable that was used in these GPU servers, was 100ohm cables. But as you mention, 85ohm is for pcie and 100ohm for SAS spec. I ordered both just to test and see what works best, if it works at all .

I finally experimented with PCIe. I have two Gigabyte boards and tested two 10g network cards: Mellanox ConnectX-3 Single SFP+ and Intel X710-DA2. Both of these cards got detected on both of these boards, but not perfectly. I'm using a Ceacent adapter and an 85Ohm cable.

On the first board CX-3 got downgraded to PCIe 3.0 x2 (instead of x4) , X710 to PCIe 3.0 x4 (instead of x8). So both cards ran with half of their lanes.
Second board detected the CX-3 properly as 3.0 x4, (attached a screenshot), but X710 again got detected as x4. This board also has ASPM explicitly disabled via a kernel parameter.

I didn't have a chance to try any stress tests yet, like iperf3 to test the stability. So far I didn't notice any PCI errors in dmesg.
I'm not sure if my boards have issues, or the cable is not compatible (or faulty).

I would be very interested to hear back from you. Especially if 100 and 85 Ohms make any difference.

 

[Gymnae]

I have tested the cable which was used with these boards as well as an asus one. did not work. i will measure the resistance of both later tonight

 

[ilmari]

Greetings!
I also bought this board and my plan is to replace my current nas/home server with this. I'd like to try object detection on my NVR and use GPU for that (my current server doesn't use object detection and doesn't have a GPU at all). If I have understood correctly, I could use either the oculink or the m.2 for the gpu, with an adapter. Which one would be preferred/cheaper way? Could someone elaborate this for me? And which adapter(s) I should buy. As this is merely a test, I would prefer not breaking a bank for this experiment. Of course I may find some other use for the GPU if the NVR use case is not worth it. My initial plan is to use GT610 because I happen to have one laying around. Thanks!

 

[hmartin]

If I have understood correctly, I could use either the oculink or the m.2 for the gpu, with an adapter. Which one would be preferred/cheaper way? Could someone elaborate this for me? And which adapter(s) I should buy.

M.2 to PCIe x4 is by far the cheapest, you can get the adapters for under $5

Be advised that the PCIe on the board is not oculink, but SlimSAS 4i and 8i. For the 4i you can expect to pay around $25-30 for cable and breakout board. 8i closer to $40. They have been linked to earlier in the thread.

 

[ilmari]

Thanks. I need the slimsas 4i for additional sata ports. For the slimsas 8i and the m.2 I don't have any use right now, so the choice for the gpu is between them. From performance point of view, is there any difference between the slimsas 8i and m.2 for the GPU?

 

[hmartin]

From performance point of view, is there any difference between the slimsas 8i and m.2 for the GPU?

PCIe x8 versus x4, but for the GPU you plan to install and your use case it won't make any practical difference.

 

[ilmari]

Thanks. I decided to go with sff-8654 to pcie and ordered the same board and cable you mentioned earlier in this thread. I guess it would be better to utilize the sas port for the gpu and spare the m.2 port for later upgrades (nvme drive, 10G nic or something else). Server hardware is totally new to me, as all my current and past servers have been on pc hardware.

 

[hmartin]

Thanks. I decided to go with sff-8654 to pcie and ordered the same board and cable you mentioned earlier in this thread.

Just so you're aware, the PCIe riser doesn't come with a power cable, and uses Micro-Fit 3.0 for power. PM me your details and I can mail you a power cable for it.

Some sellers also send the wrong SlimSAS cable. The cable I received had a different head style with a plastic hood that I had to delicately remove with a box cutter.