EU Gigabyte Mainboard MC12-LE0 Re1.0 AMD B550 AM4 Ryzen

[ook512]

3.) Can someone post how IOMMU grouping of device is done on this MB? I need passtrough SATA controller to TrueNAS VM and I would like if it share . In worst case I will use SATA(I need it only for 2x HDD) in M2 slot...

IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 1 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 2 00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 3 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 4 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 5 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 6 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 7 00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 8 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 9 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 10 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 11 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 12 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
IOMMU Group 12 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 13 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 0 [1022:1440]
IOMMU Group 13 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 1 [1022:1441]
IOMMU Group 13 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 2 [1022:1442]
IOMMU Group 13 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 3 [1022:1443]
IOMMU Group 13 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 4 [1022:1444]
IOMMU Group 13 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 5 [1022:1445]
IOMMU Group 13 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 6 [1022:1446]
IOMMU Group 13 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 7 [1022:1447]
IOMMU Group 14 01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
IOMMU Group 15 02:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 500 Series Chipset USB 3.1 XHCI Controller [1022:43ee]
IOMMU Group 15 02:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 500 Series Chipset SATA Controller [1022:43eb]
IOMMU Group 15 02:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 500 Series Chipset Switch Upstream Port [1022:43e9]
IOMMU Group 15 03:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
IOMMU Group 15 03:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
IOMMU Group 15 03:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
IOMMU Group 15 03:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
IOMMU Group 15 04:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 04)
IOMMU Group 15 05:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 41)
IOMMU Group 15 06:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
IOMMU Group 15 07:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU Group 15 08:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU Group 16 09:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1070] [10de:1b81] (rev a1)
IOMMU Group 16 09:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
IOMMU Group 17 0a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU Group 18 0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 19 0b:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
IOMMU Group 20 0b:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 21 0b:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]

Group 15 contains everything connected to the chipset (SATA, internal/fron usb, ethernet, motherboard m.2, BMC), passing the whole group to a VM does not seem like a good idea.

Verified working PCIE passthrough on proxmox:
Group 16 pcie x16 slot: gpu
Group 20 back usb ports


Not tested, but should work with passthrough: pcie x4 slot (used for a m.2 in a pcie x4 adapter, Group 14, used on the host).

 

[SlowmoDK]

I could not pass through a ASM1166 m2 6 sata expansion card, in the onboard m2 slot

Same IOMMU group as SATA and USB

 

[SlowmoDK]

1.) What are supported CPU-s with ECC support? Im think to buy AMD Ryzen 5 5600 or 5600X because I can get new one for better price than used on ebay and if I compare CPU score with current Epyc 7601 CPU it have a lot of better single core score. Single core performance is important in my case because Sophos XG firewall when decode SSL/TLS traffic can decode single connection only on single core...
If Im read spec corectlly it support ECC and PCIe4.0 . Will this CPU support running 4x4x4x4x and 4x4x8x bifurcation? Im currently not sure which option will need...
2.) Is mixing pcie3.0&4.0 in bifurcation possible? For example running intel x710-da2(PCIe 3.0 8x) and 2x Samsung PM9A3(PCIe4.0 4x) ? What will happen in this case? All devices run in PCIe3.0 mode? What will this mean for PM9A3? Will speed be limited to 4GB/s or to half of max for example 1,92GB device instead of Read: 6800/2700 to 3400/1350 or will be limited to 4000/2700?
3.) Can someone post how IOMMU grouping of device is done on this MB? I need passtrough SATA controller to TrueNAS VM and I would like if it share . In worst case I will use SATA(I need it only for 2x HDD) in M2 slot...

1. Yes, both bifurcations methods and ECC is supported

2. Yes, it's possible to mix Gens/Speeds

3. Onboard SATA and NVMe can't, use x16 slot with bifurcation or the 4x slot for passthrough

 

[PANiCnz]

If I'm only using 1 or 2 DIMM's are there preferred slots I should? Trying to boot with a 3600 and 2 x 4GB stick of RAM for testing purposes and it wont post. IPMI is accessible.

Ignore that, figured it out. A single DIMM in DIMMA1 worked for me.

Trying to update the BIOS via IPMI, have followed the guide but now its stuck on "Uploading 100%". Is it just a matter of being patient or has something gone wrong?

 

[crlt]

If I'm only using 1 or 2 DIMM's are there preferred slots I should? Trying to boot with a 3600 and 2 x 4GB stick of RAM for testing purposes and it wont post. IPMI is accessible.

Ignore that, figured it out. A single DIMM in DIMMA1 worked for me.

Trying to update the BIOS via IPMI, have followed the guide but now its stuck on "Uploading 100%". Is it just a matter of being patient or has something gone wrong?

When you update the BMC and go through the prompts it will show upload 100% and sit there, and then you have to click the green button below to begin flashing.

When you update the BIOS there was no second prompt and all of my boards so far (2/4) reach upload 100% within 10 seconds and then it immediately starts the flashing 1%.. 2%... etc.

I guess make sure you selected BMC for BMC updates and BIOS for BIOS updates. Sounds basic but that's all I've got I'm afraid.

 

[PANiCnz]

When you update the BMC and go through the prompts it will show upload 100% and sit there, and then you have to click the green button below to begin flashing.

Thanks, that's the bit I was missing!

 

[crlt]

Here are the steps I followed to update my boards. I did BMC first and then BIOS.

BMC:

1. Connect to the BMC/IPMI port (the one above the two USB ports)
2. Turn on the board with or without CPU/RAM (either way worked for me)
3. Check your router or DHCP server for the IP address of the machine and navigate to that URL in your web browser (https)... Green blinking light on the board indicates the BMC is ready but I found it usually takes a few more minutes to get an IP and have the WebUI ready.
4. Login with the username "admin" and the password printed on the sticker on the box or on the sticker attached to the PCIEx16 slot.
5. Navigate to Power Control --> Power Off to turn off the host
6. Navigate to Maintenance --> Firmware update --> upload the image for the BMC (rom.ima_enc) and click Start Firmware Update. Select BMC from the dropdown menu, preserve all configuration, and then click the green proceed to flash button. The upload progress will be shown and once it reaches 100%, you must scroll down and click the "flash selection options" button, no need to check mark any of the boxes.
7. Upon success it should automatically restart the BMC. If it fails uploading - just try the process again.

BIOS:

1. If you didn't do this yet, Insert only one RAM stick into slot A1 and insert CPU (my Ryzen Pro 4650G worked out of the box, I think 5xxx series do not work until the BIOS is updated so you will need something like a Ryzen 3600)
2. Navigate to Power Control --> Power On which will turn on the host which must be done to flash the BIOS
3. Navigate to Maintenance --> Firmware update --> Dump Bios (optional - but better safe than sorry)
4. Navigate to Maintenance --> Firmware update --> upload the image for the BIOS (image.RBU) and click start firmware update. Select BIOS from the dropdown menu and then click Proceed to flash. It should automatically upload and then begin flashing.
5. Upon success, you may notice that the Firmware Inventory still shows version F06 (or whatever older version you have), you need to power cycle the host which can be done through the power control menu and afterwards it will show the version you just flashed.

BMC and BIOS available from Gigabyte website under the Downloads sections. I only flashed F13 but from reading the past posts, the beta version F14 fixes a bug where the iGPU does not work.
BMC: MC12-LE0 (rev. 1.x) | Workstation Motherboard - GIGABYTE Global
BIOS: MC12-LE0 (rev. 1.x) | Workstation Motherboard - GIGABYTE Global

If it's useful for anyone, these RAM models worked for me although I don't have any others to test.

• MTA18ASF4G72AZ-3G2RL - Micron ECC UDIMM 32GB (i have many variants of this and they worked)
• M378A1G44BB0-CWE - Samsung Non-ECC UDIMM 8GB

Since it gets asked a lot, if you don't see your CPU or RAM listed correctly under system inventory, that means it was not recognized or is not compatible.

My boxes came with the serial number taped over, for anyone with the same issue, the serial number is also printed on the 24PIN ATX power connector, as well as being available within the BIOS itself. My boards still have anywhere from 4-8 months of warranty left although I am unsure if Gigabyte will ask for proof of purchase to process it if/when needed. You can check your warranty here: Identify your product - GIGABYTE Service

 

[_Dejan_]

Thanks for all that informations about IOMMU.
Looks like only option is using SATA cart in x4 slot. Is little offtopic but does maybe someone know simple/cheap solution like DAS which can be connected to 2 hosts and I can use it in TrueNAS VM? Im asking this because I looking option to eliminate moving HDD's problem from one host to another when I wan't run TrueNAS VM on another host... There is no problem if VM need to be restarted to see disks... Simple 2 or 4 bay solution will be good. If is rack variant it will be perfect...

For USB I see that I can use Group 20 and passthrough it to VM. Maybe I will go by USB Network device and by that way share USB over host and eliminate problem with moving USB from one host to another...

 

[ook512]

Looks like only option is using SATA cart in x4 slot.

It should be possilbe to pass raw block device to VM (e.g. using virtio-scsi).
Not sure about reliability in TrueNAS (FreeBSD), but might work in TrueNAS SCALE (Linux), as there are usually no issues with virtio block devices in Linux VMs.
It would allow using onboard SATA ports for TrueNAS disks, keeping PCIE x4 available for other things.

For USB I see that I can use Group 20 and passthrough it to VM. Maybe I will go by USB Network device and by that way share USB over host and eliminate problem with moving USB from one host to another...

For completeness, in addition to PCIE passthrough, Proxmox supports passing individual USB devices.
It works at least for a USB Bluetooth donge connected to the front USB panel (for headphones), did not test anything with high throughput (SSD, network card).
The same VM also gets Group 20 to pass back USB controller PCIE device (used for USB Soundcard and USB Hub), so combinining passthrough of whole PCIE USB controller and passthrough of individual USB devices works.

Individual USB device passthrough could be used to move a front USB-connected disk between different VMs (on the same host), though last time I tried using FreeNAS with USB3 external HDDs, it was unreliable (disk took too long to spin up, FreeNAS timed out), but that was years ago.

 

[etorix]

Virtualised ZFS requires passing through the whole SATA controller, not individual disks.
@_Dejan_ may consider running TrueNAS bare metal and hosting the other VMs within TrueNAS.

 

[Crash_0verride]

The bundle with heatsink listing has been dropped to 47,99€ btw.

Not sure if il use that heatsink but il probably grab 5 of these instead of waiting for the standalone board to be dropped in price again.

Me personally opted for low profile thermalright solution Thermalright AXP90 X47.
Recommend.
Cons - not possible to take it of (for reapply thermal paste for example) without completely remove mb. due to specific mounting. but thats quiet common for am4.

 

[_Dejan_]

@etorix Then I again need two additional servers and change switch because current one have only 4xSFP+ ports... And running 4 servers(2x proxmox + 2x TrueNAS) will again increase power usage... So this is not option...
Im looking how to make as less power usage as possible but still have redundancy which I don't nave right now...

My plan is to run 2 of this MB with 5600 CPU and on each run 64GB ECC RAM(2x32GB). about storage Im not sure what I will use. Current server use 2xPM983 960GB M.2 NVME + 2x 8TB WD RED HDD.
Im think to use in each server 2x PM863 or PM893 SATA SSD's because they have better power consumption and Im not sure if I really need NVME's ...
For network card I have plan to use Intel X710-DA2 or TrendNet 10GB SFP+ card because I really need only ons SFP+ for LAN(All is seperated with VLANs') ...
For running TrueNAS there is option to run 2 TrueNAS VM's and make remote replication between them. If main one fail is not problem and nothing important depend of them I just don't want lose data... There is really not a lot changes in one week... Or if exist some simple JBOD/DAS device which can be connected to 2 hosts by some cable's and when I failover VM to another host it can connect to attached drives...

Right now I pass whole SATA controller because current MB have IOMMU gorups very well organized:

 

[_Dejan_]

@mackspain @ook512 @SlowmoDK I found on german forum hardwareluxx.de that user "MultumInParvo" post solution for IOMMU groups...

In BIOS you need enable ACS (Access Control Services) under NBIO Common Options.

Before:


After:


Then you need apply in Proxmox GRUB config ACS patch.

GRUB_CMDLINE_LINUX_DEFAULT="quiet pcie_acs_override=downstream,multifunction"

I don't have board to test this but is someone have option please do it and report back...

 

[Cruzader]

Me personally opted for low profile thermalright solution Thermalright AXP90 X47.
Recommend.
Cons - not possible to take it of (for reapply thermal paste for example) without completely remove mb. due to specific mounting. but thats quiet common for am4.

Should be solid if you need the space above it, im not as big a fan of the top facing fans in cases not meant for that direction of airflow.

Im building 4 units that will be just compute, so pretty much just empty case beyond mobo/cards so plenty of space for a large cooler in those.

The other 5 will be Ceph baremetal, might just end up using their low profile bundled coolers tbh since not gone be much heat/load anyhow and not as much space above cpu in them.
Going with this rough layout with about 250mm depth on main unit itself so i can print it as 2 halves.

 

[mackspain]

Very interesting _Dejan_

Right now I cannot find 'acs enable' on my 5650G PRO

I am on the F14 BIOS and 126121 firmware, maybe an update is required.



Edit: So some research later, in the situation of the APUs (like my 5650g), it seems they do not have VFIO functionality, but non APU CPU should give the 'acs enable' capability. I am sure someone with more processor options than me can confirm

 

[_Dejan_]

Yes maybe your CPU/APU do not support it... Maybe in latest BIOS is function removed(I don't think that they will remove it) because one in screenshot looks like is from 2022

 

[boromartin]

Hi,

Does this board support a Ryzen 5500? The QVL does not mention it, but it is a 5000 Series CPU and the board has a B550 chipset.
The board does not POST, so I could not update the BIOS, so I assume it still has the F6 version

 

[Cute_SC2]

Welp I'm guilty now, replacing all my old servers with something more energy efficient.

 

[etorix]

Yep! Should run somewhat cooler than a pair of 8272CL with Optane DCPMM, and lighter on the energy bill. (Good luck with the sale, by the way.)

 

[Cute_SC2]

Yep Exactly this, around 400w Idle is no joke. Have a bunch of Ryzen systems that I can turn off when not needed is far better.
Will additionally go for a 96 Core Epyc Genoa in the future when the prices drop .