Proxmox homelab advice requested - low power usage, mid performance - Which TinyMiniMicro?

[Marsh]

Few years ago,
I brought a lot of IBM 49Y4242 49Y4241 Intel I340-T4 Quad Port Ethernet for $15 to $20 each.
I am still using it for now. It is not I350 , but it works for homelab.

At this time, I won't buy any more 1gb nic, at lease 2.5gbe nic. Avoid Intel I225v, look for Intel I226 model of 2.5gbe nic

For 10gb NIC been using and buying
Supermicro AOC-STGN-I2S 10 Gigabit Ethernet Adapter

 

[Marsh]

At this time, I won't buy any more 1gb nic, at lease 2.5gbe nic. Avoid Intel I225v, look for Intel I226 model of 2.5gbe nic

What I mean is , there lots of realtek 2.5gbe cards on Amazon / Ebay, price cheaper than Intel 1gb NIC.
Even USB Realtek 2.5 nic works OK.

 

[T_Minus]

What I mean is , there lots of realtek 2.5gbe cards on Amazon / Ebay, price cheaper than Intel 1gb NIC.
Even USB Realtek 2.5 nic works OK.

Which realtek 2.5 NIC are you using and which are they working on?
Proxmox? Truenas scale?Core?
Windows?

Curious

 

[bigfellasdad]

Which realtek 2.5 NIC are you using and which are they working on?
Proxmox? Truenas scale?Core?
Windows?

Curious

My pcie dual 2.5gb is this one. https://www.amazon.co.uk/dp/B07XJ8CMQX?psc=1&ref=ppx_yo2ov_dt_b_product_details
USB adapters, 2 of these https://www.amazon.co.uk/gp/product/B0856V9FDH/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1
For the USB adapters they work in proxmox, openwrt, pfsense and of course windows. I had to disable powertop settings for the usb dongles as they didnt wake up when needed, fine after changing that, I dont have that issue with the pcie card.

 

[Marsh]

Realtek RTL8125B , Proxmox , Debain based OS and Windows
Next project is to replaced internal WiFI m2 card with
M.2 A Key and E Key to 1 Port 2.5G 100Mbps Gigabit Ethernet NIC Network Card | eBay
or
https://www.aliexpress.us/item/3256804695365077.html

 

[bigfellasdad]

4 x Dell 7070 I5-9500T , 32gb ram mini cluster + 1 Lenovo M720q ( I5-8500 ) + 8 port switch
Proxmox 7.2 idle
All Dell 7070 mini + M720q + switch = 14-18w

Hi, im really intrigued on how your rig pulls so little power. My qnap QSW-1105-5T alone pulls 6w with only 3 connections, are you reading the power with the dell's sleeping, with WoL enabled?

 

[bigfellasdad]

Realtek RTL8125B , Proxmox , Debain based OS and Windows
Next project is to replaced internal WiFI m2 card with
M.2 A Key and E Key to 1 Port 2.5G 100Mbps Gigabit Ethernet NIC Network Card | eBay
or
https://www.aliexpress.us/item/3256804695365077.html

I looked at that too initially but i couldnt find a low profile bracket for it, would you replace the onboard nic for that?

 

[Parallax]

I looked at that too initially but i couldnt find a low profile bracket for it, would you replace the onboard nic for that?

I have the 2.5GbE m.2 A/E key NIC on order. I think I should be able to punch out one of the rear ports of my M75q-2 and screw it in there. Since there's no riser in these I'm not losing out on anything and I should be able to fit the SSD and caddy still.

 

[Marsh]

Hi, im really intrigued on how your rig pulls so little power. My qnap QSW-1105-5T alone pulls 6w with only 3 connections, are you reading the power with the dell's sleeping, with WoL enabled?

Me too, I agree with you.
I had to get a camera to take couple of pictures , one picture shown 14w, another picture shown 18w.

I do not believed it is true measurement due to the ultra cheap kill-a-watt meter,
I don't think the meter is accurate in the low end range.

May be true power consumption is 5w to 7w more for the whole cluster. (19w to 25w).

 

[denis_tmm]

Thank you for all your NIC suggestions. I'm going through them later on. For now I have now bought the Dell Optiplex 5060 SFF i5-8500 for 170€. Not quite as cheap as in the US but it is OK for me. I will get now 2x16GB of RAM and SSD + M2 SSD storage + PCIx 1x2,5G NIC.

Any suggestions for the SSD stuff regarding power usage?

 

[Marsh]

In case anyone interest
Low end of Dell OptiPlex mini
Dell OptiPlex 3060 Micro i5-8500T @ 2.10GHz 8GB RAM No HDD/OS $113 each with BO , 2 available

Dell OptiPlex 3060 Micro i5-8500T @ 2.10GHz 8GB RAM No HDD/OS DH549 | eBay

 

[denis_tmm]

Ok one question from the guy that hasn't done hardware upgrades for some years.

The "Dell Optiplex 5060 SFF" does have a M.2 slot with Gen3 PCIe x4 NVMe support. (like e.g. the https://de.transcend-info.com/product/internal-ssd/mte110s-112s)

From what I have read are the NVMe SSDs not very power efficient. Is this right? Is it possible to use the M.2 SATA drives (which can be pretty power efficient) in such a NVMe slot?

I wanted to have the Proxmox installation on the M.2 slot and the data for the VMs/Containers on one or two "normal" SATA3 SSDs.

Or would you go another approach in this machine from a power usage standpoint?

 

[denis_tmm]

And I'd like to add one more question. Would you rather go with "standard" ext4 when using only one single SATA3 disk? Or would you create a LVM? Or what would be your approach when just using one single NVMe (WD Red) for the system + one SATA3 SSD (WD Red as well) for the VMs/containers.

Backups will be done via NFS for the moment to my Synology NAS that is already sitting around.

 

[bigfellasdad]

I have the 2.5GbE m.2 A/E key NIC on order. I think I should be able to punch out one of the rear ports of my M75q-2 and screw it in there. Since there's no riser in these I'm not losing out on anything and I should be able to fit the SSD and caddy still.

If you could let me know your findings on this please.

 

[zer0sum]

I would aim for a Lenovo m720q as they can be found for as low as ~$80 (without cpu) or ~$130 (i3/i5) off Ebay in the US.
I'm not sure on Europe, but at that price even including shipping to you it would be really cheap

It can take 64G ram, 1 x nvme, 1 x sata and has a low profile PCIe slot that you can add a quad 1G or dual 10G network card.
The built in lan port supports vPro so you get poor man IPMI.

I don't think they can be beaten really

 

[Parallax]

If you could let me know your findings on this please.

Very straightforward, my biggest problem was the m.2 screw head got stripped trying to get the WiFi card out. After sorting that out I just put in the NIC card, punched out one of the panels in the rear baffle, and the LAN port screwed in to the space nicely. The whole thing looks good and works well.

It's Realtek which some people (and ESXi) don't like, but I couldn't see an Intel alternative on Aliexpress and Intel's 2.5GbE chips haven't been great successes either.

 

[Parallax]

And I'd like to add one more question. Would you rather go with "standard" ext4 when using only one single SATA3 disk? Or would you create a LVM? Or what would be your approach when just using one single NVMe (WD Red) for the system + one SATA3 SSD (WD Red as well) for the VMs/containers.

I do LVMs because there's flexibility to move it to a new drive and grow the partitions pretty easily.

 

[bigfellasdad]

Very straightforward, my biggest problem was the m.2 screw head got stripped trying to get the WiFi card out. After sorting that out I just put in the NIC card, punched out one of the panels in the rear baffle, and the LAN port screwed in to the space nicely. The whole thing looks good and works well.

It's Realtek which some people (and ESXi) don't like, but I couldn't see an Intel alternative on Aliexpress and Intel's 2.5GbE chips haven't been great successes either.

Thanks very much, i think ill order one or two then. I have had no problems with the usb realtek nics, other than in debian/proxmox using the wrong kernel module, causing half duplex (easy fix). The mPCIe option looks much cleaner though. thanks again

 

[Parallax]

The mPCIe option looks much cleaner though. thanks again

You're welcome! Let us know how you get on.

 

[denis_tmm]

@Marsh

Does your Dell 5060 SFF run smoothly at the moment. I have done an update in the last week. And I'm getting these kinds of kernel panics.

Feb 28 07:44:19 pve kernel: BUG: unable to handle page fault for address: ffffffff8e88a940
Feb 28 07:44:19 pve kernel: #PF: supervisor write access in kernel mode
Feb 28 07:44:19 pve kernel: #PF: error_code(0x0002) - not-present page
Feb 28 07:44:19 pve kernel: PGD 6d8415067 P4D 6d8415067 PUD 6d8416063 PMD 10eb8a063 PTE 800ffff927775062
Feb 28 07:44:19 pve kernel: Oops: 0002 [#1] SMP PTI
Feb 28 07:44:19 pve kernel: CPU: 4 PID: 678197 Comm: z_rd_int Tainted: P           O      5.15.85-1-pve #1
Feb 28 07:44:19 pve kernel: Hardware name: Dell Inc. OptiPlex 5060/0654JC, BIOS 1.7.1 07/02/2020
Feb 28 07:44:19 pve kernel: RIP: 0010:native_queued_spin_lock_slowpath+0x1e1/0x240
Feb 28 07:44:19 pve kernel: Code: 41 89 ce 44 0f b7 e8 41 83 ee 01 49 c1 e5 05 4d 63 f6 49 81 c5 40 19 03 00 49 81 fe ff 1f 00 00 77 49 4e 03 2c f5 e0 0a ec 8d <4d> 89 65 00 41 8b 44 24 08 85 c0 75 0b f3 90 41 8b 44 24 08 85 c0
Feb 28 07:44:19 pve kernel: RSP: 0018:ffffa2552192bc28 EFLAGS: 00010282
Feb 28 07:44:19 pve kernel: RAX: 0000000000000000 RBX: ffff8f41b47bf0c0 RCX: 0000000000000010
Feb 28 07:44:19 pve kernel: RDX: 0000000000140000 RSI: 0000000000140000 RDI: ffff8f41b47bf0c0
Feb 28 07:44:19 pve kernel: RBP: ffffa2552192bc50 R08: 0000000000000001 R09: 9ae16a3b2f90404f
Feb 28 07:44:19 pve kernel: R10: 0000000000000000 R11: ffff8f4155cc2000 R12: ffff8f469c531940
Feb 28 07:44:19 pve kernel: R13: ffffffff8e88a940 R14: 000000000000000f R15: 0000000000140000
Feb 28 07:44:19 pve kernel: FS:  0000000000000000(0000) GS:ffff8f469c500000(0000) knlGS:0000000000000000
Feb 28 07:44:19 pve kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 28 07:44:19 pve kernel: CR2: ffffffff8e88a940 CR3: 00000006d8410004 CR4: 00000000003726e0
Feb 28 07:44:19 pve kernel: Call Trace: (the whole file is attached)

I haven't changed anything. But what I see is something like that in the syslog: Device: /dev/sdb [SAT], SMART Usage Attribute: 190 Drive_Temperature changed from 62 to 63

But when looking at hddtemp they do report me 36 and 38 degrees celcius. (two Intel SSD D3-S4510 480GB 2.5" SATA)

These are the other temperatures:

coretemp-isa-0000
Adapter: ISA adapter
Package id 0:  +33.0°C  (high = +82.0°C, crit = +100.0°C)
Core 0:        +33.0°C  (high = +82.0°C, crit = +100.0°C)
Core 1:        +30.0°C  (high = +82.0°C, crit = +100.0°C)
Core 2:        +29.0°C  (high = +82.0°C, crit = +100.0°C)
Core 3:        +30.0°C  (high = +82.0°C, crit = +100.0°C)
Core 4:        +30.0°C  (high = +82.0°C, crit = +100.0°C)
Core 5:        +30.0°C  (high = +82.0°C, crit = +100.0°C)

acpitz-acpi-0
Adapter: ACPI interface
temp1:        +27.8°C  (crit = +119.0°C)

pch_cannonlake-virtual-0
Adapter: Virtual device
temp1:        +39.0°C

Perhaps you go an idea what could have happened.