Proxmox homelab advice requested - low power usage, mid performance - Which TinyMiniMicro?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Marsh

Moderator
May 12, 2013
2,642
1,496
113
Have not done update yet.

I could update one Dell 5060 pve server today.
What is the condition for the kernel panic?
 

denis_tmm

New Member
Jan 24, 2023
20
4
3
Every backup (no PBS) fails no matter if it goes to the internal zfs pool or to an external NFS target and no matter which VM. It does the backup till about 10-20% and crashes.
 

Marsh

Moderator
May 12, 2013
2,642
1,496
113
Stay tune.
Updating PVE now, will perform backup and restore to a NFS dump server.
 

Marsh

Moderator
May 12, 2013
2,642
1,496
113
No using ZFS , just lvm with Intel P3605 nvme , target nfs server

root@pve1:~# pveversion
pve-manager/7.3-6/723bb6ec (running kernel: 5.15.85-1-pve)

Restore Ubuntu 20 VM , VM started ok
then perform backup
INFO: starting new backup job: vzdump 1004 --notes-template '{{guestname}}' --compress zstd --storage XPEN6 --remove 0 --node pve1 --mode stop
INFO: Starting Backup of VM 1004 (qemu)
INFO: Backup started at 2023-02-28 10:26:42
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: ansible1
INFO: include disk 'scsi0' 'local-lvm:vm-1004-disk-0' 32G
INFO: creating vzdump archive '/mnt/pve/XPEN6/dump/vzdump-qemu-1004-2023_02_28-10_26_42.vma.zst'
INFO: starting kvm to execute backup task
INFO: started backup task '1036cb35-745d-483f-be95-2bb9384c3d3b'
INFO: 2% (783.0 MiB of 32.0 GiB) in 3s, read: 261.0 MiB/s, write: 225.7 MiB/s
INFO: 4% (1.4 GiB of 32.0 GiB) in 6s, read: 232.0 MiB/s, write: 223.0 MiB/s
INFO: 6% (2.1 GiB of 32.0 GiB) in 9s, read: 238.0 MiB/s, write: 200.6 MiB/s
INFO: 20% (6.7 GiB of 32.0 GiB) in 12s, read: 1.5 GiB/s, write: 151.8 MiB/s
INFO: 22% (7.3 GiB of 32.0 GiB) in 15s, read: 217.5 MiB/s, write: 212.0 MiB/s
INFO: 25% (8.2 GiB of 32.0 GiB) in 18s, read: 315.4 MiB/s, write: 209.2 MiB/s
INFO: 28% (9.0 GiB of 32.0 GiB) in 21s, read: 275.5 MiB/s, write: 218.5 MiB/s
INFO: 44% (14.2 GiB of 32.0 GiB) in 24s, read: 1.7 GiB/s, write: 153.9 MiB/s
INFO: 62% (20.1 GiB of 32.0 GiB) in 27s, read: 2.0 GiB/s, write: 164.2 MiB/s
INFO: 76% (24.5 GiB of 32.0 GiB) in 30s, read: 1.5 GiB/s, write: 183.2 MiB/s
INFO: 88% (28.2 GiB of 32.0 GiB) in 33s, read: 1.2 GiB/s, write: 156.4 MiB/s
INFO: 95% (30.6 GiB of 32.0 GiB) in 36s, read: 806.0 MiB/s, write: 163.8 MiB/s
INFO: 100% (32.0 GiB of 32.0 GiB) in 38s, read: 734.9 MiB/s, write: 79.3 MiB/s
INFO: backup is sparse: 25.22 GiB (78%) total zero data
INFO: transferred 32.00 GiB in 38 seconds (862.3 MiB/s)
INFO: stopping kvm after backup task
INFO: archive file size: 2.45GB
INFO: adding notes to backup
INFO: Finished Backup of VM 1004 (00:01:00)
INFO: Backup finished at 2023-02-28 10:27:42
INFO: Backup job finished successfully
TASK OK
 
  • Like
Reactions: denis_tmm

denis_tmm

New Member
Jan 24, 2023
20
4
3
Thank you very much for the fast response @Marsh.

Then I don't know why it crashes.

I thought about different things.
  • The SSDs draw too much power for the power supply (the original Dell 5060 SFF comes with only one disk) - but why did it work for weeks? (I didn't install any additional stuff)
  • The SSDs get too hot - but why did it work for weeks and why are there no errors during writing e.g. some ISO downloads? I don't understand these "Feb 28 17:21:45 pve smartd[1774]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Drive_Temperature changed from 65 to 64" messages from the syslog. Since at the same time when looking at hddtemp or smartctl the temperatures are never above 40 degress
  • Memory error (I have to test it out when I got more time)
  • Update related error (perhaps ZFS related)
To be honest at the moment I have no clue I just know that it worked fine and suddenly it crashes during every backup.
 

denis_tmm

New Member
Jan 24, 2023
20
4
3
I tried now to clone a new VM from my template and this one doesn't work either. No kernel panic no whatsoever in the logs. It clones about 10% and crashes instantly. Without going to the machine and resetting it manually I can't get it back to work. When doing that it boots up normally and the ZFS pool looks fine with no errors, SSD smart values look fine with no errors. I don't get it.

I don't know where to look for the error. Are there any logs that could be interesting?
 

Marsh

Moderator
May 12, 2013
2,642
1,496
113
Memory error (I have to test it out when I got more time)

I would start running memtest overnight first.
SSD overheat , open the case , use a fan to blow cool air .
I added a 80mm inside the Dell 5060 to cool the Intel P3600 nvme.
 

denis_tmm

New Member
Jan 24, 2023
20
4
3
Yeah it seems to be a hardware error that I'm not able to identify at the moment.

I powered down the machine and removed all of the drives. So there is only memory and the CPU + PSU left. The machine still struggles even getting in the BIOS. Most of the time when starting up I'm stuck at the white DELL logo. It took me 1 hour to get to the BIOS for the first time. In the BIOS it stuck again. Even with one spare RAM stick the same behaviour. So I'm not sure what it is but it has nothing to do with the SSDs or the RAM I would guess.

1) Could the CMOS battery be empty?
2) Is the CPU dying?

It is pretty strange and I don't get what the machine is doing right now.

Power ON -> Power OFF (automatically) -> Power ON -> after some time the white DELL logo is shown (I don't get into the BIOS even when triggering F2)
 

denis_tmm

New Member
Jan 24, 2023
20
4
3
Ok so the riddle is solved. One of the brand new Crucial 16G sticks was faulty. (after two to three weeks ...) I think I was just unlucky. This was my first faulty memory since years.
 
  • Like
Reactions: Marsh

copper4eva

New Member
Apr 27, 2023
1
0
1
4 x Dell 7070 I5-9500T , 32gb ram mini cluster + 1 Lenovo M720q ( I5-8500 ) + 8 port switch
Proxmox 7.2 idle
All Dell 7070 mini + M720q + switch = 14-18w
I was not aware these micro workstations could idle so low. I mean, that's like 3w or so per machine. I thought they could only idle to like 10w or so.
Is there anything I'm missing here? Anything special you did to get them idling so low.

I also want to make a low power proxmox cluster, and was thinking of getting some optiplex SFF, which can usually idle below 20w. But if I can get some micro workstations to idle as low as yours, that seems like I should seriously consider that option. Although there is the issue that most of these micro/mini/whatevers have little to no expansion slots.
 

Marsh

Moderator
May 12, 2013
2,642
1,496
113
I am using a cheap Kill-a-Watt , most likely not measuring accurate at the low end range.

IMG_1382a.JPGIMG_1382a.JPG
 
  • Like
Reactions: T_Minus

Marsh

Moderator
May 12, 2013
2,642
1,496
113
Although there is the issue that most of these micro/mini/whatevers have little to no expansion slots.
Dell mini back panel has cut out to mount 2.5gbe connector
using A+E mini pcie adapter to achieve second Ethernet port for the cluster
 

gregoinc

New Member
Apr 27, 2023
15
3
3
Awesome thread... and awesome builds. I've been running stand alone Linux and Proxmox on Intel NuC and Lenovo M720Q units for some time, and there's no doubting the efficiency benefits. There are times when I look at the workloads I am running on my Lenovo tiny's and am impressed at how well they work.

Unfortunately there are times when my performance demands hit a wall in either CPU or RAM, so I always keep a monster server idling along running heavier workloads. This is my latest build in progress... first EPYC build, and cannot wait to see Proxmox fire up on 32 cores and 256Gb of RAM. Despite using the smaller Lenovo boxes, sometimes size matters :D:p

20230428_184225.jpg
 
  • Like
Reactions: sic0048

oldpenguin

Member
Apr 27, 2023
30
10
8
EU
North Germany 35euro cent per kWH since january 1st.
Add +10KWP solar with at least 15KW worth of lithiums and have all the fun you want between march-nov; for the rest, feel free to track battery status with a raspi and shutdown via ipmi/remote ssh and power on via WOL as you see fit. Looking at how energy rates are "dancing" nowadays, this will likely pay itself in less than 5 years (YMMV, just using my own 800-1200 KWh/mo as reference). Keep it offgrid and use as much as you can, something smells like "prosumers will likely have to pay more than its worth it" soon.
 

tonyh

New Member
Sep 21, 2023
2
0
1
Hi, I would like to replace my i3 nuc SSD + NVME + 32GB (Nuc i3 specs) with a low-power refurbished server to have an extra sata
Usually my work with Proxmox is completely homemade
- VM Home Assistant
-VM OMV
-VM Debian
-VM Windows 10
- LXC Apache
The idea would be this
M2 with Proxmox, VM and lxc
Raid 1 ZFS for data
I was reading about the Dell Optiplex 5060 SFF
Could this be a good choice for the TDP and configuration chosen?
Thank you for your time