Search results

Gotta say my new cluster is super reliable. Migration happens in blink of an eye and seamless without any interruption...CEPH is really awesome! Had some kernel updates to my 3 nodes today. I reboot each node individually. It migrated all VMs automatically in seconds and reboot without any...

nice rack lol Did you setup CEPH yet? What networking config you use? My bonded pair is working great. No issues to report in the last week

Glad you got it stable now. I never had any of these issues. Before deploying any nodes I did 24hrs burn-in MemTest. Only after it PASSED, then I install fresh PVE8.2-2. Afterward, applied the Intel microcode. I actually ran single node for over 2 weeks for testing before deploying my 3 nodes...

The Samsung PM983a NVMe are holding steady at 53-55C. That's with all 3 nodes inside my 19" rack with less than optimal airflow. The Kingston 1TB with heatsink is even lower at ~35C. Would be great if we have a couple mm to add heatsinks for slot2/3. I've been keeping an eye on CPU thermals and...

I bought mine months ago.. all sold out now

Well, I gave in and spent this afternoon rebuilding my cluster from scratch. Reloaded fresh PVE 8.2-2 to all 3x MS-01. Then join them to cluster and setup CEPH. Took a couple hours to restore all my VMs/LXCs. They're all working incredibly well. I tested by shutdown a node while pinging VM...

That's troubling to hear two nodes shutdown randomly. I'm beginning to reconsider setting up CEPH at all due to the greater complexity and greater resources requirement. I've been running my 3 nodes MS-01 with Local ZFS storage and VMs replicating between the nodes with HA. That's been working...

@Techrantula are your switches L3 capable of inter-vlan routing? I'm using USW-Pro-MAX-24-POE with 8x 2.5G ports and USW-Pro-Aggregation with 28x 10G ports, both are L3 switches. So I have plenty of ports to do LACP link aggregation and the switches can offload hardware routing to give my UDM-SE...

I would avoid U.2 unless you need the larger capacity. They run hot and require more power draw. I opted for Samsung PM983a 1.88TB NVMe 22110 drives. Got them brand new on eBay for $75 each. Bought 6 (two for each nodes). They're running nice and cool. Both staying at ~52C with just onboard fan...

I haven't configured AMT on my MS-01s yet. Maybe that's why I don't have any issues. I'm also use L3 switches with better support for inter-VLAN routing.

What weirdness are you seeing with the AMT vPro 2.5G port? Mine been LACP bonded for couple weeks now, I'm no seeing anything unusual. What switches and router are you using? You got me paranoid. So I went to test my setup. Pulled network cable between each of two alternating 2.5G ports while...

Yes, I did the microcode update with TTeck scripts. So I use his PVE scripts for everything. It's saves me time. I don't know if 8.2 comes with microcode update, probably not. I will do fresh PVE 8.2 install this weekend. I got Plex running now and people are watching TV so I can't take down my...

I don't think CEPH will work with RAID storage. I think you create OSDs for each NVMe and let CEPH manage them. I also have Synology NAS for backup stores. I may create "fast" pool for slot 2 NVMe since those are GEN3x4. And "slow" pool for slot3 NVMe since those are slower GEN3x2. Then create...

PSA: if you update to latest PVE 8.2 that was release today, it will change your 10G interface names from enp2s0f0 > enp2s0f0np0, enp2s0f1 > enp2s0f1np1. This will break networking for any VMs you have using those interfaces. To fix, simply nano /etc/network/interfaces to update to new names. I...

I ordered 3x 12900H kits with 2x 16GB CRUCIAL CT16G56C46S5. I dont need 96GB for my setup but may upgrade later.

Welcome! Hoping this thread will provide some of us tips and tricks for setting up networking properly for CEPH. I have received my 3rd MS-01 on Monday. Very fast shipping as I got notice just the Friday before from DHL. Looks like Minisforum is finally cranking these puppies out in numbers. So...

Planning to do CEPH pools. I haven't yet setup it haven't gotten any free time. But one of my nodes is using 2x PM983a in ZFS mirrored just fine.

yes, the 80mm post is non-removable on slot1. I tried my damnest with pliers but couldn't remove it. Without it removed, the 22100 NVMe sit a couple mm too high hitting the fan shroud making vibrating noise. Did want to force it risking breaking something. I'm fine with Gen3x2. That still rated...

Slot1 doesn't fit 22110 NVMe drive isn't that correct? I think slot1 is only for 2280 or U.2 drives. I thought about swapping them but unfortunately with my longer 22110 enterprise drives, they can only fit slot2 & 3. The Kingston will be PVE OS drive. And the two PM983a will be CEPH OSDs. Not...

Yup, already research TB Ring Network. Seems to work OK for people who have done it...but still a bit of an experiment at this point. I think I'll just keep it simple with bonded interfaces and VLAN tagging.