Dell R510 Experiences

[mobilenvidia]

Decided to post how I'm getting on with this venerably old Server

Server in question, I've had this for while and it replaced a HPE P4500 G2 where the NICs had died and giving me other issues too
Just now setting up the R510 to backup all our Media to as currently on a single 12TB drive

Basics
Dell Poweredge R510-12 (12x 3.5" bays)
2x 1100w Power Supplies
2x Intel Xeon L5640 CPUs
64GB DDR3 RAM (8x 8GB)
1x Dell Perc H200 flashed to IT P20
12x 3TB SAS Drives
2x 600GB 10k RPM drives
2x Sun/Oracle F80 800GB PCIe SSD's in Raid 0
1x Intel 82598 based 10Gbe NIC (oldy but goodie)
IDRAC 6 Enterprise (frustratingly sssllloooowwww)
Server 2k12 r2, I'll probably update this as having some issues with Degraded Volumes (there is a MS KB fix yet to try)
Server 2019 installed, and degraded volume went away

PCIe slots
Slot one (top right) - 8x slot - 4x Electrical
Slot two (middle right) - 8x Slot - 4x Electrical
Slot three (bottom right) -8x Slot - 8x Electrical
Slot four (top left) - 8x Slot - 4x Electrical - Internal Storage slot

1x Dual USB slot on Expander board + 1 on front panel + 2 on back
SD Card on iDrac6 board for booting from but limited to 2GB partitions (card can be bigger)
2x 1GB Broadcom built in NICs
VGA on front and back (rare on 12bay units)

Here is the above crammed into case, all drive bays full, all RAM slots full, all PCIe slots full, CPUs both in there


Already heaps of drama to get to this point, so post will be updated as things come to mind.

First thing in OS and boot drive planning with the R510 and a PCIe SSD, UEFI boot doesn't work
UEFI boot via HBA's and probably RAID cards are fine
But for some reason the F80 flashed to IR mode UEFI no booty, probably an EFI BIOS on the F80 issue
Took me a while to work this out so hopefully it will help some one out
But now installing Server 2019 in MBR mode which works fine

The F80's did give me a scare, I picked them up dirt cheap ($50ea)
Came with 100% life remaining, can't ask for any better
I originally wanted one in IR mode (1x 800GB) and other in IT mode (4x 200GB)
But while working on above UEFI issue, both card become broken in error state
After much hair pulling and using all my bag of tricks and other too, got them both back and in IR mode
I'll update my bag of trick files to bring back the cards from the dead

Had some issues a while back with a HP H220i HBA flashed to IT mode years ago
I assume it died just as I was looking at starting this project, and tried to take the rest of the server with it
I had to dig deep once again and force the server to boot (PSU Green wire short to ground) as it wouldn't start, fans came on for a split second
Once forced on, server booted up, did complain about a system error that was very vague but no other issues arose
BTW, the R510 with 12x disks has no room for a LCD display to show errors, just a few LED's and a code for their meaning on the left rack ear
After a few reboots the server error went away and its worked like a charm ever since
Had already ordered a Dell H200 which since flashed to IT mode but with A BIOS as I didn't have the F80's yet, will need to reflash with no BIOS

So at the moment running server 2019, which sorted the degraded Volume issue and I was able to make that large volume Thin provisioning as well
No doubt more bits will follow

 

[mobilenvidia]

Sun/Oracle F80 performance on a R510

PCIe 4x slot


PCIe x8

 

[mobilenvidia]

Making some progress, had a couple of issues to sort since I started the above

Issue one- I got off Ebay 6x 3TB Hitachi SAS HDD identical to the 6x units I already have except the FW of the newer ones are A220 vs A120 for older
They worked fine until I setup a Virtual Disk in Server 2019 then drives would fail, different drives every time I setup a new pool then Virtual drive
But only the newer drives affected
Was about to think something at the airport zapped the platters
Last thing was a full format and force 512Byte sectors at the same time
This did the trick, drives now behave and copying 7TB of Media Files to them

Issue two - my Boot Sun/Oracle F80 kept failing after I started the file transfer would take about 1/2 hour or so
Looked at the tempurature and highest recorded was 60deg, which is way less then others are reporting as running at
Thinking it can only be tempurature or bad SSD flash memory
I unlatched the lid on the R510 forcing an Intrusion alert and fans ramp up to 7800RPM
This seems to have sorted the failure issue as now been on for hours
Have also played with IPMI fan speed forcing, running the fans at 4000RPM and F80 (boot) now sitting in high 30's
If I lower speed the temp goes into 40's if I speed up the best I seem to get is 33deg (ambient 14)
Server going into a room beside the Garage so noise not an issue
I went a step further when clearing the F80 flash areas when bringing it back from its fault state and now able to boot to F80 via UEFI
Getting 900MB/s writing to the F80 with no BIOS and 1GB/s writing to bootable F80 and also 1GB/s to the 12x3TB Drive pool

Quite happy now, things are coming together, the R510 is a great Ol' server
iDrac painfully slow, but it has everything needed to make a great home server to back things up to and grab files/movies/music from
Once setup, it should just keep going and need hardly any attention

I just need to reflash the H200 with P20 IT FW and no BIOS, leaving the F80 as sole boot
Got a bootable HDD with Centos 7 on it for diagnosticss, just need to flash a BIOS to H200 and it becomes available

 

[mobilenvidia]

These My Experiences are hopefully helping others as...

Hindsite, I should have left 1x 3.5" bay empty to place the would be shucked 12TB WD Elements in
Currently its connected to a Dell Optiplex 7070 Micro/mini then to server via 1GB LAN
All USB ports are USB 3.1 but via LAN I'm only getting about 30MB/s transfer
First starting it flies at 110MB/s basicaly 1GBps LAN connection speed but a after a minute or 2 drops to about 30MB/s
Not sure what is a foot, as If I transfer a 10GB file from drive to 7070 its doing so at 125MB/s all the way through

Moral of the story is, my 6.8TB transfer is now in its 3rd day

If I was able to put the drive in a server bay, I would be able to copy it at HDD speed
I may epxeriment with a drive removal from Pool once this is done.
I'll have a spare for future issues

 

[mobilenvidia]

Something for Sun/Oracle F40/F80 owners
Windows Driver version 2.10.72.1 (seagate driver and also newest) means that the cross flashed SSD's don't show in LSI MSM (MegaRaid Storage Manager)
Were as LSI drivers 2.10.61.81 do show them in MSM

Transfer finally finished, downloading Dropbox to its own Virtual Drive to have the files handy vs looking at them online and associated slowness
So far so good long live the R510

Here is power use over last 3 days with all 12x drives spinning
Quite happy with that



Nest Job reflash IT with no BIOS to HBA card as not needing to boot to Linux any more to diagose the F80 issue

 

[mobilenvidia]

Some F80 updates

Discovered to boot from the F80, I needed to flash EFI FW to the 9211-IT HBA where all the Spinning rust is attachedto
No need to flash the BIOS, which makes the boot time much faster
Not sure why this is, but no biggy

Thought I'd play with the F80 that was't the boot SSD
I managed to flash the 9211 BIOS and EFI FW, it even booted and worked but was showing errors when looking at with ddcli
Was hoping to see if the select drives for RAID would come up, but same Boot GUI as normal F80 boot
But to get back, I ended breaking both drives even though only working on the one

But a lesson for those who think they've bricked their F40/F80
If you can see it with Linux 'lspci' you can get it back with some perserverance
Both drives back and happily installing Server 2019 to one in PCIe x8 slot

 

[mobilenvidia]

Been a while since last update, moved into new house and had issues with HBA and the 12x drives

Working like a charm now
I've ditched the HBA route and gone back to RAID where I'm much more at home
Was having all sorts of issues with HBA and both Windows and TrueNAS drives kept dropping out but knowing they were good, different drive(s) each time
Happened both on the HP 220i (IT mode) and Dell H200 (Dell 6Gbps IT mode)

Bought a cheap H710P RAID controller with battery
Bonus was the R510 BIOS recognises it and I can setup it up via it
Updated it to latest Dell FW, I'll not bother cross flashing it
Setup a 12x3TB RAID6 array, and a 2x 600GB RAID1 array (bootable)
Installed Server 2k19
Setup some virtual drives
Copied about 10TB of files over
Copying was painless, copying at pretty well 112MB/s all the way through (LAN bottle neck) took about 1.5days vs 3.5 with HBA
No errors showing other than drives coming out of spin down mode
Fast and reiable
Should a drive fail I have 2x spare 3TB drives I can wack in

Was so easy and quick and performance is better overall

 

[mobilenvidia]

Another month or so passes by, been trying a few things

Had a scare of server dieing as after powercut, server refused to boot did everyhting but post
Quite odd as iDrac was working, pressing power button turned on fans and drives spun up, but no post
Ended up with everything pulled out, down to single CPU and DIMM
BIOS CMOS jumper set to reset, got a feeling that removing CMOS battery did the trick

Trialled ESXI 6.7 Dell version
Installs nicely on a USB
But trying to get the RAID array shared to more than one VM foiled me
And performance of the Array was about the same as a single drive

Today a couple of 80cm left 90deg angled with other end straight SFF-8087 cables arrived
Replace the Dell 90deg at both end cables which were also too short for SAS controller in bottom rear PCIe x8 slot

So goodbye ESXI welcome back Server 2019 and I'll install Rocky Linux on another drive as only need it every now and then
Was toying with PFSense but running running everything of one server leaves me with it fails everything fails

Any who system has some things rearranged
I've got the H710P in the PCEi (electrical) x8 slot for best performance
Intel 4x 1Gbe NIC card in top slot
Middle slot free for air flow
Sun/Oracle F80 in RAID mode (single 800GB drive) in dedicated drive controller slot
Did remove the wee bracket thats closest to the fans as blocking air flow, nothing grinder and removing rivets could't sort
Didnt' butcher it can put it back on with rivets

Performance with larger than cache file size

Happy with that for 12x RAID6 spinning rust