Adventures in Server Sitting

Tech Talk

To support more of my tinkering in an effort to test various Postgres cluster configurations, I decided it would be really nifty to have a virtual server. I could not only spin up VMs and containers to validate architectures, but experiment to my heart’s content with other potential technologies.

At first, I was going to buy an Antsle. But the fact such a thing existed made me wonder what other kinds of dedicated virtual device hardware might exist. That led me in endless random directions until I stumbled over a link to eBay that boasted a price that seemed both ridiculous and impossible. Apparently old servers are incredibly cheap and plentiful these days. After the shock wore off, I eventually settled on a Dell PowerEdge R710. This is what I ended up getting for about $300:

The iDRAC6 Enterprise is important, as it enables a virtual console over TCP through a Java app that launches from a web browser pointed to the configured IP address. No need for monitors, here! It’s an optional module and cheap by itself, but the particular model I bought had one already. Well, it would have, but they forgot to send that part. Though I fiddled with the server after it arrived, I couldn’t really go nuts with it without the iDRAC card, because that would mean dragging it over to my desk and actually hooking it up to one of the other inputs on my monitor. No thanks.

I called the seller’s support line and got them to send me the missing card. Having taken care of that, I did some more searches to decide how I wanted to configure the beast. For example, how should I configure the storage so that it’s expandable in the future? RAID 10? JBOD with software RAID and LVM for the sake of flexibility?

Naturally I searched Reddit for R710 threads, and stumbled into the Homelab reddit. Apparently it’s one of the most popular servers to start with because it’s so hilariously inexpensive and plentiful. A few helpful replies later and I had my answer: install two more 1TB drives and install ZFS.

But that left me with a new problem. ZFS likes to have direct control of devices, and the included PERC 6/i RAID card didn’t have device pass-through mode. Fortunately Homelab was helpful in that regard as well. It’s possible to flash a H200 RAID card to Host Bus Adapter (HBA) mode because it’s basically a re-branded LSI card. Well, as with all parts for the R710, such a card is also comparatively cheap, as are the cables necessary to connect it to the SAS backplane for the drives.

Which of course led to the next stumbling block. ZFS on Linux is still a bit sketchy as a boot device since its license isn’t compatible with the Linux Kernel. As such, it’s only included by Ubuntu, and I’d be one sketchy kernel away from having an unbootable system. It’s usually better to install Linux on a separate boot device and use the hard drives as a storage pool. Well, I’m not using the included DVD drive for anything, so out it goes. It’s being replaced with a 250GB Samsung 850 Evo. Not only can that act as a stable filesystem, but I can partition it as a ZFS read/write cache as well. After reserving 50GB for the OS, 200GB beats a 1GB RAID controller any day.

I finally got most of the parts and spent the weekend reconfiguring everything. I have to say, re-flashing a RAID controller into an unsupported configuration isn’t easy, especially when the available instructions are conflicting, haven’t been updated in a year, and weren’t written specifically for the R710. These were the most comprehensive instructions I could find, and supplied all of the files I’d need to pull off the entire surgery.

First I needed to get the iDRAC6 working so I could decouple the server from my monitor. That took over an hour because the included module includes its own dedicated LAN port which is required to access the virtual console. I didn’t know it gets bypassed unless you boot into the iDRAC and change its configuration to use the dedicated port instead. That took a lot of reboots. Did I mention that the R710 takes several minutes to boot? It’s true, and that’s something that’s apparently common with most server hardware, even if you don’t count the numerous prompts to configure various add-on components.

Well, I finally got that working, and moved on to the H200. I followed the instructions I found, and got stuck because they were wrong. For whatever reason, the R710 or the H200 don’t like the megarec.exe utility, and hung every time I tried to use it for the first two steps. Cue a few more reboots while I figured out it wasn’t just really slow at flashing the firmware and was actually stuck. With a bit more research, I eventually got everything working with these commands:

  1. Erase the existing BIOS and firmware, then replace it with a generic 6GB SAS profile.

     sas2flsh.exe -o -e 6
 sas2flsh.exe -o -f 6GBPSAS.fw -b mptsas2.rom

  2. Reboot.

  3. Flash the generic LSI HBA firmware.

     sas2flsh.exe -o -f 2118it.bin

I say I eventually got it to work, because the instructions I used were based on the megarec.exe utility working as described. Since I had to use sas2flsh.exe to erase the card’s firmware instead, I also removed the card’s BIOS. So even though the card had the right software, there was no BIOS to actually invoke it. That took several more reboots to diagnose and remedy.

But finally after three befuddling hours, everything was working. I didn’t fully remove the existing RAID card just in case all of this went sideways, so I’ll be finishing the operation later. At that point, I’ll remove the old controller so it stops complaining about having no cables attached. The 850 Evo hasn’t arrived yet, so I’m in no hurry. Once it comes in, it will replace the DVD drive, and then I can shove the server in my closet where it will supply me with VMs and containers until it dies.

The next leg of my adventure will be installing Proxmox to manage the VM army. I experimented with it today to verify all four drives function properly in a RAID-10 configuration with the new controller card, but that’s about all. All I know is that I managed to get things mostly working, and I’ve got a lot further to travel before I have something I can use as a virtual laboratory.

But hey, at least I’m closer than I was before.