I've Got Mail

I’ve finally gotten tired of maintaining my own mail server. I’ve been maintaining it since 1999 in one way or another, and I think it’s just time to take it behind the shed and put it out of my misery.

As to how I reached this decision and what I intend to do about it, well… that’s a story.

Adventures in Babysitting

Up until now I’ve gone the more traditional route of running “The Stack”:

• Postfix - Probably the most common Mail Transfer Agent (MTA) aside from Sendmail.
• Postgrey - A greylist which bounces mail with a “try again later” header. Spammers generally won’t, so it cuts way down on spam. Annoying because it requires frequent updates to the whitelist if you don’t want to wait 15+ minutes for mail from unknown senders.
• Dovecot - For external IMAP access to my mailbox.
• Roundcube - One of the best webmail front-ends in my opinion. I have my own webserver, so it makes sense to run the email client there with all of my mail.

This stack is so common that it’s packaged together in more than one available pre-packaged configuration. What makes it irritating is not necessarily all of the moving parts, but the continuous advancements in the War Against Spam.

It used to be sufficient to set up a mail server, subscribe to a few blacklists to block the worst spammers, and configure your mail server properly so it couldn’t be used as a blind relay. The email industry as a whole decides on yet another onerous policy every few years, and it’s now reached a ridiculous crescendo I can’t really deal with anymore.

These are just the additional DNS records you need to create:

• A PTR record so a reverse-IP search resolves to your mail server. Ok, that makes sense.
• A SPF DNS record to inform other servers of the trusted mail servers for your domain. Ok, sure. DNS record specifically outlining what mail servers are associated with the domain. Fair.
• A DKIM record to cryptographically sign emails so they can be independently verified. Why in the actual Hell is this necessary if SPF and PTR records exist? If the server sending the email matches those records, the DKIM should already pass. Configuring Postfix to do DKIM properly is a goddamn minefield of idiocy. These are the “easy” instructions for setting up OpenDKIM with Postfix. WTAF!
• A DMARC record to tell other servers what to do with mail that fails SPF and DKIM checks. Again, what is this for? I literally don’t care what external servers do with mail I don’t accept. Print it and shovel it into a furnace for all I care. If the records don’t match PTR and SPF, it’s not my problem. You obtuse motherfuckers.

These are not easy to set up. Worse yet, they absolutely must exist these days or basically every email server on Earth will reject messages from your domain. Just no. That’s enough. Goddamn it.

I coasted on PTR and SPF alone for years until I started getting mail rejections. Then I found out my domain was on a ProofPoint blacklist used by many other companies as a spam filter. Why? At the time I had no idea. I checked every blacklist for my server’s IP address, and I wasn’t on any of them, so I should have been fine. Nope. Proofpoint gets to play by different rules because reasons.

ProofPoint has terrible business practices because their security method is completely opaque. Worse, their IP Check service doesn’t actually do anything. Once I figured out I needed to add DKIM and DMARC, I scrambled to make sure they were present and compliant. It took ProofPoint several weeks to remove me from the blocklist because the IP check tool doesn’t launch an automated validation of the domain again, it creates a ticket in their system they have to manually attend to.

What. Are. They. Smoking?

The point is that this event nearly knocked my mailserver off the web entirely. Most mail got through just fine, but random customers of ProofPoint always rejected mail from my domain. I set up the additional DNS records and necessary Postfix configs in a hurry and it works, but it’s Duct tape and bailing wire. I’m 100% certain if I had to do it again, it would be just as painful as the first time. I don’t have time for this aggravation.

Potential Alternatives

The easiest way to fix this long-term would be to use a pre-configured and packaged stack. After some exploration, I found these:

• Mail-in-a-Box - It’s literally the same stack I already use, but more automated. Unfortunately they solve the DNS issue by… running a DNS server with the proper configuration for your domains. It’s also “opinionated” about what else should be running on the server, so could conflict with running my websites and other stuff I use the server for. I’d need to test it thoroughly.
• Mailu - This is a very similar stack as MiaB, but is designed to run in a containerized environment. That’s a neat way to approach this, and I like it. It wouldn’t be too hard to set up Podman on my VPS and go to town. The admin GUI spits out the necessary DMARC and DKIM records when prompted for additional convenience.
• Mailcow - Is another containerized setup that relies on the Postfix stack. It has a pretty heavy list of inclusions including MariaDB, Redis, SOGo groupware, and many others. My VPS is two cores and 4GB of RAM, so… maybe not.
• Stalwart Mail Server - This is a very “batteries included” mail server written in Rust. It basically does everything the full Postfix stack does, but as a single binary. It prominently displays all necessary DNS records upon request, and is comparatively dead easy to maintain. A solid maybe.

Or I could just… not. It would be practically trivial to import my existing mail into ProtonMail through an IMAP pull. Then I could point my mail DNS records to their servers and never have to think about any of this ever again. Very tempting.

Exploring Archival

Since I’m revamping my email setup one way or another, I decided it was also time to take a look at my email archival solution. So far, it’s been: “Use rsync to copy my Maildir folder to a backup location on a semi-regular basis.” High-tech, I know. But with a 1GB and 20-ish year archive, it does the job.

This strategy has a lot to be desired because such an archive is hardly interactive. Viewing the contents means pointing a mail client at it, and Linux mail clients are universally awful. I’ll admit the last time I tried to switch to using one was back in 2015 or so, but every single one I tried was worse than the last. Thunderbird was glacially slow if I dared to search my email. Sylpheed was both hideous and slow. Evolution crashed frequently, used a ton of RAM, and was… glacially slow. KMail was probably the worst of the lot, as it started its own MySQL daemon, because of course it did.

Just terrible. All of them. And nothing has changed since then. In 10 years, I couldn’t find any new mail clients that didn’t make me want to rip out my own hair. Nothing in Rust. Nothing with full-text search. Nothing with a SQLite backend. Nothing with an aesthetic more recent than 1995. In theory I could run Dovecot locally and just keep using RoundCube to browse the archive. It’s not an ideal solution, but at least it’s the Devil I Know.

As an experiment, I spent yesterday writing an Email parsing system using the Python mailbox library. It has some pretty strange quirks, but works well enough. What gets me is that it has absolutely no built-in email parsing, formatting, or encoding management. No, you gotta decode the base64 encoded blobs yourself, walk the convoluted email attachment envelopes to break content from actual attachments, and so on. It’s not so much a parsing library as it’s the most bare-bones email extraction system humanly possible.

Still, it did the job I needed. I created a parsing system that dumped all 20 years of my email into a Postgres table in less than 100 lines of Python. I then applied a GIN Trigram index to the message body to make it searchable. I ran a few test queries, and I can search all 8300 messages in the archive for literally any text in about 10ms.

Look, I’m not a genius. I did what basically any other gearhead could have done in my position. I spent about half a day fighting with a library I didn’t know, missing a bunch of features it should have had, and ended up with a solution that can search email contents 1000x faster than any of the existing Linux email clients. I get they are GUIs and do other things, but come on. I could go further and have an LLM write a competent JavaScript front-end to tie everything together and produce something 10x better than what seems to already be out there. Will I? No time, but the option is there.

My theory is that email is either dead or dying, so nobody is being serious about it. I can’t say I disagree. Due to all the draconian restrictions on servers, it’s being increasingly dominated by the big players like Gmail and Outlook. It basically exists purely for two-factor auth and de-facto account validation mechanism, spam, tracking online purchases, and getting random updates from trusted vendors. It’s a fetid miasma.

God, what we’ve done to the once venerable SMTP.

Until Tomorrow