Hosting Your Own Email Server F%#cking Sucks

A smart person learns from his mistakes. A wise one learns from others’ mistakes.

TL;DR

Hosting your own email server is not just a technical headache; it’s an embodiment of living in a low-trust society where you’re constantly guilty until proven innocent.
It’s an uphill battle against spam, security threats and the never-ending struggle of trying to reach a perfect server reputation that’s constantly “just” our of reach.
I mean you’ll find yourself constantly jumping through hoops, trying to prove you’re not a bad guy, all because of the actions of a fringe minority.
The real price isn’t just about money – it’s about the toll it takes on your time, sanity and even your faith in humanity.
Do yourself a favor and leave it to someone else. BUT HEY, if you’re feeling adventurous, here’s a quick 5-minute guide on how to host your own email server. And if you’re itching to tell me exactly what I’ve done wrong or why I’m an idiot, you’re more than welcome to leave a comment or shoot me an email at stan[at]domainyourereadingiton.com

The Setup

┌──────────────────────────────────────────────────────────────────────┐
│                                                                      │
│  ┌────────────────┐                             ┌────────────┐       │
│  │                │         WireGuard           │            │  25   │
│  │  running:      │◄───────────────────────────►│            │  465  │
│  │  postfix       │                             │ running:   │  587  │
│  │  dovecot       │ forward packets via nftables│ nftables   │  143  │
│  │  amavis        │◄────────────────────────────┤            │  993  │
│  │  spamassassin  │                             │            │  ---  │
│  │  clamav        │                             │            │  80   │
│  └────────────────┘                             └────────────┘  443  │
│      HOST SERVER                                   FRONT END         │
│                                                                      │
│   10.13.13.2 (nginx)                           10.13.13.1 (host)     │
│   10.13.13.3 (mail)                                                  │
│                                                                      │
│                                                                      │
└──────────────────────────────────────────────────────────────────────┘

The Setup is pretty much self-explanatory:

a host server with a residential/dirty ip is connected to a front-end server with a clean ip via wireguard
a front-end server re-routes all requests on specific ports to a host server
for flexibility, consider grouping packet forwarding by their purpose:
- 80,443 - nginx/httpd
- 25,465,587,143,993 - smtp+imap

Note: if you’re also running nginx on your mail server, instead of nftables/iptables you may use HAProxy on a front-end server to add X-Forwarded-For to your incoming requests. The magic word you’re looking for is option forwardfor.

Front-End Configuration

wireguard

Install wireguard via apt install wireguard or yay install wireguard-tools if not installed.
Create a file in the wireguard conf directory /etc/wireguard/wg0.conf and edit it like this:

[Interface]
Address = 10.13.13.1/24
MTU = 1420
DNS = 9.9.9.9
ListenPort = 51820
PrivateKey = # REDACTED #

[Peer]
# peer_nginx
PublicKey = # REDACTED 2 #
AllowedIPs = 10.13.13.2/32

[Peer]
# peer_mail
PublicKey = # REDACTED 3 #
AllowedIPs = 10.13.13.3/32

Note: to generate wg privatekeys use cli: wg genkey > privatekey.
To generate publickey run this: wg pubkey < privatekey > publickey.
Enable it by running wg-quick up wg0.

nftables

Create a file in your user directory (ex: /home/admin/rules.v1.nft) and edit it like this:

flush ruleset

add table ip nat
add chain ip nat PREROUTING { type nat hook prerouting priority dstnat; policy accept; }
add chain ip nat INPUT { type nat hook input priority 100; policy accept; }
add chain ip nat POSTROUTING { type nat hook postrouting priority 100; policy accept; }

add rule ip nat PREROUTING iifname "eth0" tcp dport { 25,143,587,993 } dnat to 10.13.13.3
add rule ip nat POSTROUTING oifname "wg0" ip protocol tcp ip daddr 10.13.13.3 tcp dport { 25,143,587,993 } snat to 10.13.13.1

Before applying these rules, check this:

change eth0 to whatever front-end network interface is used for communicating with a real world
- you can a list of your network interfaces via ip addr
change wg0 to the WireGuard interface name you use to connect to your host server
change IPs if you’ve edited WireGuard configuration To apply these rules run nft -f rules.v1.nft.
If you’ve lost connection to your server, restart it via your hoster’s admin panel.

In short, we’re:

deleting all existing rules
adding PREROUTING rules which forward tcp traffic from the outside world to your host server
adding POSTROUTING rules which forward tcp traffic from your host server to the outside world

Instead of all POSTROUTING rules you may just use a “Blanket Rule”: add rule ip nat POSTROUTING ip saddr 10.13.13.0/24 oifname "eth0" snat to 10.13.13.1.
Also, if you’re not planning to use HAProxy to handle http traffic you need to add 2 rules to the PREROUTING section and one rule to the POSTROUTING section:

add rule ip nat PREROUTING iifname "eth0" tcp dport { 80,443 } dnat to 10.13.13.2
add rule ip nat PREROUTING iifname "eth0" udp dport { 80,443 } dnat to 10.13.13.2

add rule ip nat POSTROUTING oifname "wg0" ip protocol tcp ip daddr 10.13.13.2 tcp dport { 80,443 } snat to 10.13.13.1

Host Server Configuration

wireguard

WireGuard configuration for nginx/httpd:

[Interface]
Address = 10.13.13.2/24
MTU = 1384
PrivateKey = # REDACTED #
ListenPort = 51820
DNS = 9.9.9.9
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT;
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT;
#PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eno1 -j MASQUERADE; iptables -A FORWARD -o %i -j ACCEPT
#PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eno1 -j MASQUERADE; iptables -D FORWARD -o %i -j ACCEPT

[Peer]
PublicKey = # REDACTED #
Endpoint = # FRONT-END SERVER IP #:51820
AllowedIPs = 10.13.13.2/32, 10.13.13.1/32, 10.13.13.3/32
PersistentKeepalive = 25

WireGuard configuration for mail assuming mail is dockerized:

[Interface]
Address = 10.13.13.3/24
PrivateKey = # REDACTED #
ListenPort = 51820
DNS = 9.9.9.9
#PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE; iptables -A FORWARD -o %i -j ACCEPT
#PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE; iptables -D FORWARD -o %i -j ACCEPT

[Peer]
PublicKey = # REDACTED 2 #
Endpoint = # FRONT-END SERVER IP #:51820
AllowedIPs = 0.0.0.0/0
PersistentKeepalive = 25

mail server

You may ask (rightfully so), “why dockerize?!” but dockerizing is the fastest way to ensure that everything related to the mail server will go through front-end’s network interface. And as far as you’ve remember, we’ve kinda commited to deploying this in, like, 5 minutes so…

I’d recommend to choose between these 2 options:

Please note that at the time of writing this only a specific version of wireguard would work for some. However, you’re welcome to find out if newer versions would work with your specific config. Here’s an example of a docker-compose.yml for docker-mailserver with all traffic proxied through wireguard:

services:
  wireguardclient:
    image: ghcr.io/linuxserver/wireguard:v1.0.20210914-ls81
#    image: ghcr.io/linuxserver/wireguard
    container_name: wireguardclient
    networks:
      docker_mail_wg:
        ipv4_address: 172.16.1.2
    ports:
      - "25:25"    # SMTP  (explicit TLS => STARTTLS)
      - "143:143"  # IMAP4 (explicit TLS => STARTTLS)
      - "465:465"  # ESMTP (implicit TLS)
      - "587:587"  # ESMTP (explicit TLS => STARTTLS)
      - "993:993"  # IMAP4 (implicit TLS)
      - "995:995"  # POP3
    dns: 9.9.9.9
    cap_add:
      - NET_ADMIN
      - SYS_MODULE
    environment:
      - TZ=America/Chicago
#      - PUID=7722
#      - PGID=7722
    restart: "unless-stopped"
    sysctls:
      - net.ipv4.conf.all.src_valid_mark=1
      - net.ipv6.conf.default.disable_ipv6=1
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /path/to/local/wireguard/configuration:/config:ro
      - /lib/modules:/lib/modules

  mailserver:
    image: docker.io/mailserver/docker-mailserver:latest
    container_name: mailserver
#    hostname: put_your_subdomain_here
    domainname: your_domain_name_here.com
    env_file: mailserver.env
    sysctls:
#      - net.ipv4.conf.all.src_valid_mark=1
      - net.ipv6.conf.all.disable_ipv6=1
    privileged: true
    network_mode: "service:wireguardclient"
    volumes:
      - ./docker-data/dms/mail-data/:/var/mail/
      - ./docker-data/dms/mail-state/:/var/mail-state/
      - ./docker-data/dms/mail-logs/:/var/log/mail/
      - ./docker-data/dms/config/:/tmp/docker-mailserver/
      - ./docker-data/dms/ssl/:/tmp/ssl:ro
      - /etc/localtime:/etc/localtime:ro
    restart: always
    stop_grace_period: 1m
    cap_add:
      - NET_ADMIN
      - SYS_PTRACE

networks:
  docker_mail_wg:
      external: true
      name: docker_mail_wg
      enable_ipv6: false
      ipam:
        config:
          - subnet: "172.16.1.0/24"

As you can see, we’re creating 2 services and 1 network:

wireguard service to connect host server to front-end server
- /path/to/local/wireguard/configuration:/config:ro must be changed
docker-mailserver service to host our mailserver
- the secret ingridient here is the network_mode: "service:wireguardclient"
docker_mail_wg network to manage the connection

Important mailserver.env values:

network-related
- PERMIT_DOCKER=none
- NETWORK_INTERFACE=wg0
security
- SPOOF_PROTECTION=1
- ENABLE_POP3=0
- ENABLE_CLAMAV=1
- ENABLE_AMAVIS=1
- ENABLE_FAIL2BAN=0 # before enabling check logs if you’re getting a real ip
- ENABLE_SPAMASSASSIN=1
SSL
- SSL_TYPE=manual
- SSL_CERT_PATH=/tmp/ssl/fullchain.pem.crt
- SSL_KEY_PATH=/tmp/ssl/privkey.pem.key
- SSL_DOMAIN=yourdomainhere.com

Once you’ve set everything up you can add a user via calling setup.sh like this: ./setup.sh email add [email protected] PASSWORD'. Finally, you may start it via docker-compose -f mailserver/docker-compose.yml up –build –force-recreate -d`.
The easy part is finally done and we’re ready for some trouble.

DNS

DMARC

Let’s set up our DMARC first: to do that you need to login to your DNS manager for your domain and set it up as a TXT record. Here’s an example:

Host/Name: _DMARC
Value: v=DMARC1; p=reject; rua=mailto:[email protected]; ruf=mailto:[email protected]; fo=0:1:d:s

It’s pretty straightforward:

if something fails: reject
rejection reasons are sent to [email protected]
select all rejection reasons

SPF

The next one is SPF record, here’s an example:

v=spf1 ip4:38.113.1.0/24 ip4:38.113.20.0/24 -all

This record is pretty straightforward as well:

emails can be sent from this ip range: 38.133.1.0/24
emails can be sent from this ip range: 38.113.20.0/24
reject if they’re sent from different email (if someone pretends to be your server without a valid ip)

rDNS

In most cases you’d have to contact a front-end server owner and ask them to point rDNS to your server domain.

Here’s the fun part: by doing these last 2 things you’re kinda exposing to everyone the fact that YES, I’M HOSTING A MAIL SERVER HERE which isn’t ideal to say the least but it’s the world we live in.

Testing

Now… We ~~celebrate~~ test!

In my experience, mail-tester.com worked great to straigten up the scores, but the real pain comes from blacklists.
Here’s mxtoolbox’s blacklist: just type in your IP and it will check if your IP is blacklisted.
Now, option A: you’re not blacklisted, you may close this guide and warm this IP by sending emails to your friends and family, rejoice if they’re delivered or add tap ‘NOT A SPAM’ to every single email you send.
Option B though is far more interesting: if your IP is blacklisted anywhere but UCEPROTECTL3, you’re good because the matters are in your hands.

You may contact each and every one of these lists and promise you won’t be sending any spam.
For some of them you’d have to connect from the IP in question (you may use a wireguard configuration do to it)
Most of them won’t even ask for your money (though some will)

Now, the infamous UCEPROTECTL3. In most cases you’re SOL. No, really, you are. If you’re in UCEPROTECTL3 list, it means some neighbouring server is sending spam and you need to find a different hoster who bans spammers.
So… you may need to start the front-end part all over again until you find ‘THE ONE HOSTER’ that cares about banning spammers.
And it doesn’t even matter if you’ve owned this IP for over a year; until your hoster promptly bans everyone, you’re SOL.

The sad conclusion

In most cases you’ll spend countless hours fighting with ’neighbor’ spammers, dealing with ~~extortionists~~ blacklists, and ‘warming’ a ‘cold’ IP. At some point you must ask yourself: if these hours you’ve spent fixing it weren’t free, how much would it cost by hiring myself to fix it?
Don’t forget: your time is valuable. In most cases you may trust someone else to do the heavy lifting: try purelymail, zoho or fastmail: for most people it just works. Please note I have absolutely no affiliation or compensation from these companies in any way, shape, or form. I tried all 3 of them (they all have free tiers!) and settled on one of them. And yes, I’m a little bit happier now since I don’t have to chase perfection: it just works©.

TL;DR#

The Setup#

Front-End Configuration#

wireguard#

nftables#

Host Server Configuration#

wireguard#

mail server#

DNS#

DMARC#

SPF#

rDNS#

Testing#

The sad conclusion#

TL;DR

The Setup

Front-End Configuration

wireguard

nftables

Host Server Configuration

wireguard

mail server

DNS

DMARC

SPF

rDNS

Testing

The sad conclusion