PasswordAuthentication no is probably the most copied line of server hardening advice on the internet. On a lot of Ubuntu cloud images, it does nothing.
I found that out while turning my new-server checklist into an Ansible playbook, ansible-server-bootstrap. It takes a fresh Ubuntu 22.04 or 24.04 VPS to a hardened Dokku host, and it’s built to be re-run. My apps use Dokku’s Docker Image Deployment: git:from-image initializes or updates each app’s repository from a Docker image. Automating the checklist meant checking it, and a few steps I had always trusted turned out not to do what I thought.
TL;DR
- A cloud-init drop-in can keep SSH password login enabled after you disable it in
sshd_config. Write your settings tosshd_config.d/00-hardening.confand confirm them withsshd -T.- Installing fail2ban before exempting your own address can ban the machine running the playbook, mid-run.
- Docker and Dokku come from their signed apt repositories, not
curl | sh, and Dokku and its plugins are pinned to exact versions.- Still not covered: ports published by Docker bypass ufw, and there are no backups yet.
A cloud-init drop-in keeps password login on
Ubuntu’s /etc/ssh/sshd_config starts with a line most people scroll past:
Include /etc/ssh/sshd_config.d/*.conf
Those drop-ins are read before the rest of the main file, and for most settings sshd uses the first value it obtains. On many VPS images, cloud-init has already written a drop-in that allows passwords:
sshd_config.d/50-cloud-init.conf PasswordAuthentication yes <- read first, wins
sshd_config PasswordAuthentication no <- ignored
sshd -t passes, the reload succeeds, and password login keeps working. The only way to see it is to ask sshd what it will actually do:
sudo sshd -T | grep -Ei '^(passwordauthentication|kbdinteractiveauthentication|permitrootlogin) '
So the playbook leaves the main file alone and writes /etc/ssh/sshd_config.d/00-hardening.conf. Drop-ins load in name order, so 00- is read before cloud-init’s 50-.
That still isn’t proof. A Match block or a provider’s own 00-aaa.conf could override it, so the role checks the effective config before trusting its own file, and removes the file if a check fails:
- name: Harden sshd, rolling back if the change doesn't take effect
block:
- name: Install the sshd hardening drop-in
ansible.builtin.template:
src: 00-hardening.conf.j2
dest: /etc/ssh/sshd_config.d/00-hardening.conf
validate: /usr/sbin/sshd -t -f %s
register: ssh_hardening_dropin
- name: Read the effective sshd config for a root login
ansible.builtin.command: /usr/sbin/sshd -T -C user=root,host=localhost,addr=127.0.0.1
register: ssh_hardening_effective
changed_when: false
- name: Check each hardening option is in effect
ansible.builtin.assert:
that: (item.key ~ ' ' ~ item.value) | lower in ssh_hardening_effective.stdout_lines
fail_msg: "{{ item.key }} is overridden elsewhere in the sshd config"
loop: "{{ ssh_hardening_options | dict2items }}"
- name: Reload sshd # noqa: no-handler
ansible.builtin.service: { name: ssh, state: reloaded }
when: ssh_hardening_dropin is changed
- name: Drop the current SSH connection
ansible.builtin.meta: reset_connection
- name: Confirm a fresh SSH login still works
ansible.builtin.wait_for_connection: { timeout: 30 }
rescue:
- name: Remove the hardening drop-in
ansible.builtin.file:
path: /etc/ssh/sshd_config.d/00-hardening.conf
state: absent
when: ssh_hardening_dropin is changed
# ...reload sshd, then fail the run
Three details carry the weight:
-Cevaluates the config for one connection, a root login from localhost, soMatchblocks that apply to it are included. Without-C,sshd -Tdoesn’t applyMatchrules.- The reload is a task, not a handler, because handlers run at the end of the play, after the check that’s meant to test them.
reset_connectionfollowed bywait_for_connectionproves a brand new login works, not just the connection Ansible already had open.
Before any of this runs, bootstrap.yml logs in as root, creates an admin user with no password, and hands over to site.yml, which connects as that admin. It’s the “open a second terminal before you disable root” step, except it can’t be skipped.
The rollback is best-effort. It only deletes a drop-in this run changed, and it needs a working connection: Ansible doesn’t run rescue for unreachable hosts. On a first run I still keep the provider’s console open.
Installing fail2ban banned the machine running the playbook
Most tutorials install fail2ban, then configure it. On a fresh server, that order can ban the machine running the playbook.
apt starts fail2ban as soon as it’s installed, and it starts reading the recent SSH log history straight away. A first run against a server that only accepts a root password leaves a few failed key attempts from your address, which is enough for a ban. Because fail2ban’s REJECT rule sits above ufw’s rule for established connections, the ban also kills the session Ansible is using. The run dies with a privilege-escalation timeout that points nowhere near the firewall.
The fix is ordering: write the ignore list before the package exists.
- name: Find the address Ansible connects from
ansible.builtin.shell: echo "$SSH_CONNECTION"
become: false
changed_when: false
register: firewall_ssh_connection
- name: Create the fail2ban jail directory
ansible.builtin.file:
path: /etc/fail2ban/jail.d
state: directory
- name: Exempt the deploy address from fail2ban
ansible.builtin.template:
src: ignoreip.local.j2
dest: /etc/fail2ban/jail.d/00-ignoreip.local
- name: Install fail2ban
ansible.builtin.apt:
name: fail2ban
$SSH_CONNECTION is your address as the server sees it, which is the address fail2ban would ban, so it’s right even behind NAT. The same role reads the SSH port from sshd -T rather than the inventory, so ufw opens the port sshd actually listens on.
SSH and the firewall were about proving that hardening actually took effect. The rest of the playbook had a different job: installing Docker and Dokku so that a second run leaves a working server alone.
Installing Docker and Dokku from signed apt repositories
Docker and Dokku both document a one-line installer that runs a downloaded script as root. The playbook installs both from their signed apt repositories instead, with Dokku pinned to an exact version:
- name: Add Dokku's apt repository
ansible.builtin.deb822_repository:
name: dokku
uris: https://packagecloud.io/dokku/dokku/ubuntu/
suites: "{{ ansible_facts.distribution_release }}"
components: main
signed_by: https://packagecloud.io/dokku/dokku/gpgkey
- name: Install Dokku
ansible.builtin.apt:
name: dokku={{ dokku_version }}
The package is also held, so the base role’s apt upgrade can’t move it. A second run is a no-op, and upgrading Dokku is a one-line version bump. Packages can still run maintainer scripts as root, so this isn’t “nothing downloaded runs”. The gain is that everything arrives through apt with its signatures checked. Docker installs first, so apt satisfies Dokku’s Docker dependency with docker-ce rather than pulling in Ubuntu’s older docker.io.
Replacing the scripts exposed one thing Dokku does quietly: it writes live-restore into /etc/docker/daemon.json. Templating that file to add log rotation would delete the setting, so the playbook merges in only the log options and compares the parsed JSON. A text comparison would never match, because Dokku writes the file unsorted, and Docker would restart on every run.
Making Dokku’s CLI safe to re-run
Dokku’s CLI does things; it doesn’t declare state. So every task reads what’s there and acts only on the difference. Three commands needed more than that:
- SSH keys.
ssh-keys:addfails if the name already exists, so each key is named after a hash of its own material (<admin_user>-<first 8 of sha1>), not its position in a list. - Plugins.
plugin:install <url>with no tag clones whatever the default branch holds that day. Each plugin is pinned to a release tag and compared withdokku plugin:list --format json. - Let’s Encrypt. The plugin won’t issue certificates until a global email is set. Querying just that value exits non-zero when it’s empty, so the playbook reads the full
letsencrypt:report --global --format jsoninstead.
dokku_plugins:
- name: postgres
url: https://github.com/dokku/dokku-postgres.git
version: "1.48.0"
- name: redis
url: https://github.com/dokku/dokku-redis.git
version: "1.43.0"
- name: letsencrypt
url: https://github.com/dokku/dokku-letsencrypt.git
version: "0.25.2"
Pinning works because Dokku’s official plugins keep the version in plugin.toml equal to the release tag. Pin a branch name instead and it never matches, so the plugin reinstalls on every run.
What it doesn’t protect you from
Docker bypasses ufw. Ports published with docker run -p or a Compose ports: entry go through Docker’s own firewall rules, so they’re reachable from the internet even though ufw status doesn’t list them. Dokku apps are fine, because nginx on 80 and 443 proxies to internal addresses. A database exposed “just for a minute” isn’t: bind it to 127.0.0.1 and use an SSH tunnel.
Backups. There are none yet. Off-server backups with a tested restore come before I put data I can’t lose on this host.
Running it yourself
git clone https://github.com/thepsalmist/ansible-server-bootstrap.git && cd ansible-server-bootstrap
uv tool install ansible-core
ansible-galaxy collection install -r requirements.yml
cp inventory.ini.example inventory.ini # your server's address
cp group_vars/all.yml.example group_vars/all.yml # admin_user, admin_ssh_keys
ansible-playbook bootstrap.yml # first run, as root (keep a console open)
ansible-playbook site.yml # every run after that
If there’s one habit to take from this: a command that exits 0 hasn’t proved anything. Have the playbook check the state the command was meant to change, the way sshd -T checks what sshd will actually do.
The same playbook also runs Prometheus, Loki and Grafana on the host, and building dashboards for a server that deploys by renaming containers had its own surprises. That’s the next post.