Backups and failover
Losing the data is unrecoverable; being down for twenty minutes is merely bad. Optimise in that order.
Losing the data is unrecoverable; being down for twenty minutes is merely bad. Optimise in that order:
- pgBackRest with point-in-time recovery, off-box and encrypted.
- Automated restore tests. An untested backup is not a backup. Restore into a scratch database on a schedule and assert the data is real.
- Only then, failover.
Cardinal has a pleasant property here: after any restore, validate the audit journal's hash chain and you get cryptographic confirmation the restored data was not tampered with, something an ordinary Postgres restore cannot tell you.
cardinal audit verify
make restore-drill in the repository does the whole loop: back up, restore to
a scratch database, verify the chain.
The chain checked against itself does not show history rewritten before the
backup was taken. A fingerprint kept
from before does: cardinal journal verify against the restored database.
Failover: start manual, and do not be embarrassed by that
A synchronous standby with a documented, rehearsed manual promotion is more reliable in the hands of one maintainer than automation nobody fully understands. Split-brain in an identity store means two primaries accepting credential and policy writes, and you may not notice for hours.
Automate it when there is a team on call, or adopt CloudNativePG if this ends up on Kubernetes, since its consensus rides on the Kubernetes API rather than dragging in etcd.