Automated Server Blueprint & Instant Recovery Declarative Infrastructure & Automated Recovery (debian-ansible)

Component 2 replaces manual, undocumented server administration with debian-ansible: a declarative, idempotent infrastructure-as-code blueprint that provisions, hardens, and orchestrates disposable Debian 12/13 hosts, our Traefik v3 + CrowdSec Edge Bastion, our NetBird WireGuard mesh, and the Git-backed highclere-tickets Docker Compose stack from a single Host-Centric Inventory.

As articulated in our engineering philosophy (Rebuild, Don't Repair and the Modulix concept model), imperative infrastructure records history (a fragile sequence of manual SSH edits, package pins, and ad-hoc cron jobs), whereas declarative infrastructure records desired state.

Following our ADR-0002 (Host-Centric Inventory and Dynamic Classification) and the Modulix Capability Translation model, every server is declared exactly once under all.hosts in inventory.yaml. Operators attach Composable Capability Flags rather than maintaining brittle, duplicated inventory groups.

┌──────────────────────────────────────────────────────────────────────────────────────────┐
│                         PUBLIC INTERNET & CLOUDFLARE EDGE WAF                            │
│                    (highclerecastleshop.co.uk / *.highclerecastle.tv)                    │
└────────────────────────────────────────────┬─────────────────────────────────────────────┘
                                             │ HTTPS (80/tcp, 443/tcp on bastion_public_interface)
                                             ▼
┌──────────────────────────────────────────────────────────────────────────────────────────┐
│  EDGE BASTION HOST: hs-ct-bast-p-02 (10.10.30.50 · roles/common + roles/bastion)         │
│  • Standalone Compose (/home/mad/docker/bastion): Traefik v3.7 + CrowdSec v1.8.1         │
│  • File-Provider-Only Routing (Zero /var/run/docker.sock exposure)                       │
│  • Let's Encrypt DNS-01 Wildcard TLS (acme.json 0600) + Cloudflare trustedIPs            │
└──────────────────────┬────────────────────────────────────────────┬──────────────────────┘
                       │ Internal SDN (10.10.30.0/24)               │ Internal SDN (10.10.30.0/24)
                       │ UFW: Allow 3000/tcp ONLY from SDN          │ UFW: Allow 3000/tcp ONLY from SDN
                       ▼                                            ▼
┌─────────────────────────────────────────────┐  ┌─────────────────────────────────────────┐
│  PRODUCTION NODE: hs-ct-apps-p-01           │  │  STAGING NODE: hs-ct-apps-s-01          │
│  (10.10.30.61 · git_branch: "main")         │  │  (10.10.30.60 · git_branch: "staging")  │
│  • apps/highclere_tickets (Docker Compose)  │  │  • apps/highclere_tickets (Compose)     │
│  • Astro SSR (:3000) + PostgreSQL 16        │  │  • Astro SSR (:3000) + PostgreSQL 16    │
│  •NODE_ENV=production                       │  │  • NODE_ENV=staging                     │
│  • EMAIL_ALLOWLIST="" (Live Postmark)       │  │  • EMAIL_ALLOWLIST=staff@... (Locked)   │
│  • highclere-etl-sync.timer (Every 60s)     │  │  • Mirrored 506 MB PostgreSQL dataset   │
│  • highclere-retention-prune.timer (03:15)  │  │                                         │
└──────────────────────┬──────────────────────┘  └─────────────────────────────────────────┘
                       │
                       │ Outbound MySQL (3306/tcp over Internal SDN / NetBird wt0)
                       │ Read-Only Incremental Batch Poll (Every 60s)
                       ▼
┌──────────────────────────────────────────────────────────────────────────────────────────┐
│  QUARANTINED LEGACY LAMP VM (10.10.30.20 · Zero Public Ingress)                          │
│  • Legacy PHP 7.4 (/admin) reachable ONLY by Castle Staff over NetBird WireGuard Mesh    │
│  • Legacy MariaDB 10.11 read by hs-ct-apps-p-01 ETL sync until Component 4 cutover       │
└──────────────────────────────────────────────────────────────────────────────────────────┘

In apps/highclere_tickets, the host's git_branch variable (defaulting to "main") controls both the Gitea repository checkout and the Jinja2 synthesis of /home/mad/docker/highclere-tickets/.env (mode '0600', owned by mad:mad, rendered with no_log: true):

# apps/highclere_tickets/templates/env.j2
APP_BIND_IP={{ highclere_tickets_bind_ip }}
{% if git_branch == 'main' %}
NODE_ENV=production
PUBLIC_SITE_URL=https://{{ bastion_domain }}
{% elif git_branch == 'staging' %}
NODE_ENV=staging
PUBLIC_SITE_URL=https://staging.{{ bastion_domain }}
{% else %}
NODE_ENV=development
PUBLIC_SITE_URL=https://{{ git_branch }}.{{ bastion_domain }}
{% endif %}
PORT={{ highclere_tickets_port }}
SESSION_SECRET={{ highclere_tickets_session_secret }}
POSTGRES_USER={{ highclere_tickets_postgres_user }}
POSTGRES_PASSWORD={{ highclere_tickets_postgres_password }}
POSTGRES_DB={{ highclere_tickets_postgres_db }}
DATABASE_URL=postgresql://{{ highclere_tickets_postgres_user }}:{{ highclere_tickets_postgres_password }}@postgres:5432/{{ highclere_tickets_postgres_db }}
LEGACY_MARIADB_URL={{ highclere_tickets_legacy_mariadb_url }}
EMAIL_MODE={{ highclere_tickets_email_mode }}
POSTMARK_SERVER_TOKEN={{ highclere_tickets_postmark_server_token }}
EMAIL_FROM={{ highclere_tickets_email_from }}
EMAIL_ALLOWLIST={{ highclere_tickets_email_allowlist if git_branch != 'main' else '' }}
CONTACT_EMAIL_DEFAULT={{ highclere_tickets_contact_email_default }}
COMPANY_VAT_NUMBER={{ highclere_tickets_company_vat_number }}

To rebuild a degraded or destroyed application node (hs-ct-apps-s-01 or hs-ct-apps-p-01) from a bare Debian 12/13 installation:

# 1. Bootstrap clean Debian VM from Ansible controller (initial run as root creates 'mad' & hardens SSH)
ansible-playbook site.yaml -i inventory.yaml --limit hs-ct-apps-p-01 -e "ansible_user=root"

# 2. Subsequent runs connect seamlessly as unprivileged 'mad' user with zero drift (changed=0)
ansible-playbook site.yaml -i inventory.yaml --limit hs-ct-apps-p-01

# 3. Restore PostgreSQL state (either from compressed pg_dump or full idempotent MariaDB ETL import)
gunzip -c highclere_tickets_backup.sql.gz | \
  docker compose -f /home/mad/docker/highclere-tickets/docker-compose.prod.yaml exec -T postgres \
  psql -U highclere highclere_tickets
# OR reconstruct directly from legacy MariaDB:
docker compose -f /home/mad/docker/highclere-tickets/docker-compose.prod.yaml exec -T app \
  npm run db:sync-legacy -- --full

# 4. Run the 6-Point Post-Install Verification Checklist on the target host:
git -C /home/mad/docker/highclere-tickets status
sudo ls -l /home/mad/docker/highclere-tickets/.env                 # Must be -rw------- (0600) mad:mad
sudo ufw status verbose | grep 3000                                # Allowed ONLY from 10.10.30.0/24
docker compose -f /home/mad/docker/highclere-tickets/docker-compose.prod.yaml ps
curl -sI http://127.0.0.1:3000/api/health                          # HTTP/1.1 200 OK
systemctl list-timers 'highclere*'                                 # ETL (60s) & Prune (03:15 UTC) active