Component 2 · Architectural Reference

Server Blueprint & Recovery Decisions Declarative Infrastructure Decisions (`debian-ansible`)

Built & Verified

This page records the seven core operational and security decisions behind our automated server blueprint, explaining how we eliminated fragile hand-maintained servers and made disaster recovery fast, calm, and predictable.

This document records the architectural and operational decisions for Component 2 (Declarative Infrastructure & Automated Recovery via debian-ansible), synthesizing our debian-ansible ADRs (ADR-0001 through ADR-0004), the Modulix capability-compiler model, and our Rebuild, Don't Repair recovery philosophy.

Executive Summary of Infrastructure Decisions

Our server infrastructure protects Highclere Castle across seven foundational decisions:

1. Rebuild, Don't Repair: We treat servers as replaceable appliances built from a master blueprint rather than fragile "Jenga towers" repaired by hand during an outage.
2. Single Master Server List: Every Castle server is listed once in a master file with simple on/off capability switches, eliminating human setup errors.
3. Self-Contained Application Deployment: The ticketing website pulls verified code directly from our version-control system without complex multi-server cluster overhead.
4. Dedicated Outer Security Shield: An outer gateway blocks cyber attacks and manages security certificates without ever exposing internal server controls.
5. Automatic Test Server Email Lock: Staging servers automatically lock outbound emails to staff addresses only, protecting real customers during testing.
6. Monitored System Timers: Fragile background scripts are replaced with monitored system timers for the 60-second office sync and 3:15 AM privacy cleanup.
7. Anti-Lockout Safety Guards: Remote login rules and firewall changes are automatically verified before activation so staff can never be locked out.

1 · Recovery Philosophy

Disposable Servers vs. The IT Jenga Tower

Decision 2.1: "Rebuild, Don't Repair" over Manual Server Troubleshooting

Disaster Recovery

What we decided: We never log into a broken server to manually patch files or guess what went wrong. Instead, all server settings live outside the machine in our master blueprint. If a server ever develops a fault, we replace it from the blueprint in minutes.

Why We Ended Manual Server Repairs

Over 15 years, manual edits turned the old server into a fragile Snowflake Server (a "Jenga tower" where nobody knew which old setting kept the system standing). By keeping all booking data safely separated in the database and building the server from a repeatable plan, replacing a server becomes a calm, 5-minute routine.

Decision 2.1: Immutable "Rebuild, Don't Repair" Convergence

Resilience & MTTR

What we decided: All Debian hosts and LXC/VM instances are treated as disposable appliances. Manual SSH mutation is prohibited. Every task in debian-ansible is strictly idempotent (yielding changed=0 on subsequent runs), and stateless application containers (/home/mad/docker/highclere-tickets) are decoupled from persistent PostgreSQL storage (postgres_data).

Why In-Place Repair Was Rejected

Repairing a long-lived Snowflake Server during an outage turns engineering into slow digital archaeology. When desired state is codified outside the machine, wiping a degraded node and re-running ansible-playbook site.yaml -i inventory.yaml guarantees a known-good production state in minutes.

2 · Inventory & Capability Compilation

Host-Centric Inventory & Pre-Flight Safety Checks

Decision 2.2: Single Master Server List with Capability Switches

Human Error Prevention

What we decided: Every Castle server is defined once in a single master list (Host-Centric Inventory) and given simple on/off capability switches (such as bastion for the outer security shield or highclere_tickets for the ticketing website).

Automatic Pre-Flight Safety Checks

Before the blueprint touches any server, it automatically checks that every required security key, password, and network address is present and valid. If anything is missing, it stops immediately and alerts the operator before any change is made.

Decision 2.2: Host-Centric Inventory (`all.hosts`) & Composable Capability Flags (ADR-0002)

ADR-0002

What we decided: Rather than duplicating hostnames across static Ansible group hierarchies, each server is declared once under all.hosts in inventory.yaml with boolean Composable Capability Flags (docker, bastion, fail2ban, netbird, highclere_tickets). Play 1 of site.yaml executes local ansible.builtin.assert pre-flight checks (validating mandatory credentials and asserting highclere_tickets_session_secret | length >= 32) before dynamically constructing target groups via ansible.builtin.group_by.

Alignment with the Modulix Capability Model

This directly implements the Modulix principle of declaring high-level capability intent per host and letting the translation layer resolve implied prerequisites (for example, highclere_tickets: true and bastion: true automatically imply docker_nodes membership and universal OS hardening).

3 · Application & Perimeter Architecture

Git-Backed Compose & Zero-Socket Edge Bastion

Decision 2.3

Self-Contained Website Deployment from Version Control

What we decided: The ticketing website and its database run as a self-contained unit on their dedicated server, pulling verified code directly from our private Git version-control system.

Why: Avoids complex multi-server cluster software that can slow down emergency recovery, while allowing website updates to be released or rolled back in seconds without altering server firewall blueprints.

Decision 2.4

Dedicated Outer Security Shield (Edge Bastion)

What we decided: Public visitors connect through a dedicated outer security server (hs-ct-bast-p-02) that blocks cyber attacks and manages website padlocks before forwarding traffic to the ticketing server over a private internal network.

Why: Protects the ticketing server from direct internet exposure and ensures internal server controls are never exposed to the public web.

Decision 2.3 · ADR-0004

Git-Backed Standalone Docker Compose over Swarm

What we decided: apps/highclere_tickets deploys the application as a Git-backed standalone Docker Compose stack (docker-compose.prod.yaml) under /home/mad/docker/highclere-tickets via community.docker.docker_compose_v2.

Why: Running a single-node stateful PostgreSQL + Astro SSR workload in Docker Swarm adds overlay networking and quorum overhead during disaster recovery. Versioning Dockerfile, docker-compose.prod.yaml, and Drizzle migrations inside highclere-tickets decouples app releases from Ansible tasks while preserving single-command host rebuilds.

Decision 2.4 · ADR-0003

File-Provider-Only Traefik v3 + CrowdSec Bastion

What we decided: roles/bastion deploys Traefik v3 (traefik:v3.7) and CrowdSec (crowdsecurity/crowdsec:v1.8.1) using dynamic file configuration (routes.yaml) with zero /var/run/docker.sock mount, DNS-01 wildcard ACME TLS, and UFW on app hosts restricting port 3000/tcp strictly to the Bastion SDN (10.10.30.0/24).

Why: Exposing the Docker socket to an edge proxy container is a critical container-escape risk. File-provider routing + SDN firewalling isolates the perimeter completely.

4 · Environment Safety & Background Scheduling

Branch-Driven Safety Locks & Native Systemd Timers

Decision 2.5

Automatic Email Lock on Staging Servers

What we decided: Setting a server's blueprint switch to staging automatically locks its email system so it can only send messages to approved Castle staff addresses.

Why: Allows staff to test the system against all 906,406 real historical records with a 100% guarantee that no real customer will ever receive a test email by mistake.

Decision 2.6

Monitored System Timers with Phased Safety Switches

What we decided: Replaced old, unmonitored background scripts with modern system timers for the 60-second office sync, the 3:15 AM privacy cleanup, and nightly security updates.

Why: Every background run is centrally logged and can be switched on in controlled stages when a new server is launched.

Decision 2.5

Single-Branch `.env` Synthesis & `EMAIL_ALLOWLIST`

What we decided: apps/highclere_tickets/templates/env.j2 (mode '0600', mad:mad, no_log: true) derives NODE_ENV, PUBLIC_SITE_URL, and EMAIL_ALLOWLIST directly from git_branch.

Why: Whenever git_branch != 'main', EMAIL_ALLOWLIST is populated automatically with highclere_tickets_email_allowlist, making accidental customer emails from staging structurally impossible.

Decision 2.6 · ADR-0001

Systemd Timers & Phased Activation Booleans

What we decided: Manual cron jobs are banned (ADR-0001). Background jobs run via native systemd units (highclere-etl-sync.timer, highclere-retention-prune.timer, and apt-daily-upgrade.timer drop-ins) gated by explicit inventory flags (highclere_tickets_migrate_enabled, highclere_tickets_etl_timer_enabled, highclere_tickets_prune_timer_enabled).

Why: Provides deterministic OnCalendar execution, structured journalctl logs, and safe phased cutovers during initial host provisioning.

5 · Host Hardening & Pipeline Governance

Unprivileged Runtime, Lockout Rollback & 3-Stage Agent Pipeline

Decision 2.7: Anti-Lockout Safety Guards & 3-Stage Blueprint Auditing

Operational Safety

What we decided: Daily website operations run under a restricted system account rather than unrestricted administrator access. Whenever remote login or firewall settings are updated, the server automatically tests the new configuration before activating it (rolling back automatically if a typo is detected), and trusted office IP addresses are protected across all security shields.

Every update to the server blueprint must pass through three independent stages: 1. Documentation Alignment, 2. Blueprint Generation, and 3. Automated Security Validation.

Decision 2.7: Unprivileged `mad` Account, `sshd -t` Rescue & 3-Stage Pipeline

Security & Governance

What we decided:

Least-Privilege Runtime (mad): Container stacks under /home/mad/docker and Git checkouts execute as the unprivileged mad user; personal admin access uses named admin_users with visudo -cf %s validation.
Lockout Prevention: roles/common wraps /etc/ssh/sshd_config.d/00-hardening.conf in an Ansible block/rescue construct verified by sshd -t, opens 22/tcp before enabling UFW, and shares security_whitelist_ips across Fail2ban and CrowdSec.
3-Stage Sub-Agent Pipeline: All repository changes follow Documenter -> Generator -> Validator (plus optional Stage 0 Researcher), enforcing FQCN modules, no_log: true on secret tasks, and changed=0 convergence.

← Back to Component 2 Blueprint Overview View Full Component Roadmap →