Technology · Recovery Philosophy · Coding Standards

Technology Stack & Engineering Methodology

Production Engineering Standards

Replacing an ageing website is not just about updating visual pages. It is about replacing fragile, manually maintained servers with automated, self-healing systems, strict version control, and disciplined engineering habits that protect Castle revenue, visitor privacy, and gate-side reliability every day of the year.

This page documents our complete technology stack transition (from legacy PHP 7.4 / CodeIgniter 3 / MariaDB MyISAM to Node.js 20, TypeScript, Astro SSR, Hono, PostgreSQL 16, and Ansible-provisioned Docker hosts), our "Rebuild, Don't Repair" operational recovery philosophy, our three-tier Git promotion pipeline, and the non-negotiable coding standards enforced across every commit.

Five Core Improvements Over the Previous System

1. Reliability & Fast Recovery

Instead of spending hours manually fixing a broken server, our automated setup can build a brand-new, fully working replacement server from scratch in minutes without losing a single customer booking.

2. Multi-Layer Security

Customer passwords are eliminated completely, staff administrative tools are hidden inside our private office network, and automated security shields block malicious traffic before it reaches Castle servers.

3. Automatic Legal Compliance

UK GDPR and HMRC 8-year data retention rules run automatically every night at 03:15 AM, while safety locks guarantee real visitors can never be emailed accidentally during staff testing.

4. Gate-Side Speed & Performance

Tickets and barcodes are generated instantaneously in memory rather than saving temporary image files on the server, allowing pages to load in under a second even on weak mobile signal at the Castle gates.

5. Full Audit Trail & Instant Rollback

Every single update to the website is permanently logged with who made it, when, and why. If any change ever causes an unexpected issue, the system can be rolled back to its previous working state in seconds.

Side-by-Side Comparison

1. Legacy System vs Modern Platform Stack

Every tool in the new platform was chosen to eliminate a specific weakness discovered in the old 2012-era website:

Area	Previous System (2012 Legacy)	New Castle Platform	Practical Benefit for Highclere
1. Server Setup & Maintenance	Manually configured single server running outdated software that no longer receives security updates. Difficult and slow to recreate after a hardware fault.	Automated Server Blueprints Servers are built automatically from a master blueprint with built-in firewalls and automatic security updates.	A fresh, identically secured server can be launched in minutes rather than days of manual IT recovery.
2. Network & Staff Protection	The staff administration login page was visible on the public internet, exposed to automated password-guessing bots.	Private Office Network & Active Shield Staff tools are locked inside a private office network, while public pages sit behind enterprise firewalls.	Outside attackers cannot even see or reach internal staff tools, while office staff work uninterrupted.
3. Updating the Website	Changes were made directly to live files on the server with no testing stage and no record of what was changed.	Three-Stage Version Control All changes pass through Development and Staging checks before going live, with every edit recorded.	Zero risk of untested edits breaking live ticket sales, plus one-command rollback at any time.
4. Website Engine & Barcodes	Older web engine that created and saved temporary image files on the server hard drive every time a visitor opened their ticket.	Modern High-Speed Engine Generates crisp, scalable digital barcodes instantly in memory without touching the server hard drive.	Instant ticket loading on mobile phones at the Castle entrance gates and crystal-clear home printing.
5. Customer & Financial Database	118 fragile data tables that lacked financial safeguards and duplicated long event instructions 410,000 times (1.28 GB).	Enterprise Banking-Grade Database Strict financial protections, consolidated customer profiles, and single-place event instructions (506 MB).	Impossible to corrupt an order mid-payment or double-book capacity; 60% smaller and significantly faster.
6. Sign-In, Email & Data Privacy	Stored customer passwords in readable text, exposed staff email addresses to spam bots, and kept 14 years of dormant accounts.	Password-Free Sign-In & Daily Privacy Cleanup Single-use 6-digit email codes, spam-protected contact forms, and automatic nightly removal of expired pre-8-year records.	Zero customer passwords to steal in a breach and continuous, hands-off UK GDPR & HMRC compliance.

Each layer of the new stack directly remediates an architectural or security vulnerability identified during our forensic audit of the legacy LAMP application:

Layer	Legacy Stack (PHP 7.4 / MyISAM)	Modern Stack (Node 20 / Postgres / Ansible)	Reliability, Security & Performance Gain
1. Infrastructure & OS Provisioning	Hand-configured LAMP server running EOL PHP 7.4; manual package drift; undocumented local state.	Fasthosts Proxmox VE + `debian-ansible` + Docker Compose Automated provisioning of `hs-ct-apps-s-01` (staging) and `hs-ct-apps-p-01` (prod) with hardened SSH, UFW, `unattended-upgrades`, and isolated `mad` execution user.	Deterministic, reproducible host builds; zero configuration drift between staging and production.
2. Ingress, WAF & VPN Containment	Legacy Nginx Proxy Manager bastion with publicly reachable `/admin` and no collaborative threat intelligence.	Cloudflare WAF + Traefik & CrowdSec (`hs-ct-bast-p-02`) + NetBird VPN Edge WAF filtering, CrowdSec real-time IP reputation blocking at the bastion, and WireGuard-based NetBird mesh isolating legacy `/admin` and internal ports (`5432`, `3306`).	Zero public attack surface for administrative tools and database ports; automated bot and scanner mitigation.
3. Source Control & Release Pipeline	Ad-hoc edits directly on live server files (`live_files_backup`); no branch gates or automated test suite.	Self-Hosted Gitea (`dev` → `staging` → `main`) Atomic commits on `dev`, fast-forward promotion (`--ff-only`) to `staging` after Validator sign-off, and fast-forward release to `main`.	Cryptographic audit trail for every line of code; zero unreviewed production changes; instant `git checkout` rollback.
4. Application Runtime & Barcodes	CodeIgniter 3 (PHP 7.4) coupling SQL inside views and writing temporary `.png` / `.gd2` barcodes to disk via `Tickets_model::drawBarcodeAsImage()`.	Node.js 20 + TypeScript + Astro SSR + Svelte 5 + Hono Strict TypeScript compilation, thin SSR templates, Hono API routes, and pure in-memory Code 128B `<svg>` barcode generation (`src/domain/barcode.ts`).	Zero filesystem I/O for tickets, sub-second TTFB on mobile connections, and compile-time elimination of type errors.
5. Persistence & Schema Evolution	118-table MariaDB 10.11 using non-transactional `MyISAM` engine, integer timestamps, `0000-00-00` dates, and 782 MB of duplicated item text.	PostgreSQL 16 + Drizzle ORM ACID transactions, foreign keys, `TIMESTAMPTZ`, `NUMERIC(12,2)`, normalized `event_instructions` (176 kB), and versioned SQL migrations (506 MB total footprint).	60.4% storage reduction, row-level locking for ticket capacity, and zero orphaned or partially written orders.
6. Auth, Messaging & Compliance	Plaintext `users.password` in DB, exposed staff `mailto:` links, and 271k dormant pre-2018 accounts retained indefinitely.	Postmark OTP + `EMAIL_ALLOWLIST` + Systemd Timers SHA-256 hashed 6-digit OTPs/Magic Links, transport-level email allowlist in dev/staging, and daily `highclere-retention-prune.timer` at `03:15 UTC`.	Zero password-breach liability, impossible to email real customers from staging, and automated UK GDPR / HMRC compliance.

Disaster Recovery & Resilience

2. The "Rebuild, Don't Repair" Recovery Philosophy

In traditional IT management, when a server develops a fault or suffers a security incident, technicians log into the broken machine and spend hours trying to patch and repair it by hand. That approach is slow, stressful on a busy Castle opening morning, and often leaves hidden problems behind.

Our platform is engineered around the opposite principle: "Rebuild, Don't Repair." We treat servers as replaceable appliances rather than fragile heirlooms. If a server ever misbehaves, we do not waste time repairing it in place: we switch it off and automatically build a clean, verified replacement in minutes.

Nothing Stored Only on the Web Server

The website application never saves unique files, temporary barcode images, or customer sign-in sessions solely on its own hard drive. Every piece of permanent data lives safely inside the dedicated database. Because the web server holds no one-of-a-kind files, it can be replaced at any moment without disrupting a single logged-in customer.

Self-Healing Data Synchronization

Our database synchronization tool is designed so it can be run as many times as needed without ever creating duplicate bookings or double-counting gift vouchers. Even if a brand-new database server is created from scratch, the system automatically populates and verifies all 906,406 active records cleanly.

Automated Server Blueprints

Instead of following a manual checklist to set up a server, our infrastructure blueprint automatically installs the operating system, configures the security firewall, locks down access, and launches the website identically every single time.

Automatic Startup Safety Checks

Whenever the website starts up, it automatically checks that every required security key, database connection, and email safety setting is present and valid before accepting visitor traffic. If a setting is missing, it alerts staff immediately rather than failing mid-booking.

Legacy systems treat servers as "pets" that must be manually nursed back to health via interactive SSH sessions when packages break, disks fill with temporary files, or state corrupts. In contrast, our platform is engineered strictly around an immutable "Rebuild, Don't Repair" philosophy: if an application container or VM exhibits anomalies, we terminate it and provision a clean replacement from code.

Every architectural decision in highclere-tickets is written specifically to make disposable recovery effortless:

1. 100% Stateless Application Containers

The Node.js / Astro SSR container writes zero persistent state to local disk. Code 128B barcodes are rendered purely in-memory as inline <svg> strings, and authentication state (auth_otp_tokens and sessions) is persisted in PostgreSQL rather than process memory. Running docker compose -f docker-compose.prod.yaml up -d --force-recreate replaces the runtime in under 3 seconds without dropping a single active customer session.

2. Deterministic, Idempotent Data & Schema State

Both schema migrations (npm run db:migrate via Drizzle) and the MariaDB-to-PostgreSQL ETL pipeline (npm run db:sync-legacy) are strictly idempotent. All ETL writes use batched INSERT ... ON CONFLICT DO UPDATE guarded by PostgreSQL advisory lock 84729104, and the daily retention job (npm run db:prune-retention) is similarly repeatable. A fresh PostgreSQL instance can reconstruct the exact 506 MB commercial state deterministically at any time.

3. Infrastructure-as-Code Host Rebuilds (`debian-ansible`)

Host configuration is never mutated ad-hoc. Our debian-ansible playbook provisions a raw Debian VM on Proxmox VE with the mad execution user, SSH key-only hardening, UFW firewall rules, Docker Engine, and NetBird VPN peer enrollment. Systemd units (highclere-etl-sync.timer and highclere-retention-prune.timer) live directly in the repository under deploy/systemd/.

4. Fail-Fast Startup Environment Contract (`src/config/env.ts`)

Shared application code is forbidden from reading scattered process.env variables. On startup, src/config/env.ts loads .env via Node's native process.loadEnvFile() and validates every required setting (DATABASE_URL, SESSION_SECRET, EMAIL_MODE, EMAIL_ALLOWLIST, COMPANY_VAT_NUMBER) against a strict Zod schema, failing fast before binding a port if a server is misconfigured.

Traceability & Change Control

3. Every Change Documented & Instant Rollback via Git

On the old website, code files were edited directly on the live server. If a change accidentally broke a page or miscalculated a price, there was no automatic log showing what had been altered or how to undo it cleanly.

In our new workflow, no file is ever edited directly on the live server. Every change follows a strict, three-stage approval pipeline that acts as both a safety net and a permanent historical record:

Stage 1: Development

Building & Automated Testing

Every improvement is built in an isolated workspace and saved as a small, clearly labelled update explaining what changed and why. Over 140 automated checks verify calculations, barcodes, and privacy rules before it can move forward.

Stage 2: Staging Preview

Safe Dress Rehearsal

Approved updates are promoted to a private Staging server that mirrors the live system (with outbound customer emails locked to staff addresses only). Here, changes are verified in a real browser without affecting the public.

Stage 3: Live Production

Instant Promotion & Rollback

Only after passing Staging is the exact verified version promoted to the public website. Because every previous version is preserved intact, we can revert the website to any earlier point in time in seconds.

Both repositories (highclere-tickets and hc-dr) enforce a strict three-tier branch promotion pipeline hosted on our internal Gitea instance. Direct commits to staging or main are prohibited:

┌──────────────────────────────────────────────────────────────────────────────────────┐
│  1. dev (Active Engineering Branch)                                                  │
│     • Granular, atomic commits (one logical change per commit)                       │
│     • Validated via: npm run check (astro check + tsc --noEmit) & npm test (Vitest) │
└──────────────────────────────────────────┬───────────────────────────────────────────┘
                                           │ Mandatory Validator Gate Sign-Off
                                           │ git checkout staging && git merge --ff-only dev
                                           ▼
┌──────────────────────────────────────────────────────────────────────────────────────┐
│  2. staging (Pre-Production QA on hs-ct-apps-s-01)                                   │
│     • Mirrors production container topology & 506 MB synced PostgreSQL dataset       │
│     • EMAIL_ALLOWLIST enforced so live customers can never be emailed during QA      │
└──────────────────────────────────────────┬───────────────────────────────────────────┘
                                           │ Stakeholder & Operational Verification
                                           │ git checkout main && git merge --ff-only staging
                                           ▼
┌──────────────────────────────────────────────────────────────────────────────────────┐
│  3. main (Production Release on hs-ct-apps-p-01)                                     │
│     • Exact SHA parity with verified staging commit (--ff-only)                      │
│     • Instant deterministic rollback to any prior commit SHA if ever required        │
└──────────────────────────────────────────────────────────────────────────────────────┘

Atomic Commits as Documentation

We never batch unrelated database changes, domain logic, and UI tweaks into monolithic commits. Each commit is scoped to a single unit of work with a descriptive commit message, turning git log and git blame into a permanent forensic record of why every line exists.

Zero-Drift Fast-Forward Releases (`--ff-only`)

Promoting from dev to staging and from staging to main uses strict git merge --ff-only. This guarantees that the exact commit SHA tested on staging is bit-for-bit identical to the commit deployed to production, making rollbacks a single deterministic command.

Code Craftsmanship & Governance

4. Engineering Methodology & Coding Style Principles

Technology tools are only as reliable as the discipline used to write them. We follow six strict working rules so the system remains simple to understand, safe to update, and free of hidden traps:

1. Agree the Business Rules Before Building

Before writing code for any component, we examine real historical bookings and agree on exact definitions for every term (such as how partial refunds, gift voucher uplifts, or family tickets work). Every decision is written down in plain language on this site so management, office staff, and engineers share the exact same understanding.

2. Keep It Simple & Readable (KISS)

We avoid overly clever or tangled code. Every calculation and rule is written in a straightforward, step-by-step style. If an urgent question arises on a busy Castle event morning, an engineer can open the relevant file and read the exact rule in seconds.

3. Separate Visual Design from Financial Logic

Web page templates only control how information looks on screen or on printed paper; they never contain financial calculations, tax rules, or barcode formulas. This means visual layouts can be refreshed in the future without any risk of disturbing ticket barcodes or VAT invoice totals.

4. Stop Old System Quirks at the Border

The old database contained years of quirky formatting and duplicate entries. Our synchronization bridge cleans and repairs all of those quirks before the data enters our new database, ensuring the new website remains completely clean from day one.

5. Deliver in Priority Stages

Rather than waiting months for an all-at-once replacement, we deliver the platform in focused, high-priority stages: starting with data protection and customer ticket retrieval, followed by public ticket sales, staff back-office tools, and coach operator bookings.

6. Automated Quality & Privacy Guardrails

Every update must pass over 140 automated tests that check barcode accuracy, VAT invoice maths, gift voucher balances, and security rules. Strict formatting rules also ensure staff email addresses and office phone numbers are protected from internet spam bots.

Our codebase enforces six non-negotiable engineering disciplines (codified in AGENTS.md and verified by our automated Validator suite):

1. Relentless Design Grilling & Ubiquitous Language (`/grill-with-docs`)

Before implementation, every slice undergoes a structured design interview stress-tested against the actual 850 MB production dataset. Canonical domain terms (Customer, Order, Order Item, Ticket, VAT Invoice, Gift Voucher, Gate List) are locked in CONTEXT.md and used identically across PostgreSQL tables, TypeScript types, and UI copy.

2. KISS: Flat, Explicit TypeScript & Strict Validation

We reject deep class inheritance hierarchies, abstract factories, and metaprogramming. Domain modules use flat, explicit pure functions with strict TypeScript types (astro/tsconfigs/strict) and Zod validation at every boundary. YAML files strictly use .yaml (never .yml), and UI/docs follow strict typography and anti-spam hygiene.

3. Decoupled Domain Layer & "Thin Views" (ADR-0003)

.astro pages and .svelte islands are strictly thin presentation templates. They are forbidden from importing drizzle-orm, pg, mysql2, or process.env. All business logic (VAT calculation, Code 128B SVG encoding, OTP verification, ticket merging) lives in pure TypeScript under src/domain/ and Hono endpoints in src/api/.

4. Zero Legacy Bleed at the ETL Boundary (ADR-0001)

Only src/etl/ may connect to the legacy MariaDB instance. Legacy quirks (Unix integer timestamps, 0000-00-00 dates, the 28 non-ticket SKU exclusion list, sagepay_refunds status reconciliation, HTML entity decoding, and duplicate email rows) are normalized inside src/etl/normalizer.ts and never leak into src/domain/ or src/pages/.

5. Specialist Sub-Agent Execution Model

To prevent context bloat and regressions, complex slices are decomposed across five focused engineering roles: SCHEMA-ETL (src/db/, src/etl/), DOMAIN-API (src/domain/, src/api/), FRONTEND (src/pages/, STYLE_GUIDE.md), VALIDATOR (QA & security gate), and DOCUMENTER (ADRs & stakeholder portal).

6. Mandatory Validator Sign-Off Gate

No slice is marked complete without passing the full Validator checklist: npm run check (0 errors), npm test (141/141 unit tests passing), two-run full + incremental ETL idempotency verification, Code 128B barcode bit-accuracy checks against legacy PHP output, secret leak scans, and EMAIL_ALLOWLIST transport enforcement.