Highclere Castle Ticketing & Commerce · Disaster Recovery Programme
View:
Viewing in Plain English mode. Switch to Technical & Engineering in the top bar at any time to view database schemas, code specifications, and architecture diagrams.
Disaster Recovery & Modernisation

Build in parallel.
Cut over with zero interruption to ticket sales.

The Castle's previous online ticketing shop was built over 15 years ago on outdated software that stores customer passwords without modern encryption and suffers from duplicate customer records. Rather than attempting a risky repair of obsolete code or pausing public ticket sales with a temporary maintenance screen, we are building and verifying a complete modern replacement in parallel (Components 1 to 6) synced every 60 seconds from the live database, before switching public traffic over seamlessly and locking the old server inside our private office network for coach bookings only.

The legacy highclerecastleshop.co.uk application runs on end-of-life PHP 7.4 and CodeIgniter 3 with 118 unindexed/MyISAM database tables and plaintext customer passwords. Rather than attempting an in-place upgrade or taking the public shop offline prematurely with a holding page, we continuously mirror legacy MariaDB into a clean PostgreSQL 16 core every 60 seconds while completing Components 1 to 6, then execute a single public cutover - severing public access to PHP 7.4 and isolating the legacy VM behind NetBird VPN strictly for Coach Bookings until Component 7 completes.

Why We Are Acting Immediately: Findings from Legacy Backup
  • End-of-Life Software Exposure: The old website ran on discontinued server software with known security vulnerabilities that cannot be safely left exposed to the public internet.
  • Unencrypted Customer Passwords: The old system stored customer and coach passwords in plain text without modern encryption, creating an urgent security and data protection risk.
  • Duplicate Accounts & Fragile Database: The legacy database lacked basic integrity checks, allowing hundreds of customers to have duplicate accounts under the same email and causing frequent login failures.
  • Silent Calculation Flaws: Flaws in the old code silently zeroed out commercial cancellation penalties on coach bookings, leaking revenue.
  • End-of-Life Runtime Exposure: PHP 7.4 and CodeIgniter 3 are exposed to active public CVE exploitation and lack modern request validation or CSRF/SQLi guardrails.
  • Plaintext Customer Passwords: Despite a December 2018 code comment intending to hash passwords, Customers_model::getUserLogin() and Coachvisits still store and compare passwords in plaintext across users (~498k rows) and coach_companies.
  • Zero Database Transactions (MyISAM): Core commercial tables (users, orders, order_item, user_info) use the legacy MyISAM storage engine - meaning no ACID transactions, no foreign-key integrity, and duplicate customer accounts sharing the same email address.
  • Silent Logic Bugs in Legacy Code: For example, Coachvisits::_calculate_penalty() (line 2257) contains an assignment bug (if ($order_id = 3649) { $penalty = 0; }) that silently zeroes out cancellation penalties for every coach booking.
Transition & Cutover Strategy

Parallel Build Followed by Seamless Public Cutover (After Component 6)

To avoid any pause in online ticket sales or disruption to office staff, the existing website continues serving visitors while our modern replacement is built, synced, and verified in parallel through Component 6.

Step 1 · Live 60s Data Mirror

Build & Verify in Parallel

While the existing website stays online, an automated background pipeline copies and cleans every customer booking, ticket, and gift voucher into our new database every 60 seconds as Components 1 to 6 are completed.

Step 2 · Public Cutover (After C6)

Switch Public Website Seamlessly

Once Customer Sign-In (C1), Server Blueprints (C2), Public Checkout (C3), Off-Site Backups (C4), Staff Administration (C5), and Management Reports (C6) are verified, public traffic switches directly to the new platform.

Step 3 · Legacy Locked to Office VPN

Isolate Old Server for Coach Tours Only

At cutover, all public internet access to the old server is removed. Office staff access the old system exclusively over our private office connection to manage Coach Bookings until Component 7 powers it down permanently.

Instant "Break-Glass" Emergency Option (Available Today)

Because Phase 0 (Live 60s Data Mirror), Component 1 (Customer Portal & Ticket Retrieval), and Component 2 (Automated Server Blueprint) are already 100% built and verified, the Estate does not have to wait until Component 6 if an emergency strikes. If the legacy website were ever breached or suffered a critical failure tomorrow, we can activate Component 1 in minutes as an immediate break-glass shield: cutting off public access to the old server while displaying an official Castle holding notice and keeping existing ticket retrieval, barcode printing, and office VPN operations fully working.

Parallel ETL Mirroring & Post-Component-6 Public Cutover

Rather than deploying a standalone customer login holding page and pausing ticket sales, we keep the legacy shop operating while continuously syncing MariaDB into PostgreSQL every 60 seconds until the complete public and back-office ticketing stack (Components 1 to 6) is ready for cutover.

Step 1 · Continuous ETL Mirror

Build Components 1 to 6 Against Live Data

highclere-etl-sync.timer runs every 60 seconds over the private network, keeping our 506 MB PostgreSQL core synchronized with live MariaDB orders and vouchers while Components 1 to 6 are verified on Staging.

Step 2 · Atomic Cutover (After C6)

Switch DNS / Bastion Routing to New Stack

After Component 6 sign-off, run a final incremental ETL delta, point public highclerecastleshop.co.uk routing to hs-ct-bast-p-02 → hs-ct-apps-p-01, and retire the customer/order ETL sync.

Step 3 · Public Ingress Cut & NetBird Lock

Isolate Legacy PHP for Coach Bookings Only

Immediately drop all public ingress to the legacy PHP 7.4 server and restrict legacy /admin strictly to the NetBird VPN so staff can manage coach_* bookings until Component 7 decommissions the legacy VM.

Break-Glass Emergency Containment Capability (Ready Now via C1 + C2)

While planned public cutover occurs after Component 6, Phase 0 + Component 1 + Component 2 are already built, tested, and continuously synced every 60 seconds. If the legacy PHP 7.4 server suffers a security compromise or outage prior to Component 6 completion, we can execute a break-glass cutover in minutes: severing public ingress to PHP 7.4, isolating legacy /admin behind NetBird VPN, and serving Component 1 (with its built-in /maintenance/ holding view for new bookings and full authenticated Code 128B Ticket + VAT Invoice retrieval for existing ticket holders).

Delivery Sequence

Revenue-Ordered Component Rollout

Instead of a "big bang" rewrite that delays launch for months, we ship one self-contained capability at a time, ordered strictly by customer urgency and commercial revenue impact:

Stage Component Audience Operational Purpose Status
Phase 0 Clean Database & 60-Second Data Mirror Office Staff & IT Established the clean modern database and automated background pipeline continuously synchronising over 906,000 active customer, order, and voucher records since 2018 while the old site remains live. Built & Verified
Component 1 Customer Portal & Ticket Retrieval Existing Ticket Holders Secure email sign-in without passwords, unified booking history, active gift voucher balances, gate-scannable admission tickets, and printable VAT receipts. Built & Verified · Staged for Cutover
Component 2 Automated Server Blueprint & Instant Recovery Estate IT & Security Replaces fragile hand-configured servers with an automated master blueprint, dedicated outer security shield, staging email safety lock, and 5-minute "Rebuild, Don't Repair" disaster recovery. Built & Verified · Staged for Cutover
Component 3 Public Admission & Event Ticket Sales New Castle Visitors Live event booking calendar, timed admission slots, dietary options for afternoon teas, gift voucher redemption, and instant email ticket delivery. Priority 3
Component 4 Automated Off-Site Backups & Verified Disaster Recovery Estate IT & Business Continuity Scheduled encrypted off-site backups of all new ticket orders and automated recovery drills on Staging proving a lost server and database can be restored in minutes. Priority 4
Component 5 Staff Administration Portal Office & Gate Teams Invite-only passwordless staff sign-in with role permissions, booking changes, date moves, cancellations, and gate scanner list downloads, replacing the old administration screen. Priority 5
Component 6 Platform Monitoring, Alerting & Management Reports Castle Management, Finance & IT 24/7 automated server and payment health alerts, plus live financial, VAT, voucher liability, and visitor capacity reports delivered each morning via a secure one-click email link. Priority 6
★ Public Website Cutover & Legacy Containment Line Full Public Launch After Component 6: Switch public website traffic to the new platform with zero downtime, remove public internet access to the old website, and lock the old server inside our private office network strictly for Coach Bookings until Component 7 is complete.
Component 7 Coach Tours Portal & Management Group Tour Operators & Staff Self-service booking portal for coach operators with automated deposit tracking, accurate cancellation rules, and complete retirement of legacy systems. Priority 7
Component 8 Gift Shop Merchandise & Castle Lodges Shoppers & Lodge Guests Online estate gift shop orders and London Lodge overnight stays. Deferred
Stage Component Primary Audience Business Goal & Scope Status
Phase 0 Clean PostgreSQL Core & 60s Legacy ETL Mirror IT / Data Engineering Stand up clean PostgreSQL 16 database, continuous 60-second MariaDB → Postgres incremental ETL sync, and daily GDPR retention pruning (906,406 active records across 506 MB) while legacy site remains live. Built & Verified
Component 1 Customer Portal & Ticket Retrieval Existing Ticket Holders Passwordless Email 6-Digit OTP & Magic Link login (via Postmark), consolidated Order History across merged legacy accounts, Gift Voucher breakdown, dynamic SVG Code 128B Ticket barcodes, and printable VAT Invoices. Built & Verified · Staged for Cutover
Component 2 Declarative Infrastructure & Recovery (debian-ansible) DevOps / SecOps Host-centric Ansible blueprint (all.hosts capability flags), hardened Traefik v3 + CrowdSec Edge Bastion (zero Docker socket exposure), Git-backed Compose deployment, branch-driven .env + EMAIL_ALLOWLIST synthesis, and deterministic "Rebuild, Don't Repair" node recovery. Built & Verified · Staged for Cutover
Component 3 Public Admission & Event Ticket Sales New Visitors Public event & timed-entry catalogue, live capacity calendar with PostgreSQL row-level locking, dietary menu selection, Gift Voucher redemption, gateway-agnostic checkout, and automated Postmark ticket dispatch. Priority 3
Component 4 Automated PostgreSQL Backup, Off-Host Storage & Declarative DR Restore DevOps / Business Continuity Scheduled out-of-process pg_dump -Fc + age/GPG encryption to off-host storage (R2/PBS/NFS), pre-migration snapshot hook, declarative Ansible restore (highclere_tickets_restore_backup), and automated Staging restore drills. Priority 4
Component 5 Staff Back-Office (/admin) Office & Gate Staff HTTPS Edge Bastion routing with invite-only Staff Member RBAC & Magic Link/OTP auth, order lookup, date transfers, ticket voiding, and direct Gate List exports. Retires the MariaDB customer/order ETL sync at cutover. Priority 5
Component 6 Platform Observability, Alerting & Management Reporting Management, Finance & DevOps Deep /api/health & internal Prometheus /api/metrics, declarative Grafana/Alertmanager capability in debian-ansible, and live RBAC Management Reports (/admin/reports) delivered via daily Magic Link notifications. Priority 6
★ Public Cutover & Legacy Containment Line Zero-Downtime Public Cutover (After Component 6): Switch highclerecastleshop.co.uk DNS/Bastion routing to hs-ct-apps-p-01, retire highclere-etl-sync.timer, sever public ingress to legacy PHP 7.4, and isolate legacy /admin behind NetBird VPN strictly for coach_* bookings until Component 7 completes.
Component 7 Coach Visits Portal & Coach Admin Coach Companies & Group Staff Full vertical slice cutover of coach_* tables to PostgreSQL. Modernised Coach Company login, per-season policy acceptance, slot requests, staff approval/rejection UI, deposit/balance payments, corrected cancellation penalty logic, and final legacy PHP 7.4 decommission. Priority 7
Component 8 Gift Shop Merchandise & Lodge Bookings Shoppers & Overnight Guests Physical gift shop catalogue/shipping calculation and London/Grotto Lodge reservation calendars (lowest revenue share, safely deferred until core ticketing and back-office are complete). Deferred