Operations & Scaling
How xlnszia scales for enterprise workloads and keeps 99.99% uptime — for operators running the stack at significant scale.
Scaling model
The reference targets: 50,000 tests, 100,000 checks/min, 10,000 concurrent users, 10TB+ metrics, 99.99% uptime.
architecture.txt text
Global LB / CDN
├── API servers (multiple uvicorn workers)
├── WebSocket servers (Redis pub/sub bridge)
├── Checker nodes (consume jobs from Kafka/RabbitMQ)
├── Database (primary + read replicas, TimescaleDB)
├── Redis (cache + WS state)
└── S3 / GCS (cold archives)Key changes by component
| Component | At scale |
|---|---|
| Database | Write master + read replicas; test_checks as a partitioned TimescaleDB hypertable with native retention; connection pooling. |
| Checker service | Distributed nodes consuming a job queue (Kafka/RabbitMQ); idempotent by (test_id, scheduled_at); priority queues (critical > standard > batch). |
| API server | Multiple workers behind a load balancer; Redis caching (dashboard 60s, configs 5m, teams 15m); rate limiting per user/IP. |
| WebSocket | Horizontal scale-out with a Redis pub/sub bridge; connection state in Redis for sticky failover. |
| Storage lifecycle | Hot (30d) → warm (compressed, 90d) → cold (S3/GCS). Backups: daily full + transaction logs (RPO 5 min). |
| Monitoring | System, application and business metrics via the Datadog / OpenTelemetry integrations. |
High availability (99.99%)
Targets: ≤ 52 min/yr downtime, RPO 0, RTO 5 min, no single points of failure.
| Component | Redundancy | Detection / action | Recovery |
|---|---|---|---|
| Database | Primary + synchronous secondary + read replicas | Health-check timeout → promote secondary | < 60s |
| Checker | Active-active nodes | Heartbeat timeout → rebalance from queue (idempotent) | < 30s |
| API servers | LB pool, multiple instances | LB health check (/api/health) → drain + rolling restart | < 10s |
| WebSocket | Sticky sessions + Redis state | Connection-count drop → clients reconnect | < 30s |
| Networking | Global LB + DNS failover (TTL < 60s) | Cross-region health checks → route to healthy region | < 60s |
What makes HA work
- Idempotent checks — alerts are deduplicated by
(test_id, rule_id), so a check run twice never double-fires. - Stateless API — JWT / PAT auth only; no server-side session to replicate.
- Graceful shutdown — the checker, WebSocket broadcaster and retention loops cancel cleanly on shutdown, so rolling restarts are zero-downtime.
- Health endpoint —
GET /api/healthfor load-balancer checks. - WS reconnection — the SPA reconnects with exponential backoff, so WebSocket failover is transparent to users.
Failover runbooks
- DB primary failure: verify secondary catches up → promote → repoint via
DATABASE_URL→ verify/api/health→ record in audit. - Checker node failure: the queue re-dispatches in-flight jobs after heartbeat timeout; replace the node.
- API server failure: the LB detects and drains; keep 3+ nodes.
- WS server failure: clients reconnect with backoff to surviving nodes; Redis holds subscription state.
- Network partition: the global LB routes to the healthy region.
Recommended deployment
2+ API servers, 2+ WebSocket servers, 3+ checker nodes, and a DB primary + synchronous secondary + 2 read replicas per region. Deploy one node at a time; schema changes use the idempotent migration path so old and new nodes coexist.