Troubleshooting & Performance
Diagnosing test failures
Use this workflow when a test fails unexpectedly:
- Isolate — is it test-specific or location-specific? Compare the same test from other
locations (HTTP dashboard → "Performance by location" + check history). If only one location
fails, the issue is the network path.
- Time-based pattern — correlate failures with peak traffic or deployment windows (check
the audit trail + deployment logs).
- Dependencies — inspect the check details:
details.dns_ms — slow DNS (> 50ms spikes → resolver issues)details.ssl_ms — handshake problemsdetails.connect_ms — a connect timeout means TCP never completes (target
overloaded, firewall, or LB saturation)
- External evidence — from the checker node:
curl -v -m 10 https://<target>/health
ping, traceroute during the window. - Recent changes — compare with the audit trail and deploys.
Likely root causes (ranked)
| # | Hypothesis | Evidence to gather |
|---|
| 1 | Target overload at peak (pool exhaustion / LB queue) | connect timeout during the window only |
| 2 | Network path saturation | traceroute / packet loss; other targets in the same region |
| 3 | Rate limiting / WAF blocking the checker IP | HTTP 429/403 in other tests |
| 4 | Monitoring misconfiguration (timeout too tight) | P95 below timeout outside peak but rising |
Recommended fixes
- Raise
config.timeout if the default is cutting off slow-but-successful responses at peak. - Add a "retry then alert" pattern — alert only after two consecutive failures.
- Add more locations to distinguish target vs network problems.
- Monitor the network path (e.g. an HTTP test against the CDN edge).
- Implement a circuit breaker on failing targets: pause checks for a few minutes after repeated failures
(approximate by toggling
is_active via a workflow).
Performance optimization
Baseline example: API P95 ~890ms, check execution ~4.5s avg, DB queries ~50ms. Goals: P95 < 500ms, check
exec < 2s, DB query < 100ms at 10× throughput.
Quick wins
| Change | Impact |
|---|
Connection pooling (pool_pre_ping=True, pool_recycle=3600) | Fewer stale connections |
Indexes on test_checks.checked_at, tests.type, tests.target, alerts/events/audit.created_at | Faster dashboard/history queries |
| Bounded reads (checks ≤ 50, alerts/events/audit ≤ 200) | No unbounded result sets |
| Retention cleanup (daily job) | Biggest DB win — tables stop growing unboundedly |
Caching strategy
| Data | TTL |
|---|
| Dashboard stats | 60s |
| Test configurations | 5 min |
| Teams & users | 15 min |
| Static assets (SPA build) | 1 day (CDN) |
| API responses | Vary by user/permissions — do not cache role-dependent data blindly |
Implementation order
- Indexes + retention (biggest, cheapest).
- Concurrent checker execution (parallel independent tests).
- Redis caching for dashboard + configs.
- Response compression (gzip) + pagination.
- Partitioning at scale; load test to verify.
