Operations & runbooks.

Quality gates

Four gates sit between a commit and a deploy:

Testing

cd backend
ulimit -n 8192 && ./gradlew test                 # all tests
./gradlew test --tests "com.cplk.api.architecture.ArchitectureTest"
ulimit -n 8192 && ./gradlew test jacocoTestReport
./gradlew check                                  # tests + 80% gate

cd e2e
npx playwright test                  # all
npx playwright test --project agent  # one role project
CPLK_E2E_BASE_URL=https://dev.cplk.org npx playwright test

Observability

Suggested SLOs

Known gotchas

Real failure modes we've already hit — keep these top of mind:

-- on the affected env's DB
UPDATE flyway_schema_history SET checksum = <new_checksum>
WHERE version = '<version>';

Runbooks

RB-01 · API 5xx spike

1. Check journalctl -u cplk-api -f for stack traces; group by exception type.
2. Hit /api/actuator/health — DB / mail / disk components.
3. Was there a deploy in the last hour? If yes, rollback via symlink swap (Deployment → Rollback).
4. If DB is slow: check pg_stat_activity for long-running queries; verify Hikari pool isn't exhausted (HIKARI_MAX_POOL_SIZE).
5. If Keycloak unreachable: validate JWKS reachable from the API VPS; verify AUTH_JWK_SET_URI.

RB-02 · Suspected cross-tenant access

1. Identify the offending request id from the audit log.
2. Inspect the call stack — does it pass through a method tagged @BypassTenantFilter? If yes, was it the right role?
3. Suspend the suspect user in Keycloak; revoke their sessions.
4. Diff old_values / new_values in audit_logs to scope the breach.
5. Write a regression test in TenantIsolationIntegrationTest before patching.

RB-03 · PayHere webhook outage

1. Inspect Nginx access logs for POST /payments/notify — are calls arriving?
2. Verify PayHere config: PAYHERE_BASE_URL, _MERCHANT_ID, _NOTIFY_URL.
3. If calls arrive but fail signature: rotate PAYHERE_MERCHANT_SECRET with PayHere support; redeploy.
4. Queue manual webhook replay via PayHere dashboard for affected orders.
5. Confirm idempotency: pulling on order_id should show a single SUCCESS row even after multiple deliveries.

RB-04 · Recurring charges not firing

1. Confirm scheduler ran: SELECT * FROM shedlock WHERE name = 'rechargeSubscriptions'.
2. If locked_until is in the future from a crashed previous run, clear it manually.
3. Check PayHereOAuthService for repeated 401s in logs — the bearer cache may be stuck.
4. Look at FinancialAuditLog(event_type='PAYMENT_FAILED') for the agency to identify the failure mode (card declined, token revoked, etc.).

RB-05 · R2 upload failures

1. Verify CF_R2_ACCESS_KEY not rotated; bucket exists; CORS allows the web origin.
2. Fall back to LocalStorageService by setting app.storage.r2.enabled=false temporarily.
3. Re-upload failed images from property_images rows whose base_key points to missing objects.

RB-06 · Email not delivered

1. EMAIL_ENABLED true? NoOpEmailService silently no-ops if false.
2. SMTP creds valid? Test with swaks --to me --server $SMTP_HOST:$SMTP_PORT --auth LOGIN --auth-user $SMTP_USERNAME --auth-password $SMTP_PASSWORD.
3. Check Notification DLQ: DlqNotificationHandler records the original message + cause.

RB-07 · Flyway migration fails on deploy

1. Do not edit the failed migration on disk.
2. Identify the cause — locked rows? wrong assumed state?
3. If checksum mismatch (someone edited an applied file): see Known gotchas.
4. If actual schema error: write a fix-forward migration with a higher version; redeploy.

RB-08 · Database backup / restore

make db-backup env=prod   # pg_dump → encrypted artefact
make db-restore env=prod FILE=cplk-prod-2026-05-13.dump

Backups are not encrypted at rest on the DB VPS by default — copy them off the box and keep them in R2 / object storage with restricted access.

Disaster recovery

Order of restoration after total VPS loss:

1. Provision new DB VPS, restore from latest pg_dump.
2. Provision new API VPS, install Java 21, Nginx, systemd unit.
3. Place /etc/cplk/api.env from secrets store; make api-deploy env=prod.
4. Re-point DNS for api.cplk.org to the new API VPS.
5. Re-deploy frontend (Cloudflare Pages is independent — should still serve).
6. Verify Keycloak — auth.cplk.org may be its own host; if not, restore that too.
7. Smoke test: log in, list properties, submit inquiry, run a sandbox PayHere checkout, verify webhook reaches API.

Operational checklist (weekly)

• Review FinancialAuditLog totals vs PayHere settlement report.
• Sample 5 audit-log entries per agency to spot unusual patterns.
• Confirm scheduled jobs ran: SELECT name, locked_at FROM shedlock.
• Confirm Nginx error log clean (no 502/504 spikes).
• Disk usage on both VPSes < 70% (logs + heap dumps grow).
• Rotate SMTP_PASSWORD and PAYHERE_MERCHANT_SECRET on the documented cadence.