Operations Runbook

CE Pro now ships with a more explicit operating model for queue health, structured logs, and restore readiness.

Mobile EAS Release Workflow

GitHub discovers the mobile release workflow at .github/workflows/eas-build.yml in the repository root. The workflow keeps mobile as the command working directory and uses mobile/package-lock.json for dependency caching.

A push that changes mobile/** on main or develop runs the mobile TypeScript and test gates only. It does not start an EAS build, publish an OTA, or submit an app to a store.

Release operations use Run workflow and default to no release action:

• Set run_build to start a native build for the selected profile and

platform.

• Set publish_update to publish a development or preview OTA. This workflow

deliberately rejects production OTA publishing.

• Set submit only with run_build and the production profile. Submission

waits for a successful native build and the protected production GitHub environment.

The workflow rejects submission without a build, submission from a non-production profile, a production OTA request, and a request that combines a native build with an OTA. These gates do not provision signing or store credentials; confirm the required EAS, Apple, and Google credentials before an authorized release run.

EAS resolves a dynamic Expo config twice. Its first local pass intentionally disables dotenv and runs before the named remote environment is downloaded. Production Maps and Firebase inputs are therefore enforced on the remote EAS builder, after protected values are mounted. The builder stops before native compilation if the Android Maps key or GOOGLE_SERVICES_JSON is missing; local release builds separately verify that the generated Android manifest contains the protected Maps metadata.

Mobile dependencies include the Babel runtime explicitly, and the Jest resolver pins transformed shared modules to that mobile installation. This is required because the EAS validation job installs mobile/package-lock.json in isolation instead of relying on a developer machine's repository-level node_modules directory.

Structured Logs

High-value API and worker paths now emit structured JSON logs instead of ad hoc text-only console lines.

The most useful event families are:

• background_job_cron.*
• background_job_batch.*
• background_job.retrying
• background_job.dead_letter
• admin_email.*
• campaign_batch.*
• recovery_email.*
• health_check.degraded
• service_reliability.probed
• service_reliability.capacity_probe_failed
• service_reliability.alert_failed

These logs are designed to help you answer:

• Did the worker run?
• Did it claim work?
• Is one job type failing repeatedly?
• Are failures retrying or dead-lettering?

Queue Monitoring

CE Pro still exposes the public GET /api/health and HEAD /api/health checks for uptime probes.

Internal monitors can also call GET /api/health with Authorization: Bearer $CRON_SECRET to receive a queue snapshot in the response body.

That queue snapshot includes:

• pending
• retrying
• processing
• dead_letter
• oldest_ready_at
• oldest_ready_age_seconds

The project-level monitor at /api/cron/monitor-service-health runs every minute with the cron bearer secret. It stores a sanitized 30-day probe history in service_health_probes and the current circuit state in service_reliability_state. It measures:

• database probe latency
• active and maximum connections
• database size
• transactions older than five minutes

Three consecutive degraded probes open the scheduled-work circuit for at least ten minutes. Two consecutive healthy probes clear it. Transition email alerts go to MAINTENANCE_ALERT_EMAIL, falling back to SUPER_ADMIN_EMAIL.

Recommended Alerts

Use these as the baseline alerts:

• app health degraded for more than 2 minutes
• queue backlog stays above normal for 10 minutes
• oldest ready job age grows beyond 5 minutes
• dead-letter count becomes non-zero
• background worker cron starts failing or stops running
• webhook delivery failures spike
• admin auth failures spike
• database connections reach 80% of max_connections
• any database transaction remains open longer than 5 minutes
• the reliability circuit opens or the health monitor stops running

Routing Timeout Response

When an estimate, admin page, or mobile workflow is slow and Vercel reports MIDDLEWARE_INVOCATION_TIMEOUT or 504:

1. Check /api/health; a quick 503 confirms the dependency failure without

tying up another middleware invocation.

1. Check the latest service_reliability.probed event and

service_reliability_state.circuit_open_until.

1. Use project-level Supabase logs and capacity metrics. Do not rely only on

the provider-wide status page.

1. If the automatic circuit has not opened, set

DISABLE_SCHEDULED_WORK=true and redeploy before doing deeper diagnostics.

1. Avoid repeated super-admin/dashboard refreshes while the database is under

pressure.

1. Once Supabase is stable, wait for two healthy probes, then verify health,

sign-in, one web estimate, and one mobile estimate detail request.

1. Remove a manual scheduler pause only after those checks pass. The automatic

circuit recovers on its own.

Protected-route middleware performs organization selection, onboarding, and role-permission resolution in one database call. Its Supabase calls retain an eight-second upper bound so a dependency incident fails closed instead of turning into a five-minute Vercel routing timeout.

Backup And Restore

The production app runs on Supabase-backed Postgres, so the expected backup source of truth is the managed Supabase production project.

That expectation still needs to be verified in the live Supabase dashboard. Until that verification is done, treat restore readiness as a tracked operations task, not an assumption.

Minimum standard:

1. Automated backups enabled.
2. Retention window documented.
3. Restore owner named.
4. Quarterly restore drill recorded.

Internal Runbooks

The repo also carries internal runbooks for the operating team:

• docs/incident-monitoring-runbook.md
• docs/backup-and-restore-runbook.md

Use those documents for the concrete response checklist, ownership fields, and restore-drill procedure.

Stress Validation

Before bigger launches, pricing changes, marketing pushes, or other traffic-shaping releases, run the repeatable stress drill documented in Stress Testing.

That drill exists so queue, health, and admin hot-path regressions show up before customers do.

If the triggering event is a hosted Supabase project move or a region migration, run the cutover checklist and post-switch validation from Supabase Region Migration before you call the environment healthy.