Platform Reliability

CE Pro now includes a shared baseline for session refresh, admin API protection, health checks, durable automation recovery, and app-level crash recovery.

These platform controls are not a replacement for route-level auth, validation, or product-specific permissions. They exist so the app behaves more predictably as traffic grows.

Mobile Multi-Tenant API Baseline

The authenticated mobile API now has one shared tenant-resolution path. It:

• validates bearer tokens with Supabase Auth
• validates X-Organization-Id as a UUID
• confirms an active, non-archived membership for that exact organization
• rejects inactive organizations
• resolves role permissions and feature flags once per request
• emits X-Request-ID, Server-Timing, private/no-store cache policy, and

authorization/workspace Vary metadata from the shared response helper

The client only accepts relative mobile API paths on the configured CleanEstimate origin, which prevents an authorization token from being sent to an arbitrary destination. Calls have a 12-second deadline. GET and HEAD calls can retry transient 408, 425, 429, and gateway/server failures with bounded backoff; mutation calls default to zero retries. Concurrent identical GETs are deduplicated by selected organization and path.

The server also deduplicates concurrent workspace-context resolution for the same verified user and exact selected organization. This is deliberately in-flight-only: the entry is removed as soon as the shared lookup settles, and failed lookups are never retained. Concurrent requests with the exact same bearer token and X-Organization-Id may share the pending token validation and workspace resolution; a different token or organization always resolves independently. Authorization for one tenant can therefore never satisfy a request for another tenant, and membership or permission changes are not hidden behind a time-based authorization cache.

Explicit tenant selection fails closed. A malformed X-Organization-Id, or a valid organization id for which the authenticated user has no active membership, is rejected rather than falling back to another workspace. Only a stale server-owned active_org_id account-metadata hint can recover to the user's first active membership when the device did not explicitly choose an organization.

On native iOS and Android, Supabase token refresh starts while the app is active and stops while it is backgrounded. When an API response still returns 401, the client checks for an already refreshed session or performs one shared token refresh, then retries a safe read once with the same organization and request ID. Concurrent rejected requests share that refresh promise only when they belong to the same account. Mutations can refresh the session for the next explicit action, but they are not replayed automatically after authentication, network, validation, or server failures.

The registered mobile contract is available from:

GET /api/openapi/mobile.json

Contract tests compare that registry with every implemented /api/mobile/**/route.ts operation and verify that every mobile route resolves one of the approved authenticated contexts. The registry is the handoff point for a future API gateway or mapper; it does not replace server-side authorization.

The Vercel source-ignore rule is anchored to the repository's root /mobile workspace. A packaging test prevents an unanchored mobile rule from accidentally excluding the Next.js routes under src/app/api/mobile.

Mobile Dashboard and Search fan independent tenant-scoped reads out with Promise.all, keep explicit result limits, and preserve representative assignment filters. Organization-first partial indexes cover active membership resolution plus the clients, leads, estimates, appointments, and jobs recency paths. This keeps common queries proportional to the selected workspace rather than total platform row count.

The native client also connects its query lifecycle to app focus and network state. Backgrounded screens pause periodic reads, disconnected devices stop starting work that cannot complete, and the active screen resumes or refreshes through the shared query lifecycle when the app becomes usable again. Repeated polls are scoped to the focused workspace screen instead of every mounted tab. Non-critical native startup work, including voice and notification setup, waits until the first app interactions finish so it does not compete with the initial workspace render.

Mobile startup now shares one tenant-scoped bootstrap query across the root access gate, feature provider, and Home screen. /api/mobile/bootstrap resolves the bearer principal once, derives access and allowlisted feature flags from that context, and loads profile, memberships, and permitted Home data in parallel. A principal without dashboard.view receives no Home payload. The query key includes the authenticated user, selected organization, and dashboard period, so a result cannot cross an account or workspace boundary.

Mobile bearer sessions use Supabase's asymmetric signing keys. The server caches the public JWKS and verifies each explicit access token locally before tenant resolution, eliminating the remote Auth request that previously preceded every mobile API route. Verification errors remain fail closed; legacy sessions or a missing JWKS use Supabase's network-backed verification path rather than trusting unverified claims. Expired or invalid local claims are classified as authentication rejections, not dependency outages, so the native single-flight refresh path receives the expected 401 response and retries each safe read at most once.

The mobile Leads reader requests up to 500 authorized leads per bounded page, and Pipeline requests up to 1,000 authorized estimates per page. Both continue until the final short page, so the larger first page reduces round trips without truncating larger workspaces. The Clients list returns its bounded result count without running an unused exact-count aggregate.

The native app applies its encrypted, previously verified workspace and access cache before the live refresh finishes. Same-account, same-workspace Home data can remain visible in the in-memory query cache during a refresh; customer and Home payloads are not added to persistent device storage by this optimization. Concurrent bootstrap consumers reuse one request, and repeated mount or focus events use the same five-minute freshness window instead of starting parallel copies of the startup reads.

Mobile transport logs now record request duration, method, status, attempt count, safe route family, request ID, authentication retry state, and parsed Server-Timing. They do not log bearer tokens, query values, customer data, or organization names. The bootstrap route records separate authorization and data durations plus result counts, also without PII.

Production Telemetry Baseline

Phase 0 scaling work now starts with production telemetry, not synthetic load tests.

Before CE Pro changes database access patterns, adds projections, provisions replicas, or raises infrastructure spend, the team must capture a production baseline that shows:

• the top 10 slow app routes by p95 and p99
• the top 20 slow SQL statements from pg_stat_statements
• the current Supabase connection snapshot, including total, active, idle, idle-in-transaction, waiting, and max_connections

The app collects sampled browser navigation and same-origin API timing events in production. Sampling defaults to 5% on public/non-admin routes, while authenticated admin pages default to 20% so workhorse-route regressions show up faster. Override non-admin sampling with NEXT_PUBLIC_ROUTE_TELEMETRY_SAMPLE_RATE, override admin sampling with NEXT_PUBLIC_ROUTE_TELEMETRY_ADMIN_SAMPLE_RATE, and disable client-side collection with NEXT_PUBLIC_ROUTE_TELEMETRY_ENABLED=false. The ingestion endpoint can also be shut off server-side with ROUTE_TELEMETRY_INGEST_DISABLED=true. Telemetry ingest uses the in-memory limiter so collecting route timings does not spend another database RPC before writing the actual sample.

Telemetry writes land in route_performance_events. The export command reads service-role-only RPCs for route timing, pg_stat_statements, and connection usage:

npm run telemetry:production -- --days=7 --out=reports/production-telemetry.md

The Phase 0 stop condition is strict: if production telemetry shows the current SLOs are already met at the current measured load, halt Phases 1-6 and re-evaluate quarterly instead of making speculative architecture changes.

The business-load target is also a hard prerequisite before optimization work beyond telemetry: the business owner must write down N simultaneous authenticated active sessions by date D, justified by the launch, customer, or usage reason. Without that number, scale work has no anchor.

Cost gates stay explicit. Any phase that would push monthly infrastructure spend above $250 needs Clean Estimate Pro business-owner approval before it starts. A higher ceiling is only unlocked when current MRR supports it: $500/month after $10k MRR, $1,000/month after $25k MRR, or a written launch-risk exception from the business owner.

Auth and Worker Backpressure

When Supabase Auth or PostgREST starts returning gateway timeouts, the sign-in screen now fails visibly instead of leaving the user stuck on Signing in.... Password sign-in waits up to 15 seconds and then shows a retryable auth-service message.

Version boundary for this mixed reliability section: The sign-in, database-circuit, fixed notification/background-job, follow-up, drip, and proposal-expiration controls below apply where their current V2 routes are registered. Every reference here to generic automation-engine routes, workflow definitions, enrollments, workflow_runs, scheduled automation queues, or business-event workflow dispatch describes the retained Clean Estimate V1 engine. Current V2 does not register that engine or its storage.

The scheduled worker routes also include incident controls so database pressure can be reduced without changing user-facing app code:

• A project-level reliability probe runs every minute and records database

response time, size, active/max connections, and long transactions. Three consecutive degraded probes open a ten-minute circuit for high-fanout scheduled work. Two consecutive healthy probes clear it.

• While the circuit is open, scheduled automation, waiting automation,

background jobs, follow-ups, drip campaigns, and webhook deliveries return a successful disabled response without claiming work. The circuit does not deactivate workflow definitions, cancel enrollments, or delete queued work. If the guard itself cannot reach the database within two seconds, those workers fail closed so user traffic gets the available capacity.

• DISABLE_CRON_JOBS=true pauses the full scheduled-work layer after each route authenticates and before it opens a Supabase connection.
• DISABLE_SCHEDULED_WORK=true is an equivalent global scheduler pause switch for incidents where the word "cron" is too narrow.
• DISABLE_BACKGROUND_JOB_CRON=true pauses only /api/cron/process-background-jobs.
• DISABLE_WEBHOOK_DELIVERY_CRON=true pauses only /api/cron/process-webhook-deliveries.
• DISABLE_FOLLOW_UP_CRON=true pauses only /api/cron/process-follow-ups.
• DISABLE_DRIP_CAMPAIGN_CRON=true pauses only /api/cron/process-drip-campaigns.
• DISABLE_EXPIRE_PROPOSALS_CRON=true pauses only /api/cron/expire-proposals.
• DISABLE_AUTOMATION_WAITING_CRON=true pauses only /api/automations/engine/process-waiting.
• DISABLE_AUTOMATION_SCHEDULED_TRIGGER_CRON=true pauses only /api/automations/engine/process-scheduled-triggers.
• DISABLE_AUTOMATION_BUSINESS_EVENT_CRON=true pauses only /api/automations/engine/process-business-events.
• DISABLE_FRANCHISE_SUMMARY_CRON=true pauses only /api/cron/refresh-franchise-summary.
• BACKGROUND_JOB_CRON_BATCH_SIZE and BACKGROUND_JOB_CRON_CONCURRENCY can raise or lower the background worker from its conservative defaults of 5 jobs and 1 concurrent dispatch.
• WEBHOOK_DELIVERY_CRON_BATCH_SIZE can raise or lower webhook delivery claiming from its conservative default of 10 deliveries.

The switches have intentionally different scopes. The automatic database circuit and either global switch pause the high-fanout workers listed above. DISABLE_AUTOMATION_SCHEDULED_TRIGGER_CRON=true pauses only the worker that materializes and drains appointment, estimate, retained V1 invoice, and contract timing events. It does not by itself stop manual enrollment, an immediate lead_created dispatch, the independent business-event worker, the waiting-run worker, or the legacy follow-up worker. The durable recovery job for a new lead uses the background-job worker, so it waits while that broader worker is paused even if the inline speed-to-lead attempt already succeeded.

Reliability alerts use a durable incident identity for open and recovered transitions. Provider-level idempotency suppresses duplicate transition emails across concurrent or cold-started app instances. During a hard database outage, when the database cannot record the failing probe, the fallback warning uses a UTC-day idempotency bucket. A healthy probe after a monitoring gap of at least ten minutes can close and report the incident even when the failed probes could not be persisted. These controls reduce duplicate alerting; they do not hide a continuing multi-day outage.

The public ingress rate-limit RPC is also lock-bounded. If one hot key is already being updated, the limiter fails closed quickly instead of holding PostgREST workers on a row lock until the platform times out the request.

Scheduled automation no longer reloads every historically overdue candidate in the application on each minute tick. A bounded database materializer records previously unseen timing events, and the consumer drains a small sequential batch so recovery does not fan database or provider traffic back out at once.

Workflow-run idempotency now uses the indexed workflow_runs.dedupe_key column. A partial unique index on (workflow_id, dedupe_key) makes duplicate prevention atomic and removes the previous 400-day JSON history scan from the worker hot path.

Durable Scheduled Automation Recovery

V1 automation reliability reference: Current V2 has fixed notification jobs and product outbox facts, but it does not register the generic workflow engine routes, workflow definitions, enrollment/run storage, or inbound automation webhook described through the business-outbox cutover sections below.

The invoice due/overdue automation references in this section describe the retained V1 automation engine. The current V2 invoice workspace has durable reviewed email/SMS delivery and payment processing, but it does not yet enroll V2 invoices into these generic scheduled workflows.

Time-based automation uses scheduled_automation_events as a durable queue and one scheduled_automation_watermarks row per workflow as its persistent scan cursor. There is no global cursor whose progress can skip another workflow's window. The materializer currently covers:

• upcoming lead, estimate, and job appointments
• missed lead appointments
• estimates approaching expiration or reaching expiration
• V1 invoices approaching their due date or becoming overdue
• upcoming contract renewals

The materializer scans a maximum six-hour time window at a time and writes at most 250 previously unseen events per call. It advances the watermark only after the current window no longer fills that bound. This means a deployment, circuit-breaker pause, or provider outage leaves a resumable gap rather than a lost in-memory timer. The recovery worker intentionally drains only four events sequentially per invocation so catch-up traffic does not immediately recreate the database or messaging-provider pressure that opened the incident.

The materializer selects only active workflows and advances only the selected workflow's row. Pausing a workflow leaves its cursor and queued events intact; reactivation resumes that workflow from its own saved position. Claiming also requires the workflow to be active. Archiving the workflow or changing its trigger cancels pending queue rows, while merely pausing it does not.

Each event has a stable (workflow_id, event_key) uniqueness boundary. Workers claim ready rows with SKIP LOCKED, own the claim with a token, and either complete it or release it with exponential backoff. A stale claim from a terminated worker becomes retryable; after eight attempts it becomes dead instead of disappearing. Before claim, the database cancels events whose workflow was archived, whose trigger changed, or whose source is no longer eligible, such as a paid invoice, accepted estimate, cancelled job, or rescheduled appointment.

Estimate expiry uses the same terminal-evidence rule during candidate materialization, the bounded status = 'expired' lifecycle update, and claim-time validation. A legacy row with accepted_at, signed_at, an acceptance fingerprint or snapshot, or an accepted maintenance agreement is not mutable even if its status still reads draft, sent, or viewed. Skipping it at all three gates prevents both a transaction-wide immutability error and an incorrect expiration message.

Due customer-contact events older than the 24-hour automatic replay window are dead-lettered and alerted rather than sent unexpectedly after a long outage. Future reminders and recent catch-up events remain eligible. This age boundary does not erase the event; it keeps the reason and source available for review.

The consumer reconciles any workflow run already created for the event before settling the queue row. A running workflow with no executing step can resume. An executing provider step is not blindly replayed because the provider may have accepted the message before its acknowledgement was lost. Stable queue, run, and enrollment keys materially reduce duplicate delivery, but the system does not claim that an external email, SMS, voice, or webhook provider can offer perfect exactly-once delivery.

Every scheduled worker completion emits queue counts for materialized, claimed, completed, retried, and dead events plus pending, processing, oldest pending, and maximum-attempt metrics. Use those structured logs with the dead-letter count during recovery; a successful HTTP response alone does not prove the backlog is empty.

Durable Record-Event Automation

Invoice and payment workflow triggers in this section are retained V1 automation behavior. Current V2 writes the first invoice.paid product outbox event exactly once when the ledger first reaches a zero collectible balance, but does not yet enroll it or the broader retained invoice/payment trigger set into configurable Workflow Runs.

In V1, supported record changes are admitted transactionally to the same durable event storage with queue_kind = 'business'. V1 database triggers cover estimate, job, lead-status, client, review, inbound message, and phone occurrences. Each active workflow receives a stable occurrence key, while claim-time validation prevents an obsolete or test-environment source from becoming live customer contact.

/api/automations/engine/process-business-events runs independently every minute with its own lease and conservative batches. It never depends on the time-scheduler cursor or its scoped pause switch, so a backlog of reminders cannot starve a newly accepted estimate, completed job, payment event, or customer reply. Both lanes retain bounded terminal history and expose separate queue-age and dead-letter metrics. A pending or retrying business event older than 24 hours is dead-lettered and alerted rather than automatically starting customer-facing work after a long outage.

V1 external automation webhooks use admit_inbound_automation_webhook to persist a bounded live event and enqueue matching workflows atomically. The endpoint requires a live webhooks:write API key plus a stable Idempotency-Key header or event_id; test keys cannot enqueue live workflows. JSON payloads are limited to 256 KiB in UTF-8 and again by database size. Reusing an idempotency key with different content returns a conflict. Declared routing entities must belong to the authenticated organization, and provider executors repeat that organization check before resolving a recipient. Payload history is retained only for the documented recovery window and cleaned in bounded work.

The generic recurring_schedule trigger remains outside this durable timing materializer. Both builders omit it, workflow creation rejects it, and activation rejects an older draft that still has that trigger type. Supported date and record-event triggers remain available. This boundary prevents an unsupported cron-and-audience workflow from appearing active without a worker that can execute it.

First Business-Outbox Cutover

The first production rollout is a controlled shadow-admission cutover because old app instances use legacy run keys while the new database triggers use database-owned occurrence keys. Follow this order exactly:

1. Set DISABLE_AUTOMATION_BUSINESS_EVENT_CRON=true in production before

applying the outbox migration. Keep the time-based scheduler switch independent.

1. Apply the database migrations. Capture triggers are active immediately and

begin recording business events, while the paused business worker prevents delivery during the compatibility window. Record the migration's shadow_started_at value from automation_business_outbox_cutover.

1. Deploy the compatible app and record the exact UTC time when that deployment

is promoted. Wait at least five minutes for old serverless instances and in-flight requests to drain.

1. As service_role, call

reconcile_business_automation_cutover(p_promoted_at, p_batch_size) with that exact promotion timestamp. Repeat bounded calls with the same timestamp until remaining_to_review is 0. The timestamp becomes immutable after the first call.

1. Inspect the returned completed, quarantined, and unmatched counts. The

reconciler pairs a legacy-key run to at most one post-event queue row, completes an already settled match, dead-letters an ambiguous active or failed match, and leaves an unmatched event pending for durable delivery.

1. Review dead letters and queue-age metrics, then remove

DISABLE_AUTOMATION_BUSINESS_EVENT_CRON only when the cutover backlog is understood. There is no separate activation migration.

Do not unpause the worker before reconciliation and do not substitute a new promotion time on later batches. Either action can turn the shadow window into duplicate customer contact or suppress a legitimate unmatched event.

Durable Lead-Created Safety Net

Eligible connector, web-form, and server-API lead intake marks the new lead for speed-to-lead automation. During the lead transaction, the database snapshots the exact active lead_created workflow IDs or the exact active, deliverable legacy speed-to-lead sequence. An AFTER INSERT trigger then writes a deduplicated internal background job with that immutable route in the same transaction as the lead. If the queue insert cannot commit, an admitted automation-enabled lead insert does not commit either. A configuration read failure also fails closed. If no deliverable automation route exists, CE Pro still captures the lead but clears its automation marker instead of claiming that outreach was queued.

The request and recovery worker both use the persisted route, so activating or deactivating a workflow after intake cannot switch a lead from a workflow to a legacy sequence or contact it twice. Healthy traffic keeps its low-latency inline attempt, while the background job reconciles the same workflow run or legacy enrollment after an app crash, timeout, or ambiguous response. Legacy mode now uses the configured sequence as its sole contact path; it no longer adds a separate hard-coded immediate SMS or auto-dial outside that sequence. Manual mobile lead creation sets the automation marker to false on both its normal API path and direct database fallback, so an outage does not unexpectedly turn a manual create into automatic customer outreach.

The replay endpoint reloads the authoritative lead, checks that automation is still enabled, and uses lead_created:<lead_id> as the workflow dedupe key. It reconciles only the workflow IDs or legacy sequence saved at admission. A partial unique index on active or paused (sequence_id, entity_type, entity_id) enrollments closes the race between inline work and the queue consumer.

Lead automation recovery jobs allow ten attempts and have a 24-hour safe replay window. The background-job claimant reclaims expired processing leases only for explicitly replay-safe internal effects, and it moves a stale final attempt to dead_letter for monitoring instead of leaving it permanently stuck. Generic messaging or campaign work stranded after a provider may have accepted the request is quarantined rather than automatically replayed. As with the scheduled queue, an already executing workflow provider step is quarantined rather than blindly replayed when delivery is ambiguous.

Every workflow step first writes a durable execution reservation. The engine must persist the step outcome, run cursor, enrollment, audit, and terminal run state before its queue item can be acknowledged. Database write failures stop the run and keep recovery visible; they cannot silently turn a failed or unrecorded provider action into a completed automation.

Waiting-Run Lease Recovery

claim_waiting_workflow_runs now accepts a bounded stale-lease window and safe replay age. The production route uses five minutes and 24 hours. It continues to claim due waiting rows, and it can also reclaim a recent running row whose worker lease or inline activity went stale when no step remains in executing. Reclaimed running rows increment retry_count, receive a new lock token, and retain an explicit recovery message.

Automatic replay is denied when provider outcome may be ambiguous. A stale run with any executing step is changed to paused, its lock is cleared, and a run_recovery_quarantined audit row records ambiguous_executing_step. A non-executing stale run older than the 24-hour recovery window is also paused and audited as stale_beyond_recovery_window. Both paths require operator review instead of risking a duplicate or unexpectedly old customer contact.

Automation Dead-Letter Alerting

Transitions into a scheduled- or business-event dead state, a background-job dead_letter state, or a legacy follow-up delivery pause attempt an immediate email to MAINTENANCE_ALERT_EMAIL, falling back to SUPER_ADMIN_EMAIL. The logical record ID is included in the provider idempotency key so concurrent workers or a lost alert response do not intentionally send the same transition email twice. Alert-provider failure does not undo the durable dead-letter or paused state.

The existing daily maintenance run also calls collect_automation_reliability_metrics(). It reports aggregate scheduled backlog age, overdue and stale queue work, recent dead events, failed and stale workflow runs, overdue waiting runs, ambiguous executing steps, recent quarantines, stale and overdue follow-ups, failed delivery attempts, and delivery-paused enrollments. These daily aggregates contain counts and ages, not customer content; the immediate operational email identifies only the durable record and organization needed for investigation. A high-severity daily result is sent through the existing maintenance alert email. Immediate email plus daily detection improves visibility; it is not a guarantee that an email provider or external delivery provider cannot fail.

If Supabase warns that the project is about to deplete or has depleted its Disk IO Budget, set the global scheduler pause first, redeploy so every route has the guard, avoid super-admin/dashboard refreshes, then either wait for the next daily IO budget replenishment or temporarily upgrade compute while the team captures pg_stat_statements and fixes the disk-heavy queries. Compute add-ons require the Supabase organization to be on a paid plan; free-tier projects must be upgraded to a paid Supabase plan before the emergency compute add-on can be applied.

Super-admin pages are dynamic-only so protected platform dashboards never block a Vercel build on live Supabase reads.

Sitewide Page-Load Controls

Navigation throughout the public website, authenticated admin app, and customer surfaces uses intent-based prefetching. Links do not preload every destination merely because a desktop and mobile navigation tree exists in the page. The exact destination is prefetched once when a user points to, focuses, or touches that link, preserving fast follow-through without flooding a new page load with speculative React Server Component requests.

The policy now covers the dense Clients, Leads, Jobs, Invoices, Estimates, and Residential record lists as well as the main authenticated navigation inventories. Record links explicitly disable viewport prefetch and warm the destination only after pointer, keyboard, or touch intent. A short destination dedupe window prevents the desktop row, mobile card, and next-action control from warming the same route repeatedly.

Loading boundaries give route transitions an immediate visual response while the destination's required data is still resolving. Optional tracking is loaded after the interactive page work, so analytics does not compete with the content and controls needed for the first useful render.

Large interactive workspaces should also keep inactive panels out of the initial client bundle. The Sales Workspace streams tenant-scoped quote configuration behind its immediately usable shell, then loads quote, lead, schedule, customer-search, and follow-up code on intent. Customer estimate pages follow the same pattern for proposal, maintenance, interaction, and photo panels. Public estimate requests first authorize the raw estimate record, so a shared-quote redirect can happen before residential offer hydration or optional reads. For legacy estimate rendering, offer hydration runs alongside independent organization, pricing, and design reads. Canonical acceptance reuses the already-authorized estimate and organization result instead of repeating those tenant-scoped reads. Every database read remains constrained to the estimate's organization. Independent holiday-light design backdrops and mockups are also fetched concurrently while preserving their presentation order.

Read-only workspace bootstrap paths use their own organization-scoped cache keys. In particular, opening Sales can read the existing residential catalog and lead-source defaults without running the write-oriented new-tenant bootstrap sequence, and that read-only cache cannot suppress a later bootstrap-enabled setup request.

The authenticated Home dashboard now computes its seven-day and thirty-day residential totals, conversion rate, average value, accepted revenue, trending services, and optional Holiday Lights counts in one organization-scoped SQL summary. A cache miss returns one compact row instead of downloading every matching estimate to the app server for JavaScript aggregation. The existing five-minute workspace cache remains in front of this query, and the summary keeps exact user-facing counts and monetary values.

These rules apply sitewide rather than only to Command Center. When adding a new shared navigation component:

• keep automatic viewport prefetch off for large menus or duplicated

desktop/mobile navigation trees

• use the shared intent-prefetch link for internal destinations
• do not eagerly import optional tracking, support, or visualization code into

the initial route bundle

• keep tenant-specific data behind the authenticated, organization-scoped

server path even if the route shell can render immediately

Admin Alert Backpressure

The admin alert bell is a read path. It should not regenerate managed alerts or write to ai_alerts on every poll. Managed alert generation is reserved for the dashboard alert scope, is throttled by ADMIN_ALERT_SYNC_TTL_MS with a five-minute default, and skips unchanged rows so realtime subscriptions do not create read/write feedback loops.

The admin shell now owns one tenant-scoped live-data feed for alert consumers. Desktop navigation, mobile navigation, and the Command Center alert panel reuse that feed and its realtime refresh path instead of mounting independent polls and subscriptions. The feed key includes the selected organization, and a workspace switch creates a distinct feed rather than reusing another tenant's alerts.

If the admin shell feels slow, check telemetry_route_performance_summary for /api/admin/alerts and telemetry_pg_stat_statements_top for ai_alerts updates. A healthy alert path should show reads and occasional inserts or updates, not continuous update traffic while the user is idle.

Admin Navigation Read Pressure

The admin shell keeps ordinary page navigation on the lightest available read path.

Current controls:

• Protected-route middleware resolves membership selection, franchise access,

onboarding state, and role-permission overrides in one service-role RPC. Together with the bounded Supabase fetch, a typical admin route now needs an auth verification and one organization-context database request instead of several sequential PostgREST reads.

• The sidebar gamification badge is read-only during normal page loads and no longer seeds catalogs, upserts reward rows, recalculates profiles, or records daily-login XP from the shared layout.
• Duplicate desktop and mobile shell consumers share live alert and unread-message state instead of issuing separate initial reads, polling loops, and realtime subscriptions.
• Large navigation trees and the dense Settings destination grid use intent

prefetching so a page load does not trigger a speculative request for every visible admin destination.

• Command Center branches into the signed-in user's role-specific data path before loading office-only metrics. Sales and crew workspaces do not wait for unrelated owner/manager queries.
• Optional shell data is kept off the destination page's critical render path, and route loading boundaries provide immediate feedback while required tenant data resolves.
• Admin API middleware rate limiting uses the in-memory limiter for /api/admin/* traffic so ordinary schedule, client, and dashboard reads do not spend a database RPC before the route handler authenticates and authorizes the request.
• The Clients list avoids an unused exact count on the standalone contact query and narrows client-health work to the visible client ids when possible.
• The Clients page includes account rows, contacts, account choices, and health

context in its initial secured response instead of rendering an empty shell and repeating authentication through startup API calls.

• Lead, estimate, and job detail workspaces include their primary record

payload in the initial secured response. The lead workspace also resolves assignable team members in that same server pass. Refresh and mutation actions retain their API paths, but first render does not repeat the read after hydration.

• The central Price Book includes its catalog and inventory choices in the

initial secured response instead of requiring a second authenticated startup request before items appear.

• The Estimates list API keeps exact visible totals and uses a short-lived cached summary for type/source metadata so the workbench does not show approximate counts.
• Dashboard first-load metrics keep exact visible totals, use narrower reads, and cache the expensive chart RPC for five minutes per workspace/month range.
• Schedule reference data for crews, team members, trucks, vehicles, services,

and crew-pay settings is returned by one tenant-scoped database function and cached briefly per workspace. Cold visits avoid a multi-request setup fan-out, while repeat visits still reuse the bounded reference document.

If most admin pages take multiple seconds to open, first check telemetry_pg_stat_statements_top for get_org_and_descendants, check_public_rate_limit, user_gamification_profile, rewards, and get_client_health. Those should not climb quickly during light single-user navigation.

Report And Dashboard Accuracy

Admin dashboards and analytics reports must not use approximate counts, sampled data, hidden row caps, or whole-dollar rounding for user-facing business totals.

Current accuracy controls:

• Revenue Analytics loads accepted residential, fleet, commercial building, and holiday lights revenue in paginated batches so it does not silently stop at the platform row limit.
• Service mix, sales-cycle, geography, retention, operations, schedule, lead-source ROI, holiday-lights, franchise, and margin reports page through their report inputs instead of relying on the default PostgREST window.
• Owner, residential, sales, commercial, fleet, holiday-lights, commissions, crew-pay, client, franchise, and pipeline dashboard totals use exact source rows or exact database counts for visible KPIs.
• Schedule calendar metrics now page through the selected range of jobs and sales follow-up events before calculating job counts, scheduled revenue, and average ticket.
• User-facing money totals display to the cent across admin dashboards, reports, job details, schedule views, pipeline cards, commissions, crew pay, client LTV/open pipeline, revenue charts, and exports.
• Report money is summed as integer cents where precision matters before values are converted back for display/export.
• If a report source query fails, the route should fail visibly instead of showing a partial total or fake zero as if it were complete.

Forecasts and estimates are still allowed where the workflow is explicitly predictive, such as pipeline forecast scenarios or modeled margin cost. Those surfaces must be labeled as forecasts or estimated costs and must not be presented as actual revenue, payment, or performance history.

Root Session Proxy

Protected app surfaces now pass through a shared root session proxy entrypoint instead of relying on scattered session refresh logic.

That middleware covers:

• Admin routes
• Super-admin routes
• Onboarding routes
• Logged-in customer routes
• Selected public customer-link routes that already use signed tokens

The goal is simple: keep authenticated browser sessions fresher and reduce the "randomly got sent back to login" class of failure during normal admin use.

For admin workspace resolution, the request-scoped org hint is now treated as a fast-path hint rather than trusted authority. The app still validates that the hinted active org is actually reachable from the selected membership before it reuses that context.

The same root session proxy also refreshes browser cookies on mixed-auth API surfaces that can be reached from staff workflows, including:

• /api/email
• /api/sms
• /api/estimates/*
• /api/holiday-lights/*

Those routes still enforce their own route-level auth or customer-token rules. The proxy only refreshes an existing browser session when one is present.

Inbound webhook routes are still intentionally excluded from the root session proxy because they authenticate with provider signatures, scoped tokens, or internal secrets instead of browser cookies.

The proxy now also strips the internal request-scoped org hint headers from all inbound traffic before forwarding the request deeper into the app. Those headers are only re-applied after the admin session is resolved inside the trusted proxy path, which closes the spoofing gap where an external caller could try to inject active-org context directly.

Admin API Baseline Rate Limiting

All /api/admin/* routes now inherit a baseline middleware rate limit of 60 requests per minute before any route-specific limits apply.

Important notes:

• This is a shared floor, not a replacement for tighter limits on sensitive endpoints.
• Existing route-level auth and permission checks still matter.
• The middleware limiter uses in-memory protection for admin API traffic to keep protected office reads from paying an extra database RPC. Public and token-based surfaces still use the shared database-backed limiter unless their route opts into a different policy.
• The middleware now buckets admin API traffic by authenticated user when one is available, and falls back to an IP-scoped key only when the caller has no usable session. That avoids one fast admin user starving everyone else behind the same office network.

Health Endpoint

CE Pro now exposes:

• GET /api/health
• HEAD /api/health

Use this endpoint for uptime monitors, platform health probes, and deployment checks.

The check verifies:

• the app can boot with the required runtime environment
• Supabase is reachable from the app runtime

Responses are intentionally cheap and uncached.

The health path now also uses bounded timeouts for both the dependency check and the optional internal queue snapshot. The Supabase reachability query carries its own abort signal, so a slow dependency check is cancelled at the timeout boundary instead of leaving an upstream request running after the endpoint has already degraded. If Supabase is slow or the queue snapshot cannot complete quickly, the endpoint degrades fast instead of hanging until the platform kills the request.

Healthy Response

{
  "status": "ok",
  "checked_at": "2026-03-21T15:00:00.000Z",
  "services": {
    "app": "ok",
    "supabase": "ok"
  }
}

Degraded Response

If the app can respond but a required dependency check fails, CE Pro returns 503 with a degraded status.

Startup Environment Validation

The shared Supabase bootstrap path now validates its required environment variables from one place instead of silently failing later in request handling.

That baseline validation currently covers the critical Supabase runtime config:

• NEXT_PUBLIC_SUPABASE_URL
• NEXT_PUBLIC_SUPABASE_ANON_KEY
• SUPABASE_SERVICE_ROLE_KEY

If one of those values is missing or malformed, the app now fails loudly instead of limping into production with partial behavior.

Global Error Recovery

CE Pro now includes app-wide error boundaries for route-level and full-app crashes.

That means a render failure should now fall back to a recovery screen with a retry action instead of dropping the user onto a blank white page with no path forward.

Route-specific error boundaries can still exist where a workflow needs a more tailored recovery state.

The shared fallback also routes admins, logged-in customers, and public visitors back to the right surface instead of always sending everyone to the admin dashboard.

Background Delivery Jobs

CE Pro now moves more slow outbound delivery work off the request path and onto the shared background jobs queue.

That queue now handles:

• one-time campaign fan-out batches
• manual admin inbox email sends
• commercial fleet and building proposal sends, including PDF generation, email, SMS, and delivery-state updates
• forgot-password recovery emails
• fresh team setup-link emails
• self-serve account password reset emails from Settings

The main effect is that the app can acknowledge the request faster while the minute-by-minute job worker handles the actual send work in the background.

Queued email-style delivery work now also gets retry headroom instead of acting like a one-shot fire-and-forget handoff.

That means:

• forgot-password sends retry on transient provider failures
• setup-link and account-reset sends retry instead of silently dead-lettering after one blip
• queued admin inbox emails retry before they are marked failed

What Users See

• Marketing sends now queue batches instead of holding the browser open while every recipient is processed inline.
• Manual admin emails show as queued instead of pretending delivery already finished.
• Commercial proposal sends now show as queued while the background worker renders the PDF, sends email/SMS, and records delivery state. Set COMMERCIAL_PROPOSAL_SEND_QUEUE_DISABLED=true to roll this path back to synchronous delivery.
• Password recovery and setup-link requests now rely on the background worker instead of waiting on synchronous provider delivery inside the request.

Hot-Read Server Caching

Phase 2 also adds short-lived server-side caching to a few high-traffic office reads so common dashboard and setup screens stop re-running the same database work on every load.

The current cache coverage includes:

• admin dashboard metrics
• services catalog reads
• add-on structure reads
• lead source reads and lead-source lookup helpers
• pricing config reads
• tag summary reads

These caches are intentionally short and org-scoped. Cache keys and invalidation tags include the organization boundary, so adding tenants does not allow one workspace to reuse another workspace's dashboard or reference data. Authorization resolution is not placed in these TTL caches; the mobile authorization optimization described above shares only an identical, concurrently in-flight lookup.

Important notes:

• Admin reference-data writes now invalidate their matching cache tags.
• Cached responses still keep the existing browser cache headers where those were already part of the route contract.
• Dashboard data currently relies on TTL-based refresh rather than write-trigger invalidation, which keeps the implementation safe while still cutting read pressure.
• Lead-source admin reads also now avoid selecting a non-existent updated_at column on workspaces where that field is absent, which closes the stress-test 500 that was keeping the lead-source cache path from helping at all.

Shared Org Cache

The short-lived server caches now use a shared org-cache abstraction with standard bucket types instead of one-off inline cache definitions.

That makes it easier to review:

• cache scope
• TTL intent
• org isolation
• invalidation tags

The current buckets in use are:

• reference
• dashboard
• settings

RSC-First Admin Lists

The current V2 Jobs and Invoices workspaces load their first page through bounded server-backed queries instead of depending on an unbounded browser fetch. Invoices now has a durable V2 table and route, a job entry point, reviewed delivery, governed office collection, and controlled refunds. V1-only parts include estimate/bulk creation, embedded card/tips/credits, broader automation, accounting sync, and mobile workflows.

That means:

• first paint is more useful on those pages
• the first page is already present when the client boots
• follow-up pagination and refreshes reuse the same query path

Query-Backed Pagination

Current V2 Jobs uses explicit paginated list APIs instead of the old fixed 200 row cap. V2 Invoices uses authoritative 25-record keysets with matching totals. Newest, oldest, due ascending, and total descending each have a matching ordered index path and ID tie-breaker; the cursor is signed to the exact scope, filters, and sort rather than relying on a drifting offset. Single-location scope uses location-leading seek indexes for all four sorts, bounding the page path independently of sibling-location invoice volume.

That reduces overfetching and gives those pages a clearer path to larger-office datasets.

Important note:

• Some list-level summary cards still intentionally describe the loaded slice, not the full matching dataset, unless the page explicitly labels otherwise.
• Current V2 Jobs and Invoices search/status filters run through their paginated query paths, so a matching record does not vanish merely because it lives beyond the first loaded page.
• Both workspaces parameterize free-text search before it enters a database filter.
• Jobs and Invoices have dedicated primary-list indexes for their current query shapes.
• Jobs treats an out-of-range offset as empty; Invoices rejects malformed or

cross-scope/sort cursors and returns a stable next keyset when more rows exist.

• The Jobs out-of-range protection matches the real PostgREST response shape,

including 416 Range Not Satisfiable; Invoices no longer uses offsets.

Super-Admin Browser QA

The repo now includes a read-only super-admin portal browser QA script:

npm run qa:super-admin -- --base-url=https://cleanestimate.pro --magic-link

Run it with SUPER_ADMIN_EMAIL, NEXT_PUBLIC_SUPABASE_URL, and SUPABASE_SERVICE_ROLE_KEY available in the environment. The script verifies that signed-out users are redirected away from /super-admin, signs in through a one-time Supabase magic link for the configured super-admin identity, opens the Dashboard, Organizations, organization detail, Users, Waitlist, and Activity surfaces, and records screenshots plus report.json under output/playwright/super-admin-portal-qa/.

The super-admin Users and Waitlist tables now use deterministic date rendering during hydration so browser QA does not surface React text-mismatch errors on those operational pages. The production CSP also allows the Google Analytics collect endpoint currently used by the deployed tag, which prevents analytics CSP noise from hiding real super-admin portal failures.

Signup & Onboarding Browser QA

The repo now also includes a live self-serve signup and onboarding browser QA script:

npm run qa:signup -- --base-url=https://cleanestimate.pro

Run it with AGENTMAIL_API_KEY, NEXT_PUBLIC_SUPABASE_URL, SUPABASE_SERVICE_ROLE_KEY, and INTERNAL_API_SECRET available in the environment. The script creates a fresh AgentMail inbox, signs up a new owner and organization through /signup, completes the Company Info, Services, Pricing, and Get Started onboarding steps, verifies the new owner can reach /admin, checks the database state for the owner/org/membership/trial, confirms password login from a clean browser session, then sends and opens a real owner access-link email delivered to AgentMail.

Evidence is written under output/playwright/signup-onboarding-qa/ with full-page screenshots and report.json.

In V1, the admin dashboard's Lead Sources shortcut points at /admin/analytics/lead-source-roi. Current V2 does not register that route; its smaller current source workflow is documented in Lead Sources.

Invoice Number Allocation (V1 parity reference)

V1 invoice creation allocates invoice numbers through one shared atomic counter path across:

• admin single-invoice creation
• admin bulk invoice creation
• API v1 invoice creation

That removes the older count-then-insert race where concurrent creates could reach the same invoice number under load.

The allocator also seeds itself from any already-existing invoice numbers for the same org and date, so rollout does not break on workspaces that created invoices before the counter-backed path existed.

Invoice checkout authorization and provider identity

Signed-in invoice Checkout does not run through the ordinary staff role. The customer session table stays service-only, and one service transaction revalidates the exact session and identity binding before it locks the allowed invoice and creates or reuses its payment session. Identity-changing office statements use the shared writer-gate/email-lock protocol so updates, deletes, bulk changes, and matching-email inserts cannot race stale customer authority.

Stripe payment mirrors key replay identity by organization, provider, connected-account snapshot, and provider object ID. A separate partial unique index also prevents two organizations from claiming the same account/object pair. Checkout registration must return the exact organization, account, provider object, and invoice expected by the transaction before the URL can be returned. The isolated PostgreSQL release test exercises the restricted staff-role/service boundary with query, lock, and idle-transaction timeouts in addition to the two-connection deposit-allocation race.

Invoice delivery and ledger serialization

Sending freezes the invoice document but does not collapse mutable provider delivery state into that snapshot. Each reviewed email/SMS channel creates an append-only attempt with its own idempotency and provider outcome. Retry creates a later attempt; provider, consent, configuration, or recipient failures stay queryable as action-needed evidence.

Customer Checkout, admin partial/split collection, deposit allocation, and refund mutation share the invoice/payment lock order before deriving the remaining or refundable amount. Manual and provider tenders and their refunds remain separate immutable rows, so a projection can calculate gross settled, refunded, net paid, and collectible balance without editing history. The zero-balance transition inserts the lifetime first invoice.paid occurrence in the same transaction behind a uniqueness boundary. Concurrent final tenders, provider retries, refunds, and later recollection therefore converge without overcollection or a second first-paid event.

Office manual collection and hosted Checkout take the invoice money locks and then lock/revalidate an active actor with both manage_invoices and issue_refunds plus current global or matching-location entitlement. Permission, member-active, and location-assignment changes share that determinant boundary, so a reassignment that commits first denies stale collection, Checkout, refund, and provider-refund retry.

Database guards deny raw application-role payment inserts, refund-command state changes, and delivery-outcome updates. Only the corresponding service/provider path can publish those terminal facts. Pending refund recovery is service-only and revalidates the complete tenant, invoice, payment, command, stored actor, amount/reason/idempotency identity, provider account, and provider payment before it resumes, including that actor's current financial grants and office entitlement.

The web mutation returns its committed terminal money result separately from the follow-up detail refresh. A refresh failure therefore produces a refresh notice without recasting succeeded money as a failed mutation, and the same idempotency identity remains available for a safe replay.

The admin card path uses the existing connected-account Checkout registry. It returns Open secure card checkout and copy-link data only after the exact organization, account, invoice, balance, version, and return surface are registered. The staff-device URL may return to the authenticated office invoice, while the copied URL is registered with a customer-safe public invoice/confirmation return and cannot redirect into admin authentication. Compatible reuse is bound to that complete identity, and the ledger does not settle until the verified webhook arrives.

Manual payment reference/note text remains in the office ledger projection. Customer bearer, account, payment-history, and PDF projections select an allowlisted method label and safe accounting fields explicitly; they do not select freeform staff text or provider payloads.

Public invoice GET requests do not record a view. The interactive browser must send a separate scoped acknowledgement after rendering the customer invoice; scanner, crawler, unfurl, office-preview, and Checkout-poll traffic never takes that path.

Phase 4 Audit Follow-Up

The post-Phase-4 review also shipped a few correctness fixes that are easy to miss if you only look at the higher-level roadmap:

• lead-source reference reads now respect the includeAll=false active-only filter again
• current V2 Jobs server hydration marks the initial query payload as fresh; V1 Invoices preserves the matching behavior for future parity
• the background-job cron now claims a smaller bounded batch per run so slow workers are less likely to hit the serverless timeout and leave claimed jobs waiting for stale-lock recovery

Structured Logging And Queue Health

The background job processor and internal delivery workers now emit structured log events instead of free-form console output.

Those logs make it easier to separate:

• healthy empty worker runs
• claimed batch runs
• retrying jobs
• dead-letter failures

The structured logger now also avoids writing full recovery-email and admin-email recipient addresses into production logs. Those logs keep enough masked recipient detail for debugging without leaving plaintext inbox addresses in the log stream.

The health endpoint also supports an internal queue snapshot when called with the same bearer secret used by the cron worker.

Use the dedicated Operations Runbook for alert thresholds, queue checks, and restore-readiness expectations.

Use the dedicated Stress Testing guide for the repeatable 1K-user readiness drill, default session mix, cookie-pool setup, and pass/fail thresholds.

If you are moving CE Pro to a new hosted Supabase project or region, use the dedicated Supabase Region Migration guide before switching traffic.

Queue Correctness Follow-Up

The background job dedupe layer now only blocks duplicate work while a matching job is still in an active queue state:

• pending
• processing
• retrying

Completed, failed, and dead-letter jobs no longer permanently reserve the same dedupe key, which means legitimate follow-up queueing can happen again after the earlier work has fully finished.

Campaign dispatch stat reconciliation also now runs through a database-side sync step instead of an app-side read-then-write cycle. That removes the race where multiple finishing workers could overwrite each other's campaign status or message totals.

The campaign batch worker now isolates failures at the recipient level instead of aborting the whole batch on the first provider error. A single transient email failure no longer prevents the rest of the batch from being attempted, and the stats sync step still runs after the batch work completes.

The background-job processor also now reports jobs that got stuck before they could be finalized as a separate stuck_processing outcome instead of counting them as dead-letter work they never actually reached.

Recovery Email Burst Protection

Forgot-password requests now rate-limit by normalized email address instead of only by source IP.

That means:

• rotating IPs or VPN exits no longer bypass the per-address recovery limit
• rapid duplicate forgot-password requests collapse onto one queued recovery job instead of stacking multiple identical email sends

Permission Hardening Follow-Up

The Phase 3 admin-route hardening pass now keeps read and destructive permissions separated on the detail routes that were still inconsistent.

The two practical outcomes are:

• estimate delete operations now require estimates.manage instead of slipping through the same top-level gate used for estimate reads
• lead detail reads now use the view-side client permission instead of the stronger write tuple used for lead mutation routes
• invoice attachment reads and writes now follow billing permissions instead of estimate permissions, so billing-only roles can manage invoice files without inheriting estimate access and estimate-only roles do not gain invoice attachment visibility by accident

That keeps destructive behavior aligned with the route intent while avoiding unnecessary lead-detail lockouts for read-capable office roles.

Log Redaction Follow-Up

The Phase 5 structured logging pass now masks email addresses on the enqueue-failure paths too, not just in the background workers that actually deliver mail.

That means recovery-email, setup-link, account-reset, and admin-email queue failures no longer write plaintext recipient addresses into structured logs while operators are debugging a queue outage.

API V1 CORS Cache Safety

The API v1 response helper now only emits Vary: Origin when it also emitted an Access-Control-Allow-Origin header.

That makes cached server-to-server responses less likely to interfere with later browser requests on installations that put an edge cache in front of the external API surface.