Platform Reliability

CE Pro now includes a shared baseline for session refresh, admin API protection, health checks, and app-level crash recovery.

These platform controls are not a replacement for route-level auth, validation, or product-specific permissions. They exist so the app behaves more predictably as traffic grows.

Mobile Multi-Tenant API Baseline

The authenticated mobile API now has one shared tenant-resolution path. It:

• validates bearer tokens with Supabase Auth
• validates X-Organization-Id as a UUID
• confirms an active, non-archived membership for that exact organization
• rejects inactive organizations
• resolves role permissions and feature flags once per request
• emits X-Request-ID, Server-Timing, private/no-store cache policy, and

authorization/workspace Vary metadata from the shared response helper

The client only accepts relative mobile API paths on the configured CleanEstimate origin, which prevents an authorization token from being sent to an arbitrary destination. Calls have a 12-second deadline. GET and HEAD calls can retry transient 408, 425, 429, and gateway/server failures with bounded backoff; mutation calls default to zero retries. Concurrent identical GETs are deduplicated by selected organization and path.

The server also deduplicates concurrent workspace-context resolution for the same verified user and exact selected organization. This is deliberately in-flight-only: the entry is removed as soon as the shared lookup settles, and failed lookups are never retained. Concurrent requests with the exact same bearer token and X-Organization-Id may share the pending token validation and workspace resolution; a different token or organization always resolves independently. Authorization for one tenant can therefore never satisfy a request for another tenant, and membership or permission changes are not hidden behind a time-based authorization cache.

Explicit tenant selection fails closed. A malformed X-Organization-Id, or a valid organization id for which the authenticated user has no active membership, is rejected rather than falling back to another workspace. Only a stale server-owned active_org_id account-metadata hint can recover to the user's first active membership when the device did not explicitly choose an organization.

On native iOS and Android, Supabase token refresh starts while the app is active and stops while it is backgrounded. When an API response still returns 401, the client checks for an already refreshed session or performs one shared token refresh, then retries a safe read once with the same organization and request ID. Concurrent rejected requests share that refresh promise only when they belong to the same account. Mutations can refresh the session for the next explicit action, but they are not replayed automatically after authentication, network, validation, or server failures.

The registered mobile contract is available from:

GET /api/openapi/mobile.json

Contract tests compare that registry with every implemented /api/mobile/**/route.ts operation and verify that every mobile route resolves one of the approved authenticated contexts. The registry is the handoff point for a future API gateway or mapper; it does not replace server-side authorization.

The Vercel source-ignore rule is anchored to the repository's root /mobile workspace. A packaging test prevents an unanchored mobile rule from accidentally excluding the Next.js routes under src/app/api/mobile.

Mobile Dashboard and Search fan independent tenant-scoped reads out with Promise.all, keep explicit result limits, and preserve representative assignment filters. Organization-first partial indexes cover active membership resolution plus the clients, leads, estimates, appointments, and jobs recency paths. This keeps common queries proportional to the selected workspace rather than total platform row count.

The native client also connects its query lifecycle to app focus and network state. Backgrounded screens pause periodic reads, disconnected devices stop starting work that cannot complete, and the active screen resumes or refreshes through the shared query lifecycle when the app becomes usable again. Repeated polls are scoped to the focused workspace screen instead of every mounted tab. Non-critical native startup work, including voice and notification setup, waits until the first app interactions finish so it does not compete with the initial workspace render.

Production Telemetry Baseline

Phase 0 scaling work now starts with production telemetry, not synthetic load tests.

Before CE Pro changes database access patterns, adds projections, provisions replicas, or raises infrastructure spend, the team must capture a production baseline that shows:

• the top 10 slow app routes by p95 and p99
• the top 20 slow SQL statements from pg_stat_statements
• the current Supabase connection snapshot, including total, active, idle, idle-in-transaction, waiting, and max_connections

The app collects sampled browser navigation and same-origin API timing events in production. Sampling defaults to 5% on public/non-admin routes, while authenticated admin pages default to 20% so workhorse-route regressions show up faster. Override non-admin sampling with NEXT_PUBLIC_ROUTE_TELEMETRY_SAMPLE_RATE, override admin sampling with NEXT_PUBLIC_ROUTE_TELEMETRY_ADMIN_SAMPLE_RATE, and disable client-side collection with NEXT_PUBLIC_ROUTE_TELEMETRY_ENABLED=false. The ingestion endpoint can also be shut off server-side with ROUTE_TELEMETRY_INGEST_DISABLED=true. Telemetry ingest uses the in-memory limiter so collecting route timings does not spend another database RPC before writing the actual sample.

Telemetry writes land in route_performance_events. The export command reads service-role-only RPCs for route timing, pg_stat_statements, and connection usage:

npm run telemetry:production -- --days=7 --out=reports/production-telemetry.md

The Phase 0 stop condition is strict: if production telemetry shows the current SLOs are already met at the current measured load, halt Phases 1-6 and re-evaluate quarterly instead of making speculative architecture changes.

The business-load target is also a hard prerequisite before optimization work beyond telemetry: the business owner must write down N simultaneous authenticated active sessions by date D, justified by the launch, customer, or usage reason. Without that number, scale work has no anchor.

Cost gates stay explicit. Any phase that would push monthly infrastructure spend above $250 needs Clean Estimate Pro business-owner approval before it starts. A higher ceiling is only unlocked when current MRR supports it: $500/month after $10k MRR, $1,000/month after $25k MRR, or a written launch-risk exception from the business owner.

Auth and Worker Backpressure

When Supabase Auth or PostgREST starts returning gateway timeouts, the sign-in screen now fails visibly instead of leaving the user stuck on Signing in.... Password sign-in waits up to 15 seconds and then shows a retryable auth-service message.

The scheduled worker routes also include incident controls so database pressure can be reduced without changing user-facing app code:

• A project-level reliability probe runs every minute and records database

response time, size, active/max connections, and long transactions. Three consecutive degraded probes open a ten-minute circuit for high-fanout scheduled work. Two consecutive healthy probes clear it.

• While the circuit is open, scheduled automation, waiting automation,

background jobs, follow-ups, drip campaigns, and webhook deliveries return a successful disabled response without claiming work. If the guard itself cannot reach the database within two seconds, those workers fail closed so user traffic gets the available capacity.

• DISABLE_CRON_JOBS=true pauses the full scheduled-work layer after each route authenticates and before it opens a Supabase connection.
• DISABLE_SCHEDULED_WORK=true is an equivalent global scheduler pause switch for incidents where the word "cron" is too narrow.
• DISABLE_BACKGROUND_JOB_CRON=true pauses only /api/cron/process-background-jobs.
• DISABLE_WEBHOOK_DELIVERY_CRON=true pauses only /api/cron/process-webhook-deliveries.
• DISABLE_FOLLOW_UP_CRON=true pauses only /api/cron/process-follow-ups.
• DISABLE_DRIP_CAMPAIGN_CRON=true pauses only /api/cron/process-drip-campaigns.
• DISABLE_EXPIRE_PROPOSALS_CRON=true pauses only /api/cron/expire-proposals.
• DISABLE_AUTOMATION_WAITING_CRON=true pauses only /api/automations/engine/process-waiting.
• DISABLE_AUTOMATION_SCHEDULED_TRIGGER_CRON=true pauses only /api/automations/engine/process-scheduled-triggers.
• DISABLE_FRANCHISE_SUMMARY_CRON=true pauses only /api/cron/refresh-franchise-summary.
• BACKGROUND_JOB_CRON_BATCH_SIZE and BACKGROUND_JOB_CRON_CONCURRENCY can raise or lower the background worker from its conservative defaults of 5 jobs and 1 concurrent dispatch.
• WEBHOOK_DELIVERY_CRON_BATCH_SIZE can raise or lower webhook delivery claiming from its conservative default of 10 deliveries.

The public ingress rate-limit RPC is also lock-bounded. If one hot key is already being updated, the limiter fails closed quickly instead of holding PostgREST workers on a row lock until the platform times out the request.

Scheduled appointment automation now groups active workflows by organization, appointment scope, and reminder window. One bounded candidate query serves all workflows in the same group instead of repeating the same leads, estimates, or jobs read for every workflow. Matching entity dispatch is capped at four concurrent items.

Workflow-run idempotency now uses the indexed workflow_runs.dedupe_key column. A partial unique index on (workflow_id, dedupe_key) makes duplicate prevention atomic and removes the previous 400-day JSON history scan from the worker hot path.

If Supabase warns that the project is about to deplete or has depleted its Disk IO Budget, set the global scheduler pause first, redeploy so every route has the guard, avoid super-admin/dashboard refreshes, then either wait for the next daily IO budget replenishment or temporarily upgrade compute while the team captures pg_stat_statements and fixes the disk-heavy queries. Compute add-ons require the Supabase organization to be on a paid plan; free-tier projects must be upgraded to a paid Supabase plan before the emergency compute add-on can be applied.

Super-admin pages are dynamic-only so protected platform dashboards never block a Vercel build on live Supabase reads.

Sitewide Page-Load Controls

Navigation throughout the public website, authenticated admin app, and customer surfaces uses intent-based prefetching. Links do not preload every destination merely because a desktop and mobile navigation tree exists in the page. The exact destination is prefetched once when a user points to, focuses, or touches that link, preserving fast follow-through without flooding a new page load with speculative React Server Component requests.

Loading boundaries give route transitions an immediate visual response while the destination's required data is still resolving. Optional tracking is loaded after the interactive page work, so analytics does not compete with the content and controls needed for the first useful render.

Large interactive workspaces should also keep inactive panels out of the initial client bundle. The Sales Workspace streams tenant-scoped quote configuration behind its immediately usable shell, then loads quote, lead, schedule, customer-search, and follow-up code on intent. Customer estimate pages follow the same pattern for proposal, maintenance, interaction, and photo panels. Public estimate requests first authorize the raw estimate record, so a shared-quote redirect can happen before residential offer hydration or optional reads. For legacy estimate rendering, offer hydration runs alongside independent organization, pricing, and design reads. Canonical acceptance reuses the already-authorized estimate and organization result instead of repeating those tenant-scoped reads. Every database read remains constrained to the estimate's organization. Independent holiday-light design backdrops and mockups are also fetched concurrently while preserving their presentation order.

Read-only workspace bootstrap paths use their own organization-scoped cache keys. In particular, opening Sales can read the existing residential catalog and lead-source defaults without running the write-oriented new-tenant bootstrap sequence, and that read-only cache cannot suppress a later bootstrap-enabled setup request.

These rules apply sitewide rather than only to Command Center. When adding a new shared navigation component:

• keep automatic viewport prefetch off for large menus or duplicated

desktop/mobile navigation trees

• use the shared intent-prefetch link for internal destinations
• do not eagerly import optional tracking, support, or visualization code into

the initial route bundle

• keep tenant-specific data behind the authenticated, organization-scoped

server path even if the route shell can render immediately

Admin Alert Backpressure

The admin alert bell is a read path. It should not regenerate managed alerts or write to ai_alerts on every poll. Managed alert generation is reserved for the dashboard alert scope, is throttled by ADMIN_ALERT_SYNC_TTL_MS with a five-minute default, and skips unchanged rows so realtime subscriptions do not create read/write feedback loops.

The admin shell now owns one tenant-scoped live-data feed for alert consumers. Desktop navigation, mobile navigation, and the Command Center alert panel reuse that feed and its realtime refresh path instead of mounting independent polls and subscriptions. The feed key includes the selected organization, and a workspace switch creates a distinct feed rather than reusing another tenant's alerts.

If the admin shell feels slow, check telemetry_route_performance_summary for /api/admin/alerts and telemetry_pg_stat_statements_top for ai_alerts updates. A healthy alert path should show reads and occasional inserts or updates, not continuous update traffic while the user is idle.

Admin Navigation Read Pressure

The admin shell keeps ordinary page navigation on the lightest available read path.

Current controls:

• Protected-route middleware resolves membership selection, franchise access,

onboarding state, and role-permission overrides in one service-role RPC. Together with the bounded Supabase fetch, a typical admin route now needs an auth verification and one organization-context database request instead of several sequential PostgREST reads.

• The sidebar gamification badge is read-only during normal page loads and no longer seeds catalogs, upserts reward rows, recalculates profiles, or records daily-login XP from the shared layout.
• Duplicate desktop and mobile shell consumers share live alert and unread-message state instead of issuing separate initial reads, polling loops, and realtime subscriptions.
• Large navigation trees use intent prefetching so a page load does not trigger a speculative request for every visible admin destination.
• Command Center branches into the signed-in user's role-specific data path before loading office-only metrics. Sales and crew workspaces do not wait for unrelated owner/manager queries.
• Optional shell data is kept off the destination page's critical render path, and route loading boundaries provide immediate feedback while required tenant data resolves.
• Admin API middleware rate limiting uses the in-memory limiter for /api/admin/* traffic so ordinary schedule, client, and dashboard reads do not spend a database RPC before the route handler authenticates and authorizes the request.
• The Clients list avoids an unused exact count on the standalone contact query and narrows client-health work to the visible client ids when possible.
• The Clients page shell no longer blocks server navigation on the heavy account tree API. The page opens with a loading state and hydrates the account/client rows through the existing client-side endpoint.
• The Estimates page shell follows the same pattern. It no longer waits on server-side calls back into /api/admin/estimates and /api/admin/team; the browser hydrates the first list and team filters after the shell is interactive.
• The Estimates list API keeps exact visible totals and uses a short-lived cached summary for type/source metadata so the workbench does not show approximate counts.
• Dashboard first-load metrics keep exact visible totals, use narrower reads, and cache the expensive chart RPC for five minutes per workspace/month range.
• Schedule reference data for crews, team members, trucks, vehicles, services, and crew-pay policies is cached briefly per workspace so repeat visits do not re-run the same setup reads.

If most admin pages take multiple seconds to open, first check telemetry_pg_stat_statements_top for get_org_and_descendants, check_public_rate_limit, user_gamification_profile, rewards, and get_client_health. Those should not climb quickly during light single-user navigation.

Report And Dashboard Accuracy

Admin dashboards and analytics reports must not use approximate counts, sampled data, hidden row caps, or whole-dollar rounding for user-facing business totals.

Current accuracy controls:

• Revenue Analytics loads accepted residential, fleet, commercial building, and holiday lights revenue in paginated batches so it does not silently stop at the platform row limit.
• Service mix, sales-cycle, geography, retention, operations, schedule, lead-source ROI, holiday-lights, franchise, and margin reports page through their report inputs instead of relying on the default PostgREST window.
• Owner, residential, sales, commercial, fleet, holiday-lights, commissions, crew-pay, client, franchise, and pipeline dashboard totals use exact source rows or exact database counts for visible KPIs.
• Schedule calendar metrics now page through the selected range of jobs and sales follow-up events before calculating job counts, scheduled revenue, and average ticket.
• User-facing money totals display to the cent across admin dashboards, reports, job details, schedule views, pipeline cards, commissions, crew pay, client LTV/open pipeline, revenue charts, and exports.
• Report money is summed as integer cents where precision matters before values are converted back for display/export.
• If a report source query fails, the route should fail visibly instead of showing a partial total or fake zero as if it were complete.

Forecasts and estimates are still allowed where the workflow is explicitly predictive, such as pipeline forecast scenarios or modeled margin cost. Those surfaces must be labeled as forecasts or estimated costs and must not be presented as actual revenue, payment, or performance history.

Root Session Proxy

Protected app surfaces now pass through a shared root session proxy entrypoint instead of relying on scattered session refresh logic.

That middleware covers:

• Admin routes
• Super-admin routes
• Onboarding routes
• Logged-in customer routes
• Selected public customer-link routes that already use signed tokens

The goal is simple: keep authenticated browser sessions fresher and reduce the "randomly got sent back to login" class of failure during normal admin use.

For admin workspace resolution, the request-scoped org hint is now treated as a fast-path hint rather than trusted authority. The app still validates that the hinted active org is actually reachable from the selected membership before it reuses that context.

The same root session proxy also refreshes browser cookies on mixed-auth API surfaces that can be reached from staff workflows, including:

• /api/email
• /api/sms
• /api/estimates/*
• /api/holiday-lights/*

Those routes still enforce their own route-level auth or customer-token rules. The proxy only refreshes an existing browser session when one is present.

Inbound webhook routes are still intentionally excluded from the root session proxy because they authenticate with provider signatures, scoped tokens, or internal secrets instead of browser cookies.

The proxy now also strips the internal request-scoped org hint headers from all inbound traffic before forwarding the request deeper into the app. Those headers are only re-applied after the admin session is resolved inside the trusted proxy path, which closes the spoofing gap where an external caller could try to inject active-org context directly.

Admin API Baseline Rate Limiting

All /api/admin/* routes now inherit a baseline middleware rate limit of 60 requests per minute before any route-specific limits apply.

Important notes:

• This is a shared floor, not a replacement for tighter limits on sensitive endpoints.
• Existing route-level auth and permission checks still matter.
• The middleware limiter uses in-memory protection for admin API traffic to keep protected office reads from paying an extra database RPC. Public and token-based surfaces still use the shared database-backed limiter unless their route opts into a different policy.
• The middleware now buckets admin API traffic by authenticated user when one is available, and falls back to an IP-scoped key only when the caller has no usable session. That avoids one fast admin user starving everyone else behind the same office network.

Health Endpoint

CE Pro now exposes:

• GET /api/health
• HEAD /api/health

Use this endpoint for uptime monitors, platform health probes, and deployment checks.

The check verifies:

• the app can boot with the required runtime environment
• Supabase is reachable from the app runtime

Responses are intentionally cheap and uncached.

The health path now also uses bounded timeouts for both the dependency check and the optional internal queue snapshot. The Supabase reachability query carries its own abort signal, so a slow dependency check is cancelled at the timeout boundary instead of leaving an upstream request running after the endpoint has already degraded. If Supabase is slow or the queue snapshot cannot complete quickly, the endpoint degrades fast instead of hanging until the platform kills the request.

Healthy Response

{
  "status": "ok",
  "checked_at": "2026-03-21T15:00:00.000Z",
  "services": {
    "app": "ok",
    "supabase": "ok"
  }
}

Degraded Response

If the app can respond but a required dependency check fails, CE Pro returns 503 with a degraded status.

Startup Environment Validation

The shared Supabase bootstrap path now validates its required environment variables from one place instead of silently failing later in request handling.

That baseline validation currently covers the critical Supabase runtime config:

• NEXT_PUBLIC_SUPABASE_URL
• NEXT_PUBLIC_SUPABASE_ANON_KEY
• SUPABASE_SERVICE_ROLE_KEY

If one of those values is missing or malformed, the app now fails loudly instead of limping into production with partial behavior.

Global Error Recovery

CE Pro now includes app-wide error boundaries for route-level and full-app crashes.

That means a render failure should now fall back to a recovery screen with a retry action instead of dropping the user onto a blank white page with no path forward.

Route-specific error boundaries can still exist where a workflow needs a more tailored recovery state.

The shared fallback also routes admins, logged-in customers, and public visitors back to the right surface instead of always sending everyone to the admin dashboard.

Background Delivery Jobs

CE Pro now moves more slow outbound delivery work off the request path and onto the shared background jobs queue.

That queue now handles:

• one-time campaign fan-out batches
• manual admin inbox email sends
• commercial fleet and building proposal sends, including PDF generation, email, SMS, and delivery-state updates
• forgot-password recovery emails
• fresh team setup-link emails
• self-serve account password reset emails from Settings

The main effect is that the app can acknowledge the request faster while the minute-by-minute job worker handles the actual send work in the background.

Queued email-style delivery work now also gets retry headroom instead of acting like a one-shot fire-and-forget handoff.

That means:

• forgot-password sends retry on transient provider failures
• setup-link and account-reset sends retry instead of silently dead-lettering after one blip
• queued admin inbox emails retry before they are marked failed

What Users See

• Marketing sends now queue batches instead of holding the browser open while every recipient is processed inline.
• Manual admin emails show as queued instead of pretending delivery already finished.
• Commercial proposal sends now show as queued while the background worker renders the PDF, sends email/SMS, and records delivery state. Set COMMERCIAL_PROPOSAL_SEND_QUEUE_DISABLED=true to roll this path back to synchronous delivery.
• Password recovery and setup-link requests now rely on the background worker instead of waiting on synchronous provider delivery inside the request.

Hot-Read Server Caching

Phase 2 also adds short-lived server-side caching to a few high-traffic office reads so common dashboard and setup screens stop re-running the same database work on every load.

The current cache coverage includes:

• admin dashboard metrics
• services catalog reads
• add-on structure reads
• lead source reads and lead-source lookup helpers
• pricing config reads
• tag summary reads

These caches are intentionally short and org-scoped. Cache keys and invalidation tags include the organization boundary, so adding tenants does not allow one workspace to reuse another workspace's dashboard or reference data. Authorization resolution is not placed in these TTL caches; the mobile authorization optimization described above shares only an identical, concurrently in-flight lookup.

Important notes:

• Admin reference-data writes now invalidate their matching cache tags.
• Cached responses still keep the existing browser cache headers where those were already part of the route contract.
• Dashboard data currently relies on TTL-based refresh rather than write-trigger invalidation, which keeps the implementation safe while still cutting read pressure.
• Lead-source admin reads also now avoid selecting a non-existent updated_at column on workspaces where that field is absent, which closes the stress-test 500 that was keeping the lead-source cache path from helping at all.

Shared Org Cache

The short-lived server caches now use a shared org-cache abstraction with standard bucket types instead of one-off inline cache definitions.

That makes it easier to review:

• cache scope
• TTL intent
• org isolation
• invalidation tags

The current buckets in use are:

• reference
• dashboard
• settings

RSC-First Admin Lists

The Jobs and Invoices workspaces now load their first page on the server and hydrate into client-side query state instead of waiting for a mount-time waterfall before anything useful appears.

That means:

• first paint is more useful on those pages
• the first page is already present when the client boots
• follow-up pagination and refreshes reuse the same query path

Query-Backed Pagination

The Jobs and Invoices admin workspaces also moved off the old fixed 200 row cap and onto explicit paginated list APIs.

That reduces overfetching and gives those pages a clearer path to larger-office datasets.

Important note:

• Some list-level summary cards still intentionally describe the loaded slice, not the full matching dataset, unless the page explicitly labels otherwise.
• Jobs and Invoices search and status filtering now run through the paginated query path too, so records do not appear to vanish just because they live beyond the first loaded page.
• Jobs and Invoices now sanitize free-text search terms before they are interpolated into the PostgREST filter DSL, which avoids malformed search expressions on punctuation-heavy queries.
• Jobs and Invoices now also have dedicated (org_id, api_environment, created_at DESC) indexes for the primary admin list path, so those pages benefit from the same scaling work instead of relying on less-specific secondary indexes.
• Jobs and Invoices now also treat out-of-range page offsets as empty pages instead of surfacing a server error when a workspace does not have enough rows to satisfy the requested offset yet.
• That out-of-range protection now matches the real PostgREST response shape too, including the 416 Range Not Satisfiable case where the HTTP status is returned alongside the error payload instead of inside it.

Super-Admin Browser QA

The repo now includes a read-only super-admin portal browser QA script:

npm run qa:super-admin -- --base-url=https://cleanestimate.pro --magic-link

Run it with SUPER_ADMIN_EMAIL, NEXT_PUBLIC_SUPABASE_URL, and SUPABASE_SERVICE_ROLE_KEY available in the environment. The script verifies that signed-out users are redirected away from /super-admin, signs in through a one-time Supabase magic link for the configured super-admin identity, opens the Dashboard, Organizations, organization detail, Users, Waitlist, and Activity surfaces, and records screenshots plus report.json under output/playwright/super-admin-portal-qa/.

The super-admin Users and Waitlist tables now use deterministic date rendering during hydration so browser QA does not surface React text-mismatch errors on those operational pages. The production CSP also allows the Google Analytics collect endpoint currently used by the deployed tag, which prevents analytics CSP noise from hiding real super-admin portal failures.

Signup & Onboarding Browser QA

The repo now also includes a live self-serve signup and onboarding browser QA script:

npm run qa:signup -- --base-url=https://cleanestimate.pro

Run it with AGENTMAIL_API_KEY, NEXT_PUBLIC_SUPABASE_URL, SUPABASE_SERVICE_ROLE_KEY, and INTERNAL_API_SECRET available in the environment. The script creates a fresh AgentMail inbox, signs up a new owner and organization through /signup, completes the Company Info, Services, Pricing, and Get Started onboarding steps, verifies the new owner can reach /admin, checks the database state for the owner/org/membership/trial, confirms password login from a clean browser session, then sends and opens a real owner access-link email delivered to AgentMail.

Evidence is written under output/playwright/signup-onboarding-qa/ with full-page screenshots and report.json.

The admin dashboard's Lead Sources shortcut now points at the live /admin/analytics/lead-source-roi report instead of prefetching a missing analytics route, which keeps browser QA console output focused on actual signup and auth failures.

Invoice Number Allocation

Invoice creation now allocates invoice numbers through one shared atomic counter path across:

• admin single-invoice creation
• admin bulk invoice creation
• API v1 invoice creation

That removes the older count-then-insert race where concurrent creates could reach the same invoice number under load.

The allocator also seeds itself from any already-existing invoice numbers for the same org and date, so rollout does not break on workspaces that created invoices before the counter-backed path existed.

Phase 4 Audit Follow-Up

The post-Phase-4 review also shipped a few correctness fixes that are easy to miss if you only look at the higher-level roadmap:

• lead-source reference reads now respect the includeAll=false active-only filter again
• server-hydrated Jobs and Invoices data now marks the initial React Query payload as fresh instead of immediately re-fetching on mount
• the background-job cron now claims a smaller bounded batch per run so slow workers are less likely to hit the serverless timeout and leave claimed jobs waiting for stale-lock recovery

Structured Logging And Queue Health

The background job processor and internal delivery workers now emit structured log events instead of free-form console output.

Those logs make it easier to separate:

• healthy empty worker runs
• claimed batch runs
• retrying jobs
• dead-letter failures

The structured logger now also avoids writing full recovery-email and admin-email recipient addresses into production logs. Those logs keep enough masked recipient detail for debugging without leaving plaintext inbox addresses in the log stream.

The health endpoint also supports an internal queue snapshot when called with the same bearer secret used by the cron worker.

Use the dedicated Operations Runbook for alert thresholds, queue checks, and restore-readiness expectations.

Use the dedicated Stress Testing guide for the repeatable 1K-user readiness drill, default session mix, cookie-pool setup, and pass/fail thresholds.

If you are moving CE Pro to a new hosted Supabase project or region, use the dedicated Supabase Region Migration guide before switching traffic.

Queue Correctness Follow-Up

The background job dedupe layer now only blocks duplicate work while a matching job is still in an active queue state:

• pending
• processing
• retrying

Completed, failed, and dead-letter jobs no longer permanently reserve the same dedupe key, which means legitimate follow-up queueing can happen again after the earlier work has fully finished.

Campaign dispatch stat reconciliation also now runs through a database-side sync step instead of an app-side read-then-write cycle. That removes the race where multiple finishing workers could overwrite each other's campaign status or message totals.

The campaign batch worker now isolates failures at the recipient level instead of aborting the whole batch on the first provider error. A single transient email failure no longer prevents the rest of the batch from being attempted, and the stats sync step still runs after the batch work completes.

The background-job processor also now reports jobs that got stuck before they could be finalized as a separate stuck_processing outcome instead of counting them as dead-letter work they never actually reached.

Recovery Email Burst Protection

Forgot-password requests now rate-limit by normalized email address instead of only by source IP.

That means:

• rotating IPs or VPN exits no longer bypass the per-address recovery limit
• rapid duplicate forgot-password requests collapse onto one queued recovery job instead of stacking multiple identical email sends

Permission Hardening Follow-Up

The Phase 3 admin-route hardening pass now keeps read and destructive permissions separated on the detail routes that were still inconsistent.

The two practical outcomes are:

• estimate delete operations now require estimates.manage instead of slipping through the same top-level gate used for estimate reads
• lead detail reads now use the view-side client permission instead of the stronger write tuple used for lead mutation routes
• invoice attachment reads and writes now follow billing permissions instead of estimate permissions, so billing-only roles can manage invoice files without inheriting estimate access and estimate-only roles do not gain invoice attachment visibility by accident

That keeps destructive behavior aligned with the route intent while avoiding unnecessary lead-detail lockouts for read-capable office roles.

Log Redaction Follow-Up

The Phase 5 structured logging pass now masks email addresses on the enqueue-failure paths too, not just in the background workers that actually deliver mail.

That means recovery-email, setup-link, account-reset, and admin-email queue failures no longer write plaintext recipient addresses into structured logs while operators are debugging a queue outage.

API V1 CORS Cache Safety

The API v1 response helper now only emits Vary: Origin when it also emitted an Access-Control-Allow-Origin header.

That makes cached server-to-server responses less likely to interfere with later browser requests on installations that put an edge cache in front of the external API surface.