Monitoring Workflow Runs

Current V2 scope: V2 exposes recent runs and ordered step outcomes for every linear workflow. Cancellation and bounded retry are available to teammates with Automations manage.

Current V2 Run History

Open Automations > Runs. Each row identifies the workflow, subject, enrollment type, start time, completion time when available, and current status. Open the run to review every ordered step, including wait, message, tag, notification, and task outcomes.

Common states are Waiting, Running, Completed, Stopped, and Failed. A run can stop safely before another send because the proposal was accepted, declined, expired, withdrawn, or superseded, or because the customer replied or booked. Provider and recipient safety results remain attached to the durable step instead of being reported as a successful send.

Runs use the immutable live revision that enrolled them. Editing or activating later cannot rewrite an existing run's copy or timing. Pausing blocks new enrollment and the runner rechecks workflow state at dispatch boundaries, so a message is not knowingly released through a pause race.

Cancel records when the run was canceled, who canceled it, and the reason. Retry creates the next bounded attempt with a link to the previous failed or canceled step. Retrying does not erase cancellation evidence or silently replay a provider action whose delivery result is uncertain.

Automations view can inspect this history. Automations manage is needed for cancellation, retry, lifecycle changes, and dry runs. Preview and test results are not customer sends and never appear as provider delivery success.

Email/SMS execution applies the customer's channel and marketing preferences, opt-out rules, and current recipient data. Business-hour holds use the configured location/workflow timezone. Each channel/action has a durable idempotency key, and ambiguous provider execution is retained for review instead of blindly resent.

Broader Clean Estimate V1 Reference

Every time a workflow fires, it creates a run. The run history shows you exactly what happened, step by step, for every contact that entered the workflow.

Invoice version boundary: Invoice-due and payment-failed automation recovery on this page describes the retained Clean Estimate V1 workflow engine. Current V2 has durable reviewed email/SMS delivery, emits the lifetime-first invoice.paid product event exactly once when the invoice ledger first reaches a zero collectible balance, and now also raises invoice.overdue from a daily sweep (see Automation Triggers) rather than the scheduled-recovery cursor described below. Invoice-due, payment-failed, and the broader V1 invoice lifecycle trigger set otherwise remain not yet admitted to Workflow Runs.

Delayed runs and scheduled reminder checks now use the same internal automation engine auth path behind the scenes, including the same secret precedence and timing-safe bearer-token validation. That reduces environment-specific mismatches where waiting runs and scheduled triggers could be accepted by one internal engine endpoint but rejected by another.

When one inbound event or webhook matches multiple runs at once, CE Pro now fans those run executions out in parallel instead of walking them one by one. That keeps the run queue moving faster after bursty events while still capturing per-run failures in the response and audit trail.

Scheduled Trigger Recovery

In V1, appointment, estimate-expiration, contract-renewal, invoice-due, and invoice-overdue triggers are first recorded as durable scheduled events. The scheduler keeps a separate saved cursor for each workflow, converts due records into unique queue rows, and claims a small batch at a time. Progress in one workflow cannot move another workflow's cursor forward. A deploy, worker timeout, or temporary database pause therefore does not depend on an in-memory timer surviving.

When scheduled processing resumes, CE Pro continues from the saved cursor and catches up in bounded batches. Before an older event is allowed to run, the queue checks the current source record. A reminder is cancelled when it is no longer valid because, for example, the appointment was rescheduled, the source was resolved, the estimate was accepted, or the workflow was archived or changed to a different trigger. A temporary scheduler pause retains its queued work; an explicitly disabled workflow follows the active-run rule below.

Acceptance and signature evidence is authoritative even on an older estimate whose status label still says Draft, Sent, or Viewed. CE Pro does not change that estimate to Expired or send an expiring/expired workflow. Other genuinely mutable estimates continue through the normal bounded expiry update.

Each workflow and event combination has a stable dedupe key. Concurrent workers cannot claim the same event, and a retry cannot create another run for the same key. Worker failures use backoff; an expired worker claim becomes available again, and an event that exhausts its attempts becomes a monitored dead letter instead of disappearing.

A due customer-contact event more than 24 hours outside its safe automatic replay window becomes a monitored dead letter rather than sending unexpectedly after a long outage. Its durable record remains available for review.

The scheduled queue exists before a Workflow Run appears. A due event that is waiting for retry or catch-up will not yet show in the workflow's Run History. Operators should check the scheduled-queue metrics when the expected run is absent, especially the pending count, oldest pending time, recent retried total, and dead-letter count.

Record-Event Recovery

Record-driven events such as estimate sent or accepted, job status changed or completed, inbound replies and calls, reviews, and supported client events are admitted to a durable business-event queue in the same database transaction as the source change. A serverless timeout after the source save therefore leaves repairable work instead of a silently lost trigger. The normal low-latency attempt and the recovery worker use the same workflow-and-occurrence key, so they settle one run.

Business events have their own minute worker, claim capacity, lease, metrics, and scoped incident switch. Pausing only the time-based scheduled-trigger worker does not pause this lane. Before dispatch, the worker checks the saved event against its authoritative source, retries bounded transient failures, and quarantines work that is too old or ambiguous rather than surprising a customer with an unsafe replay. A pending or retrying business event more than 24 hours old becomes a monitored dead letter instead of starting its workflow automatically after a long outage.

The retained V1 engine admits invoice-paid and payment-failed events to this business-event lane. Current V2 does not; its lifetime-first paid product event and email/SMS delivery attempts are separate durable records and are not represented as Workflow Runs.

Inbound automation webhooks are accepted only with a live scoped API key and a stable Idempotency-Key header or event_id. CE Pro stores the bounded event payload and queues matching workflows before returning an accepted response; repeating the same key with the same payload returns the original admission instead of creating a second run. Reusing it with different content returns a conflict.

What a Scheduler Pause Does

A scheduler pause delays processing; it does not turn active workflows off, delete enrollments, or discard queued events. When the database circuit breaker opens, the protected high-fanout workers stop claiming work so interactive app traffic gets priority. The worker resumes after the circuit clears and then drains its backlog conservatively.

Pausing an individual workflow preserves that workflow's queued events and cursor, but it does not let an active run continue. When execution sees that a workflow is disabled or no longer active, it cancels the run and its pending or running steps, including a waiting delay, with an automation-disabled reason. Reactivating the workflow allows eligible queued work to be processed from its own saved position; a canceled run does not wake again. Archived workflows and workflows whose trigger type changed have their pending scheduled events cancelled.

The scope matters:

• The scheduled-trigger-only switch pauses the V1 appointment, estimate,

contract, and invoice timing worker where that engine is still in use. By itself, it does not stop manual enrollment, the independent durable business-event worker, delayed-run processing, or the legacy follow-up worker.

• The automatic database circuit and the global scheduled-work switches pause

the broader protected worker set, including scheduled triggers, delayed automation, follow-ups, background jobs, drip campaigns, and webhook delivery.

• An eligible inbound lead from a connector, web form, or server API can still

make its immediate speed-to-lead attempt when only the scheduled-trigger worker is paused. Its durable recovery job waits whenever the background-job worker is also paused, then retries after recovery. A lead created manually in the mobile app does not automatically start that outreach.

This recovery design reduces missed and duplicate work, but it does not promise that an external SMS, email, voice, or webhook provider will never have an ambiguous delivery. CE Pro does not blindly replay a provider step that was already marked as executing when acknowledgement was lost.

Waiting-Run Lease Recovery

A delay or business-hours hold normally resumes the same Workflow Run when its wake time arrives. The worker claims the run with a lease so overlapping worker invocations cannot resume it together. If a serverless worker stops after the claim, a recent run with no executing step becomes eligible for recovery after the five-minute lease window. If the workflow is disabled before the wake, the run is canceled with its waiting step instead of being resumed.

Automatic recovery deliberately stops at uncertain boundaries. A stale run is changed to Paused and receives a run_recovery_quarantined audit event when it has a step still marked Executing, because a provider may already have accepted that action. A stale Running run older than the 24-hour safe replay window is also paused for review. These runs are not silently completed or automatically re-sent; review the step and provider history before deciding what to do next.

New-Lead Speed-to-Lead Safety Net

For an eligible connector, web-form, or server-API lead with automation side effects enabled, CE Pro commits a deduplicated automation recovery job in the same database transaction as the lead. That transaction records the exact active workflow IDs or exact deliverable legacy sequence selected for the lead. The normal inline path and recovery worker use that same immutable route and stable event key, so a later settings change cannot switch execution modes. If no route is deliverable, the lead is still captured with automation disabled instead of reporting a queued contact that cannot run. Manual mobile lead creation intentionally suppresses this automatic customer outreach, including during the direct database fallback.

The same lead cannot have two active or paused enrollments in the same speed-to-lead sequence. The configured legacy sequence is the sole legacy contact path; CE Pro does not also send a separate hard-coded text or auto-dial. A crashed replay-safe recovery worker's lease can be reclaimed within 24 hours; after the final allowed attempt, the job moves to the background-job dead-letter state for investigation. An ambiguous stale generic messaging job is quarantined instead of automatically replayed. This safety net protects workflow dispatch and enrollment. Provider acceptance still remains subject to the provider's own availability and delivery guarantees.

Before a workflow email, text, or call reaches its provider, the engine creates a durable step reservation. A second worker cannot reserve the same active step for another send. If the provider may have accepted an action but its outcome cannot be persisted, the run stays visible for quarantine and review; the queue does not acknowledge it as complete.

Legacy Follow-Up Delivery Retries

An email, SMS, or Both step advances only after every channel requested by that step has a successful send record — the provider accepted it. That is not a delivery confirmation, and for SMS no later confirmation exists. If email succeeds and SMS fails, the enrollment remains on the same step. Its next attempt reads the existing email success and sends only SMS. Follow-up email also carries a stable enrollment, step, and channel idempotency key to the email provider, protecting the same partial-delivery retry if the worker lost its local acknowledgement.

Before calling either provider, the worker records an in-flight channel attempt. If it cannot read delivery history or reserve that attempt, it contacts no provider and backs the enrollment off; the fifth consecutive persistence failure pauses it with an infrastructure dead-letter reason. If SMS is accepted but the final delivery-log update fails, the in-flight record is treated as ambiguous and the enrollment pauses instead of sending the text again. Email can reconcile the same in-flight attempt with its stable provider idempotency key.

Retryable delivery failures use bounded backoff. The first four failures for the still-failing channel retry after approximately 5, 10, 20, and 40 minutes. The fifth pauses the enrollment with a delivery_dead_letter reason. A missing required recipient, non-retryable provider rejection, or unsupported delivery channel pauses immediately with a permanent-failure reason instead of silently moving to the next step.

Every attempt remains in the delivery log with its channel, status, provider reference when available, and failure classification. Correct the recipient or provider configuration before starting a replacement enrollment. Previously successful channels remain settled for the original enrollment and step, so resuming that exact record does not intentionally send them again.

Reliability Alerts and Daily Checks

When a scheduled or business event, background automation job, or legacy follow-up delivery first reaches its durable dead-letter or delivery-paused state, CE Pro attempts an immediate email to the configured maintenance or super-admin address. The stable record ID is also the email-provider idempotency key, so concurrent workers and a lost response do not intentionally create another alert for the same transition.

The durable queue, enrollment, and audit records remain the source of truth if the alert provider is unavailable. The daily maintenance run separately checks recent dead letters, overdue scheduled events and waiting runs, stale worker leases, ambiguous executing steps, quarantined runs, overdue follow-ups, and delivery-paused enrollments. A high-severity result is included in the daily maintenance alert email. An immediate email or a successful worker HTTP response is therefore not proof that every backlog is clear.

Viewing Runs

From a Lead or Client

Open a lead and review the Automations card beside its summary, or open a client and select the Automations tab. Each active or historical run shows:

• The automation name and current status
• The last step that ran
• The next step and its scheduled time
• The team member who manually enrolled the contact and when
• A collapsible list of completed or failed steps

Use Refresh automations after changing a workflow or manually enrolling a contact. Waiting means the run is honoring a delay or business-hours rule; it does not mean the automation failed.

In the Mobile App

Open a lead or client and choose Automations. The mobile view shows the same active and historical runs, last completed step, next step, scheduled time, completed-step history, and enrollment actor as the web workspace.

Team members with Automations manage can tap Enroll, choose an active Manual Trigger automation, and start it without leaving the contact. The same server checks block SMS opt-outs, missing required email or phone information, and duplicate active enrollment. The app refreshes the run immediately after a successful enrollment so the next action is visible.

From the Workflow Detail Page

1. Go to Automations and click on a workflow name
2. Click the Run History tab

You will see a table of all runs for that workflow. Each row shows:

• Contact name -- who the workflow ran for
• Started -- when the run began (relative time like "2h ago")
• Status -- Running, Waiting, Completed, Failed, or Cancelled
• Current Step -- which step the run is on (for active runs)
• Duration -- total elapsed time

From the Global Runs Page

To see runs across all workflows, use the API at /api/automations/runs. The workflow detail page is the primary UI for monitoring.

Run Statuses

When a delay expires or a business-hours hold clears, CleanEstimate Pro promotes that same run back to Running automatically before the next step executes. Scheduled wake-ups support Vercel's cron request method, and the engine checks the stored wake time before resuming so a call cannot skip ahead of a delay or business-hours hold.

Queue pending, retry, processing, and dead states are pre-run delivery states, not Workflow Run statuses. Once a queued event successfully starts or reconciles its workflow run, the table above applies.

Expanding a Run

Click on any run row to expand it and see the step-by-step execution timeline. Each step shows:

• Step icon and label -- what type of action it was
• Status -- completed, failed, skipped, or pending
• Timestamp -- when the step executed
• Duration -- how long the step took
• Output -- for communication steps, a preview of the message that was sent

For failed steps, a red error box shows the error message. For condition steps, the result shows which branch was taken (TRUE or FALSE). If any step fails, the run finishes with a Failed status so your run history and success-rate metrics stay accurate.

Test Runs

Before activating a workflow, use the Test Run button on the workflow detail page. This opens a modal where you can:

1. Enter mock contact data (name, email, phone, estimate total, etc.)
2. Click Run Test
3. See a simulation of every step without actually sending messages

The test results show:

• Whether entry conditions pass or fail
• Each step's rendered output with your mock data filled in
• Delay durations and computed wake times
• Condition evaluation results

This is the safest way to verify your workflow before it goes live.

Cancelling a Run

To cancel an active run:

1. Expand the run in the Run History tab
2. Click the Cancel button

This immediately stops the run. Any pending delay steps are abandoned, and no further messages are sent. The associated enrollment is also completed.

Cancellation is useful when a customer contacts you directly and you want to stop the automated sequence.

Workflow Overview Stats

The Overview tab on the workflow detail page shows aggregate metrics:

• Total Runs -- how many times the workflow has executed
• Completed -- successful completions
• Failed -- runs that encountered errors
• Success Rate -- percentage of runs that completed without errors
• Last Run -- when the most recent run started

A bar chart shows runs per day over the last 30 days, with green bars for completed and red bars for failed runs.

Troubleshooting Failures

When a run fails:

1. Expand the run to find the failed step (marked with a red X)
2. Read the error message -- common issues include:

• SMS opt-out -- the customer opted out, so the SMS was skipped
• Rate limited -- too many messages sent in a short period
• Webhook timeout -- an external endpoint did not respond in time
• Missing data -- a required field like phone number was not available

1. Fix the underlying issue (update the contact, check integrations)
2. If needed, manually enroll the contact again to restart the workflow

If a run cannot initialize, enrollment now reports the failure instead of showing a false success. The run is marked Failed, its active enrollment is closed, and the error appears on the lead or client automation panel so the contact can be enrolled again after the configuration is corrected.

If no run exists for an expected scheduled action, check the scheduled queue before manually enrolling the contact. The event may still be pending or waiting for a retry after a circuit-breaker pause. Manually starting another workflow while a durable event is pending can create a separate intentional run with different enrollment history.

If the failed step was a webhook, double-check that the destination is a public HTTPS endpoint. CE Pro blocks localhost, private-network, and other internal-only targets.

Tips

• Check run history after activating a new workflow to confirm it is behaving as expected.
• Use the 30-day chart to spot trends. A sudden increase in failures may indicate an integration issue.
• A scheduled event can retry before it settles its workflow run. A Workflow

Run already marked Failed does not automatically restart from the beginning; investigate its step history before re-enrolling the contact.