Scheduled Runbook Execution & Drift Monitoring

What it does

The Ops Script Schedules page (under the Configure nav group) lets you run library scripts on a recurrence - hourly, daily, weekly, or monthly. Each run is captured with full per-target output, and consecutive runs are compared for drift. When drift or failure is detected, configurable action rules fire email notifications or open Tasker tasks for follow-up.

Use cases

• Configuration drift detection - Run a "report current state" script daily; get alerted when output changes (registry key flipped, service stopped, group membership changed).
• Compliance evidence collection - Capture proof of control state on a recurring basis. Each run is a timestamped snapshot in OpsScheduleRuns with full stdout/stderr.
• Remediation drift watch - After remediating a finding, schedule a daily verification script. Get a Tasker task opened the moment the remediation reverts.
• Tenant-wide M365 enforcement - Cloud-target scripts (☁️) run against Graph/EXO from a runbook context, not on endpoints. Example: ensure mailbox Calendar default permission stays Reviewer (Example Company's first use case).

Script execution-target taxonomy ADO #498

Every library script is tagged with an executionTarget field that determines where and how it runs:

Why this matters: a cloud script attempted via the device-execute path will fail at runtime - there's no agent to receive it. The scheduler enforces the contract at save time: cloud scripts cannot have device targets, device/hybrid scripts must have at least one. Existing scripts default to device (matches prior behavior).

Cloud auth context (optional companion)

Cloud scripts can additionally declare cloudAuthContext to document what auth the runbook host must provide. Values:

• exo-managed-identity - Exchange Online cmdlets via Automation Account managed identity
• graph-managed-identity - Microsoft Graph via Automation Account managed identity
• exo-interactive - Exchange Online cmdlets via interactive admin sign-in
• graph-interactive - Microsoft Graph via interactive admin sign-in
• azure-managed-identity - Azure Management API via managed identity

Device dispatch preparation and retries

Manual and scheduled device/hybrid jobs first appear as Preparing. No target can run that job until preparation finishes. A failed or interrupted preparation stays blocked and is marked Failed; the job retains its target list for diagnosis. The existing watchdog checks abandoned preparations every 15 minutes. You can cancel or kill a Preparing job; Pause and Resume apply after it becomes active.

If a manual execution times out, keep the Run Script dialog open and retry there with the same targets and options. It checks the same request instead of sending a second batch. Closing the dialog, changing the options or starting another run creates a new request. Check the original job before deliberately rerunning a script with side effects. The job detail distinguishes a preparation failure from a command that ran and later failed.

API and MCP callers should supply and retain a retry key for device/hybrid execution: idempotencyKey in the REST body (or Idempotency-Key header), and idempotency_key for execute_script. Reuse it after an uncertain response. Keys contain 8–128 letters, digits, dots, colons, underscores or hyphens and belong to the signed-in actor, tenant and organization. Reusing one with different targets, script content/version, run-as or timeout is refused. Retry protection lasts while the original job record exists. Calls without a key are separate requests. Cloud webhook execution and separate manual schedule-run requests do not gain retry protection from this key.

Creating a schedule

1. Open TATER Ops → Configure → Script Schedules and click + New Schedule.
2. Pick a script. The dropdown shows the execution-target icon next to each script.
3. For device/hybrid scripts: enter target hostnames (one per line, lowercase). Cloud scripts leave this empty.
4. Set the frequency and choose a UTC time. Weekly schedules require a named weekday; monthly schedules require a day from 1 to 28. The editor shows the next three scheduled times in UTC and device-local time. Hourly schedules use the selected minute of each hour; their hour field does not restrict the run to one hour.
5. Add action rules. Trigger options: on-drift, on-failure, or always. Action options: email (with recipients) or tasker-task (with category + priority).
6. Save. The cron checks every 5 minutes and runs schedules whose nextRunAt has passed.

UTC remains fixed when local daylight-saving time changes. The preview shows scheduled times, not a guarantee of when a device or cloud job will start. Invalid times or missing required days keep the editor open and are refused by the API and MCP create/update paths. Existing one-time schedules retain their explicit UTC run time when edited and preview one occurrence.

API and MCP callers: supply a valid UTC time from 00:00 to 23:59; valid one-digit hours normalize to two digits. Omitting time retains the existing 00:00 default. Weekly recurrence requires an integer dayOfWeek from 0 (Sunday) to 6; monthly recurrence requires an integer dayOfMonth from 1 to 28. A replacement recurrence is validated in full. Other partial updates preserve the saved recurrence and next-run time. This change does not repair or reschedule older records; review the displayed cadence before editing them.

How drift detection works

When a run completes, the scheduler queries the most recent previous run for the same schedule and compares normalized stdout per target. Any host whose stdout differs from the prior run is flagged. The run record stores:

• driftDetected: true/false
• driftSummary.changedTargets - list of hostnames that changed
• driftSummary.sampleDiff - first ~2KB of difference (truncated)

When drift is found, run status rolls up to Drift (instead of Success) so it shows up amber in the runs list and triggers any on-drift action rules.

Drift detection precision

Phase 1 compares whitespace-normalized stdout text. Scripts that emit timestamps, GUIDs, or other always-changing data will appear to drift every run. Best practice: structure your script output so the stable signal (config values, group membership IDs, permission strings) is on its own lines, separated from the noisy metadata. Phase 2 will add JSON diff with per-property Before/After.

Action rules

Each schedule supports up to 20 action rules. They fire after a run finalizes, in order of trigger:

• on-drift - fires when driftDetected = true. Use to flag configuration changes for review.
• on-failure - fires when run status is Failed or Mixed. Use to escalate broken automation.
• always - fires on every completed run. Use for low-volume nightly status emails.

Email actions send via the configured notifications channel (Graph or SMTP). Tasker-task actions create a new task in TATER Ops under the supplied category and priority, with the run summary embedded in the task description and the originating schedule linked.

Recent runs & per-target output

A run that is still unfinished two hours after it started is stopped by a watchdog. Its status comes from what each target did: Success when every target completed, Mixed when some completed and some did not, Failed when none did. A target marked Cancelled had no claim recorded when the run stopped, so it has no result. A target marked Timeout was recorded as running and may have executed. TATER cannot see the endpoint, so verify there before re-dispatching anything that is not safe to repeat.

The Recent Runs tab lists the last 50 runs across all schedules. Click any row to open the run detail modal, which shows:

• Drift diff panel (amber) - sample of what changed between consecutive runs
• Per-target table - host, status, exit code, duration, and clipped stdout (2KB per host)
• Outcome line - how many targets completed, failed, were cancelled with no result, or timed out, and, when the watchdog stopped the run, the reason
• For Skipped runs (cloud scripts in Phase 1) - the reason the run was skipped

Cloud script execution

Cloud-targeted schedules execute. They fire the same per-org Azure Automation webhook (Run-OpsScriptCloud) that the on-demand Run button uses — one shared dispatch, so a scheduled run cannot behave differently from a manual one.

A cloud run stays Running until the runbook posts its result back, then finalises Success, Failed, or Drift from that result. It is deliberately NOT finalised the way device runs are: device runs are judged by counting agent commands, and a cloud job has none — so the device logic would see “nothing pending”, conclude the run was over before the runbook had started, and record Failed every time.

If the organisation has no Ops Script Cloud Webhook URL configured, the run records Failed with that as the reason. It is not skipped and it is not green.

Is this schedule actually running?

A green dot and a future next-run time are not evidence that anything ran. Every schedule now carries a health verdict, computed from its real run history rather than from its settings, and shown on the schedule list, the schedule detail, and list_ops_schedules over MCP.

Two things it deliberately does not count as an execution: a Skipped run (a skip is not a run) and a Running one (it has not finished). A schedule whose history is nothing but skips therefore reports stale — which is the honest answer, and the one the old surfaces could not give.

If the verdict cannot be computed, the schedule shows health unknown rather than nothing. Absent evidence must not read as good news.

Permissions

• Auditor+ - view schedules, view runs.
• Admin+ - create, edit, delete schedules; trigger ad-hoc runs.

Related

• Ops customization - categories, priorities, statuses, teams.
• Agent deployment - getting the TATER Agent installed on target devices.
• MCP setup - using AI agents to query schedule history and trigger runs.