Task Scheduler: Framing, Scale Math, and Architecture

This lesson opens Chapter 7: Distributed Task Scheduler & Job Orchestrator (Production System Blueprint 4: Reliable Delayed Execution, Fault-Tolerant DAG Workflows, and Fenced Distributed State). It covers sections 1 to 3 of the chapter.

At a glance

1. The Scenario & Problem Framing

PROMPT:

"Design a globally distributed task scheduling and workflow orchestration platform (like Temporal, Celery, or Quartz) capable of processing 100,000,000 tasks per day. Support arbitrary delayed execution (run at timestamp T, from 1 second to 30 days in the future), recurring cron jobs, and complex directed acyclic graph (DAG) workflows. Guarantee at-least-once execution, sub-second schedule precision, robust isolation across multi-tenant workloads, and complete resilience against worker crashes, network partitions, and silent coordinator failures."

In typical request-reply microservices (e.g., viewing a profile or adding an item to a cart), failure is immediate and loud. If a service crashes, the client receives an HTTP 500 error, retry policies activate, error budgets drop, and PagerDuty alerts fire within seconds.

In a distributed task scheduler, the most dangerous failure mode is complete, deafening silence.

When a scheduler node crashes or deadlocks due to a silent thread pool exhaustion:

1. No HTTP 500 errors are returned to users.
2. No database query timeouts are thrown.
3. Existing tasks simply sit in storage queues, un-dispatched and un-executed.
4. The system produces zero alerts while millions of reminder emails, loan interest calculations, subscription renewals, and data pipeline ETL runs silently miss their execution windows.

Teams typically discover a scheduler outage hours—or days—later, when customer support is overwhelmed by users complaining that their accounts were never billed, or regulators demand to know why compliance filings were not executed at midnight.

Senior engineers treat task scheduling as a simple database polling problem: insert a row into a tasks table with a run_at timestamp, and run a background worker that executes:

SELECT * FROM tasks
WHERE run_at <= NOW() AND status = 'PENDING'
ORDER BY run_at ASC
LIMIT 100
FOR UPDATE SKIP LOCKED;

To an L6 Staff evaluator, this naive implementation fails immediately at scale:

1. B-Tree Index Lock Thrashing: Polling a table with 100,000,000 rows every 500 milliseconds requires constant B-Tree index traversal and write lock acquisition. At 10,000 tasks per second, database CPU sits at 100% on lock contention alone.
2. The Zombie Worker Problem: A worker claims a task, executes for 30 seconds, encounters a 40-second JVM Stop-the-World garbage collection pause or hypervisor stall, and loses its lease. The scheduler declares the worker dead and reassigns the task to Worker B. When Worker A wakes up, both workers execute the same side effect simultaneously, causing duplicate credit card charges or corrupted database records.
3. The Midnight Thundering Herd: In enterprise environments, 40% of all scheduled cron jobs are naively configured for 00:00:00 UTC. At midnight, a thundering herd of 10,000,000 tasks becomes ready simultaneously, crushing the queue broker and starving urgent real-time tasks.

Staff engineers treat distributed task scheduling as a multi-tiered time-indexing, admission control, and distributed consensus problem. You do not poll a single relational table. You decouple scheduling (when), dispatching (who), execution (how), and state tracking (did it run) into independent, fault-isolated subsystems protected by monotonic fencing tokens and proactive canary heartbeats.

flowchart TB
    P["THE CORE PARADOX<br/><br/>A task scheduler must guarantee that every task executes at the right time, even when workers crash mid-execution, networks partition, and scheduler nodes die silently without throwing an exception.<br/><br/>Staff Goal: Decouple cold long-term storage from hot in-memory timing tiers, enforcing monotonic fencing tokens so zombie workers can never corrupt state, and deploying active canary watchdogs that detect scheduler death in < 120s."]

The L5 vs L6 Contrast — What the Evaluator Actually Hears

The Senior Candidate (L5): Immediately proposes inserting rows into a relational tasks table with a status and run_at timestamp, and running a worker loop with SELECT ... FOR UPDATE SKIP LOCKED every 500ms. When asked, "What happens when 500 workers poll a 100M-row table continuously?", the candidate suggests adding a larger Redis cache. When asked, "What happens if a worker pauses for 45 seconds during a garbage collection cycle and its lease expires?", they suggest increasing the lease timeout. They fail to recognize that increasing timeouts just delays failure detection and still leads to zombie workers executing duplicate side effects.

The Staff Candidate (L6): Separates the system into four distinct, fault-isolated domains within the first 60 seconds: "A task scheduler must decouple scheduling (when), dispatching (who), execution (how), and audit state tracking (did it run). Polling a single database table at 10,000 tasks/second will destroy database I/O. We use a two-tier storage model: durable cold tasks live in time-partitioned PostgreSQL, while a window loader hydrates tasks due in the next 60 minutes into sharded Redis Sorted Sets. To eliminate zombie worker split-brain, every dispatched task receives a monotonic fencing token validated at the database write boundary. Finally, because the most dangerous failure mode is silent deadlocks, we deploy an active synthetic canary task executing every 60 seconds."

2. Step 1: Commit to Intent & Scale Math

Before proposing technology stacks (Redis, Kafka, Postgres, etcd), an L6 Staff engineer establishes the core operational intent. The phrase "task scheduler" hides three conflicting operational models:

The Three Incompatible Intents

1. Reliability-First (Financial Reminders & Regulatory Tasks):

The Philosophy: Every single task must execute effectively once. Dropping a task is a critical compliance violation; executing a task twice results in duplicate charges and lawsuits.

The Architectural Commit: Persistent transactional task store, durable completion outbox, distributed fencing tokens, and a 15-minute cross-referencing watchdog daemon.

2. Latency-Sensitive (Real-Time Timeouts & Gaming Sessions):

The Philosophy: Tasks must trigger within milliseconds of target timestamp T (e.g., connection keep-alives, fraud detection evaluation windows, interactive user timeouts).

The Architectural Commit: In-memory Hierarchical Timing Wheels (Kafka/Netty style) running inside worker memory, direct worker memory dispatch, and volatile storage.

3. Throughput-Optimized (Batch ETL & Media Transcoding):

The Philosophy: Process tens of millions of tasks per hour with maximum hardware efficiency. Individual execution delay (e.g., running 10 seconds late) is completely acceptable as long as overall pipeline throughput remains saturated.

The Architectural Commit: Batched Kafka queues, dynamic worker auto-scaling, work-stealing algorithms, and coarse-grained time partitioning.

STAFF MOVE

Senior: "We will use an AWS SQS queue with delayed delivery and configure an auto-scaling ECS worker pool."

Staff: "Before choosing a queue primitive, are these tasks idempotent? SQS delayed delivery caps at 15 minutes, provides no schedule cancellation or inspection API, and guarantees only at-least-once delivery. If tasks involve financial debits or third-party webhooks, at-least-once without platform-level idempotency and distributed fencing tokens will cause duplicate transactions whenever a worker crashes post-execution. I will design a two-tier storage system: cold long-term tasks live in time-partitioned PostgreSQL, a window loader hydrates tasks due in the next 60 minutes into sharded Redis Sorted Sets, and workers enforce monotonic fencing tokens to structurally prevent zombie writes."

Why: Jumping to SQS ignores the fundamental constraints of task scheduling: long-delay lookaheads (>15m), task cancellation, priority inversion, and zombie worker state corruption.

Capacity Planning & Scale Math

In an L6 interview, you must derive your storage tiers, memory footprints, and worker pool sizing from first principles.

SCALE REQUIREMENTS

• Total Daily Volume: 100,000,000 tasks per day.
• Average Throughput: 100,000,000 / 86,400 ≈ 1,157 tasks/second.
• Peak Throughput (2.5x Burst + Midnight Cron Surge): 10,000 tasks/second.
• Delay Range: 1 second to 30 days in the future.
• Average Task Execution Duration: 500 milliseconds.
• Target Scheduling Precision: Within 1 second of scheduled timestamp T.
• State Retention: 30 days of execution history for auditing.

SIZING STORAGE: COLD LONG-TERM STORAGE (SYSTEM OF RECORD)

Payload per Task Record: 500 bytes (UUID, tenant ID, scheduled timestamp, status, retry count, payload JSON, fencing token).

Active Future Tasks (at any point): Assuming 2 days of average forward pipeline:

Active Future Records ≈ 200,000,000 tasks * 500 bytes = 100 GB

Completed Task Audit Log (30-day retention):

• Daily Volume: With B-Trees: 100,000,000 tasks/day * 30 days = 3,000,000,000 records (3 Billion rows!)
• Raw Data Sizing: 3 Billion records * 500 bytes = 1.5 Terabytes raw data
• With B-Tree indexes (2x overhead) ≈ 3.0 Terabytes

Storage Strategy: A single relational database cannot handle 10,000 continuous writes/sec while storing 3 Billion rows. We partition the relational store into 1-day physical table partitions based on scheduled_at. At the end of each day, completed task partitions are archived to S3 Parquet format for Athena querying, keeping live database disk usage under 200 GB!

SIZING THE HOT TIMING TIER (REDIS SORTED SETS)

The Hot Tier only holds tasks scheduled to run within the next 60 minutes:

Hot Window Volume = 1,157 tasks/sec * 3,600 seconds ≈ 4,165,200 tasks

In Redis, each task in a Sorted Set (ZSET) requires ~128 bytes (Member = Task UUID string, Score = 64-bit millisecond epoch).

Redis Memory Footprint = 4,165,200 * 128 bytes ≈ 533 Megabytes!

Result: The entire hot scheduling window across the entire planet requires less than 600 MB of RAM! Sharding across 16 Redis nodes provides over 100x headroom and guarantees that ZPOPMIN execution latency remains under 0.5 milliseconds.

SIZING THE WORKER FLEET (LITTLE'S LAW)

Using Little's Law (L = λ * W):

In-Flight Concurrent Tasks = Arrival Rate (λ) * Average Duration (W)
Peak In-Flight Tasks = 10,000 tasks/sec * 0.5 seconds = 5,000 concurrent tasks

Assuming each containerized worker process safely handles 10 concurrent threads without CPU thrashing:

Worker Fleet Required = 5,000 concurrent tasks / (10 threads/worker) = 500 worker containers

Sizing for 70% target utilization with a 2x peak safety margin requires 1,000 worker instances distributed across 3 Availability Zones.

KEY SYSTEM SIZING NUMBERS WORTH MEMORIZING

In a Staff interview, having pre-calculated operational metrics memorized demonstrates authentic production experience:

3. End-to-End System Architecture

The architecture decouples the slow, durable long-term storage from the lightning-fast, in-memory dispatch engine:

Figure 7.1: End-to-End Distributed Task Scheduler & Orchestrator Architecture

The Architecture Components in Depth

flowchart TB
    subgraph TITLE["DISTRIBUTED FAULT-TOLERANT TASK SCHEDULER ARCHITECTURE"]
        Client["Client Applications / Microservices"]
        C1["COMPONENT 1: Task Ingestion & Admission<br/>• UUIDv7 Time-Sortable Task Allocation<br/>• Tenant Token Bucket Rate Limiting<br/>• Immediate vs. Delayed Split Router"]
        C7["COMPONENT 7: Canary Watchdog & Dead Man's Switch<br/>• Independent AWS Lambda Host<br/>• 60s Synthetic Probe (120s)<br/>• Silent Deadlock Detection"]
        Canary["Synthetic Canary Watchdog<br/>(Continuous Canary)"]
        C2["COMPONENT 2: Cold Storage & Pre-Loader Fleet<br/>• Partitioned PostgreSQL Cluster (By Day)<br/>• Pre-Loader Daemons (etcd Lease per Shard)<br/>• Sweeps [NOW, NOW+60m] into Redis ZSETs"]
        C3["COMPONENT 3: Hot Delay Buffer<br/>• 16-Node Redis ZSET Primary Cluster<br/>• Score = Timestamp, Member = Task ID<br/>• Memory Footprint < 600 MB Cluster"]
        C4["COMPONENT 4: High-Precision Tick Dispatcher Fleet (Go / Rust Worker Pool)<br/>• etcd Shard Leadership Leases (10s TTL Heartbeat Renewal)<br/>• Monotonic Fencing Token Allocation (FetchAndAdd(token))<br/>• Execution Envelope Stamping & Overdue SLA Validation"]
        C5["COMPONENT 5: Distributed Message Broker (Kafka / RabbitMQ Decoupling Tier)<br/>• tasks.critical (500 Dedicated Workers, p99 < 500ms)<br/>• tasks.standard (350 Preemptible Workers, p99 < 5s)<br/>• tasks.bulk (150 Yielding Workers, p99 < 15m)"]
        C6["COMPONENT 6: Polyglot Worker Execution Fleet & Heartbeat Lease Monitor<br/>• Multi-AZ Container Pods (Go, Python, Java, Node.js Runtimes)<br/>• Active Execution Heartbeat Renewal in etcd/Redis (30s Lease TTL)<br/>• Monotonic Fencing Token Verification on Result Write (Zombie Worker Rejection)"]
        Client -->|"gRPC Ingress / Task Submission"| C1
        Canary --> C7
        C1 -->|"Due in > 60 Minutes"| C2
        C1 -->|"Due in <= 60 Minutes"| C3
        C2 --> C3
        C3 -->|"ZPOPMIN on 100ms Tick"| C4
        C4 -->|"Priority Partitioned Publish"| C5
        C5 -->|"Pull Ready Tasks & Establish Leases"| C6
    end

Component 1: Task Ingestion API & Admission Gateway

Wire Protocols & Contract Validation: Exposes high-throughput gRPC and HTTP/3 endpoints fronted by Envoy ingress proxies. Validates incoming task specifications: cron expressions, one-shot delayed timestamps, execution timeouts, maximum retry budgets, and binary/JSON payload payloads.

Idempotent Admission & UUIDv7 Identification: Requires clients to provide an idempotency_key (or derives one from payload attributes). Assigns each task a 128-bit time-sortable UUIDv7 identifier (timestamp_ms + rand_bits), ensuring chronological clustered index insertion in PostgreSQL and eliminating B-tree page split fragmentation.

Tenant Admission Control: Enforces per-tenant rate limits via local in-memory token buckets (e.g., maximum 5,000 tasks/second per tenant). Quota breaches return HTTP 429 Too Many Requests with a dynamic Retry-After header. Writes confirmed tasks into PostgreSQL cold storage; if the task execution time is ≤ 60 minutes, it simultaneously hydrates the task directly into the Redis Hot Delay Buffer.

Component 2: Cold Task Storage Tier & Lookahead Pre-Loader Fleet

Partitioned Relational Persistence: A sharded PostgreSQL database cluster partitioned by scheduled execution day (tasks_y2026m09d04). Stores the durable system of record for all future scheduled work (days, weeks, or months in advance) with strict ACID transactional guarantees.

Distributed Range Pre-Loader Daemons: A horizontally scalable fleet of stateless background daemons running every 10 minutes. Each daemon acquires an exclusive distributed range lease on a specific partition shard via etcd (lease:preloader:<shard_id> with 30s TTL).

Sliding Window Hydration: Sweeps tasks scheduled for execution within the upcoming window [NOW, NOW + 60 minutes]. Daemons stream task IDs and execution timestamps into the Redis Hot Delay Buffer using pipelined ZADD commands, marking tasks as status = 'BUFFERED' in PostgreSQL. This lookahead tier shields the primary relational database from high-frequency, second-by-second polling.

Component 3: Hot Delay Buffer (Redis ZSET Cluster)

Sharded Timing Architecture: A distributed in-memory timing buffer partitioned across 16 Redis primary nodes using consistent hashing: hash(tenant_id, task_id) % 16.

Data Structures & Memory Footprint: Tasks are indexed in Redis Sorted Sets (ZSET) where member = task_id and score = execution_timestamp_ms. Because each record requires only ~128 bytes, the entire planet's 60-minute hot scheduling window (~4,165,000 tasks) fits within less than 600 Megabytes of RAM, delivering ultra-low memory overhead and sub-millisecond query latencies.

Atomic Batch Dequeue: Dispatchers retrieve ready tasks using atomic ZPOPMIN commands executed via Lua scripts, popping all elements whose score <= current_epoch_ms in O(log N + M) time without contention from concurrent producers.

Component 4: High-Precision Tick Dispatcher Fleet & Fencing Coordinator

Precision Tick Loop: A pool of leader-elected Go/Rust dispatcher processes running a continuous 100-millisecond tick loop (time.NewTicker(100 * time.Millisecond)). Each dispatcher maintains exclusive ownership of a subset of Redis shards via active etcd leases renewed every 10 seconds.

Monotonic Fencing Token Allocation: When popping due tasks from Redis, the dispatcher generates a monotonically increasing 64-bit fencing token:

FencingToken = FetchAndAdd(task_fencing_token)

The execution envelope is stamped with (task_id, fencing_token, dispatch_epoch, attempt_number).

Overdue SLA & Backpressure Check: If a dispatcher stalls (e.g., during network jitter or GC pause) and pops a task past its scheduled time, it computes lateness = NOW() - scheduled_time. If lateness exceeds the tenant's SLA_grace window, the task is diverted to an overdue alert queue; otherwise, it is dispatched to the message broker.

Component 5: Distributed Message Broker (Kafka / RabbitMQ)

Execution Decoupling: Decouples the high-precision temporal dispatch loop from variable-latency worker execution. If worker nodes experience temporary saturation or third-party downstream timeouts, the broker buffers in-flight tasks without dropping requests or causing dispatcher tick drift.

Priority-Segregated Execution Channels: Routes tasks into three physically isolated priority queues:

1. tasks.critical: High-priority operational workflows (payment capture, 2FA codes, account provisioning). Allocated a dedicated reserve of 500 worker instances (p99 latency < 500ms).
2. tasks.standard: Standard asynchronous jobs (customer invoices, transactional emails, webhooks). Allocated 350 preemptible workers (p99 latency < 5s).
3. tasks.bulk: Heavy batch processing (nightly reporting, analytics ingestion, database vacuuming). Allocated 150 yielding workers (p99 latency < 15m).

Tenant Fair-Share Sharding: Message partitions are keyed by tenant_id, guaranteeing that an aggressive burst of 100,000 tasks from Tenant A cannot starve the worker queues of Tenant B.

Component 6: Polyglot Worker Execution Fleet & Heartbeat Lease Monitor

Isolated Container Runtimes: A horizontally autoscaling fleet of worker pods running across multi-AZ Kubernetes clusters. Workers subscribe to broker queues, deserialize task arguments, and execute customer payloads across polyglot runtimes (Go, Python, Java, Node.js).

Active Execution Heartbeat Lease: Upon task pickup, the worker establishes a dynamic lease in etcd or Redis: SET lease:task:<task_id> <worker_id> EX 30. While processing, a background goroutine/daemon refreshes the lease every 10 seconds.

Fencing Token Verification on Commit: Upon successful task completion, the worker records the result in PostgreSQL using an atomic conditional update:

UPDATE tasks SET status = 'COMPLETED', result = :res, completed_at = NOW()
WHERE id = :task_id AND fencing_token = :fencing_token;

If the worker suffered an extended network partition or GC pause causing its 30-second lease to expire, another worker will have claimed the task with a higher fencing token. The stale worker's commit matches 0 rows and fails safely, preventing silent state corruption.

Component 7: Synthetic Canary Watchdog, Dead Man's Switch & Alerting Engine

Out-of-Band Isolated Probing: An autonomous watchdog service running completely outside the primary scheduler cluster (deployed in an independent cloud region or AWS Lambda).

Continuous Synthetic Health Verification: Submits a synthetic canary task every 60 seconds scheduled to execute in exactly 5 seconds. If the canary task fails to report completion within 120 seconds, the watchdog immediately triggers a P0 PagerDuty emergency alert, detecting silent scheduler hangs before customer jobs are impacted.

The Dead Man's Switch for Periodic Jobs: Protects high-value cron jobs (e.g., nightly compliance ledger settlement). Monitors the completed audit ledger; if an expected job fails to register a completion signal within interval + max_tolerance, the switch trips immediately, alerting on the absence of a success signal rather than waiting for nonexistent error logs.

Step-by-Step Task Scheduling & Execution Flow

The following sequence details the complete lifecycle of a task from initial admission to lookahead pre-loading, 100ms tick dispatch, worker execution, and fenced state commitment:

sequenceDiagram
    participant Client as Client API
    participant Ingest as Ingestion API
    participant Cold as Cold Storage
    participant Redis as Redis ZSET
    participant Disp as Dispatcher
    participant Broker as Message Broker
    participant Workers as Worker Fleet
    Client->>Ingest: (1) Submit Task (due in 45m)
    Ingest->>Cold: (2) Write DB (Partition)
    Ingest->>Redis: Hydrate ZSET (due <= 60m)
    Redis-->>Disp: (3) Tick 100ms ZPOPMIN
    Disp-->>Redis: Return Batch
    Disp->>Broker: (4) Stamp Fencing Token
    Workers->>Broker: (5) Pull Task (Ready Queue)
    Workers->>Workers: (6) Acquire Lease (etcd 30s TTL)
    Workers->>Workers: (7) Execute Job Payload Logic
    Workers->>Cold: (8) Commit Fenced Result to DB<br/>UPDATE tasks SET status='COMPLETED' WHERE token = :fencing_token

STEP 1: TASK SUBMISSION & ADMISSION CONTROL

A client microservice issues an RPC to submit a scheduled task: POST /api/v1/tasks with idempotency_key = "billing_sub_9841", scheduled_at = "2026-09-04T10:45:00Z" (45 minutes in the future), and execution payload arguments. The Ingestion API validates payload syntax, checks tenant token buckets for rate limiting, and assigns a time-sortable UUIDv7 identifier (0191b2c4-89a1-7000-8000-a1b2c3d4e5f6).

STEP 2: RELATIONAL PERSISTENCE & HOT TIER HYDRATION

The Ingestion API inserts the record into PostgreSQL:

INSERT INTO tasks_y2026m09d04 (id, tenant_id, scheduled_at, status, payload)
VALUES ('0191b2c4...', 'tenant_acme', '2026-09-04 10:45:00', 'SCHEDULED', '{"account": 42}');

Because the scheduled execution time is within the next 60 minutes (≤ NOW + 60m), the ingestion worker immediately computes the target Redis shard (hash("tenant_acme", "0191b2c4...") % 16) and executes a pipelined ZADD:

ZADD sched:shard:4 1788489900000 "0191b2c4..."

If the task was scheduled for 30 days in the future, the Redis write is skipped; the future task remains safely in PostgreSQL until the Cold Storage Pre-Loader Fleet sweeps it into Redis 60 minutes prior to execution.

STEP 3: HIGH-PRECISION 100MS TICK & BATCH DEQUEUE

Every 100 milliseconds, the leader-elected Dispatcher node for Shard 4 executes a tick. It queries Redis for all due tasks:

-- Atomic ZPOPMIN for due tasks
local now = tonumber(ARGV[1])
local ready = redis.call('ZRANGEBYSCORE', KEYS[1], '-inf', now, 'LIMIT', 0, 100)
if #ready > 0 then
  redis.call('ZREM', KEYS[1], unpack(ready))
  return ready
end
return {}

Redis atomically removes and returns the batch of ready task IDs in < 0.4 milliseconds.

STEP 4: MONOTONIC FENCING TOKEN ALLOCATION & DISPATCH ENQUEUE

The Dispatcher stamps the task with a monotonic 64-bit fencing token allocated from etcd (token = 1048201). It wraps the task into an execution envelope:

{
  "task_id": "0191b2c4...",
  "tenant_id": "tenant_acme",
  "fencing_token": 1048201,
  "attempt": 1,
  "payload": {"account": 42}
}

The dispatcher publishes the envelope to the message broker on the priority-isolated topic tasks.critical.

STEP 5: WORKER PICKUP & PRIORITY QUEUE CONSUMPTION

An idle worker process in the containerized execution fleet pulls the task from tasks.critical. Because tasks.critical maintains a dedicated pool of 500 reserve workers, the queue wait time is under 12 milliseconds.

STEP 6: DYNAMIC HEARTBEAT LEASE ACQUISITION

Before running the user's business logic, the worker claims an active execution lease in Redis or etcd:

SET lease:task:0191b2c4... "worker_pod_88" NX PX 30000

A background goroutine starts a 10-second ticker to periodically refresh the 30-second TTL while execution is in progress.

STEP 7: SANDBOXED TASK EXECUTION

The worker invokes the application payload in an isolated execution sandbox, enforcing strict CPU, memory, and wall-clock execution limits (e.g., 5.0-second execution deadline).

STEP 8: FENCED RESULT COMMITMENT & STATUS TRANSITION

The task completes successfully. The worker stops its heartbeat goroutine and executes an atomic fenced commit against PostgreSQL:

UPDATE tasks_y2026m09d04
SET status = 'COMPLETED', completed_at = NOW(), result = '{"status": "SUCCESS"}'
WHERE id = '0191b2c4...' AND fencing_token = 1048201;

Because the fencing token matches, the write succeeds. The worker deletes the lease key and acknowledges the message off the broker queue. Concurrently, the Synthetic Canary Watchdog records the system as fully healthy.