Compensating Transactions with Step Functions
A saga-based order processing system on Step Functions implementing idempotent operations, bounded retries, compensating actions, manual repair procedures, failure testing, and a transactional outbox over DynamoDB Streams to close the write/publish gap.
Distributed orders fail halfway: payment taken but inventory unreserved, with no consistent way to roll forward or back.
Shows the patterns that make distributed transactions safe: each step has a compensating action, retries are bounded, and nothing is left half-done without a repair path.
Order steps run in sequence with a defined compensation per step.
Database writes and event publications stay consistent.
Unrecoverable states land in a repair queue with operator steps.
Orchestrated saga: forward steps with paired compensations, plus an outbox so side effects publish reliably after the write commits.
Order API ──> Step Functions Saga ──┬──> Reserve ──> Charge ──> Confirm
│
[failure] ──> Compensate (reverse order) ──> Repair Queue
│
DynamoDB Outbox ──> Streams ──> Event Publicationsaga-orders/
├── saga.asl.json # State machine definition
├── src/
│ ├── steps/ # Forward operations
│ ├── compensate/ # Compensating actions
│ └── outbox.py # Outbox publisher
└── tests/
└── failure_matrix.md # Failure scenarios + repairsRun active operations audits utilizing the custom sandbox telemetry receiver widget below.
Steps span services with no shared transaction; failures must unwind predictably.
Orchestrated saga via Step Functions
Accepted centralization for explicitness and auditability.
Extract the outbox publisher into a shared construct for other services.
Retried steps must never double-charge or double-reserve.
The gap between writing to the database and publishing the event.
Root Cause:Crash between the two leaves state and notifications inconsistent.
Walked through crash points in the saga and found the dual-write window.
Transactional outbox: write state and outbox row atomically, publish via DynamoDB Streams.
Adds a table and a publisher component to operate.
In distributed systems, "write then publish" is a bug waiting for a crash.
Compensations deserve the same design rigor as forward paths.
Orchestration makes complex failure handling legible.
Customers experience rollbacks as reliability when they are clean and communicated.
I would add saga execution dashboards showing live compensation activity.
Audit raw repository script configurations directly inside the active terminal workspace.
Keep exploring
A multi-cloud serverless deployment leveraging AWS Lambda, DynamoDB, and Azure Functions to build scalable pipelines that minimize operational overhead.
ArchitectureAPI Gateway → Lambda / Azure Functions → DynamoDB / CosmosDB
A security orchestration pipeline that detects unauthorized EC2 security group modifications via EventBridge, orchestrates response with Step Functions, isolates instances with Lambda, captures EBS forensic evidence, and publishes reports to SNS.
ArchitectureEventBridge → Step Functions → Lambda → EBS Snapshots → SNS
A production-style distributed queue processing system: a Lambda worker drains DynamoDB orders from SQS with structured correlation logging, multi-dimensional CloudWatch monitoring, SNS alert routing, injected failure drills, and runbook-driven recovery measured against SLOs.
ArchitectureSQS → Lambda Worker → DynamoDB → CloudWatch → SNS Alert Queue